TLDR; The Tester in me was not satisfied by the TDD of FizzBuzz so I expanded the coverage with some simple acceptance testing modeled by automated @Test methods.
Previously on “Testers TDD” I created a version of FizzBuzz using TDD we now move on to the Acceptance Testing of FizzBuzz, but do we need to?
Because the TDD code had such a small scope. The ’tester’ in me wasn’t sure if it was good enough or not.
Why not?
- Such a small set of input data
- I can see code coverage is high, but input value coverage is low
- Do I really understand the requirements?
- I created example acceptance criteria and haven’t used it in the testing
Approaches
There are a number of approaches I can take to automate at a requirement or acceptance criteria level.
- Use a different algorithm as oracle and then compare both with a wider set of data
- Data Sampling e.g. use the acceptance criteria examples
- Canned Data Set which has increased scope
- Automatically Generate Data using a simpler Algorithm
- Output from previously checked results
- Output from other people’s implementation as input
First - verify against a different algorithm
I initially wrote a ‘different algorithm’ for calculating FizzBuzz and used that as the comparison for my FizzBuzzConverter:
@Test
public void checkAllNumbers(){
boolean isMultipleOfThree=false;
boolean isMultipleOfFive=false;
FizzBuzzConverter fizzBuzz = new FizzBuzzConverter();
for(int checkThis=1; checkThis<=100; checkThis++){
String expectedVal = String.valueOf(checkThis);
isMultipleOfFive = checkThis%5==0 ? true: false;
isMultipleOfThree = checkThis%3==0 ? true: false;
if(isMultipleOfFive && isMultipleOfThree){
expectedVal="FizzBuzz";
}else{
if(isMultipleOfFive){
expectedVal="Buzz";
}else{
if(isMultipleOfThree){
expectedVal="Fizz";
}
}
}
System.out.println(fizzBuzz.convert(checkThis));
Assert.assertEquals(expectedVal, fizzBuzz.convert(checkThis));
}
}
The above code is pretty complicated and I imagine is representative of a non-TDD FizzBuzz implementation.
The basic principle here is ‘an alternative algorithm oracle’. This is much slower. The test takes about 4 milliseconds to run.
Having an ‘alternative’ algorithm is a fairly common way of checking code for acceptance purposes.
Data Sampling
For BDD style approaches we often use a sample of data represented by a set of examples.
@Test
public void checkAcceptanceCriteria(){
FizzBuzzConverter fizzBuzz = new FizzBuzzConverter();
Assert.assertEquals("1", fizzBuzz.convert(1));
Assert.assertEquals("2", fizzBuzz.convert(2));
Assert.assertEquals("Fizz", fizzBuzz.convert(3));
Assert.assertEquals("4", fizzBuzz.convert(4));
Assert.assertEquals("Buzz", fizzBuzz.convert(5));
Assert.assertEquals("Fizz", fizzBuzz.convert(6));
Assert.assertEquals("7", fizzBuzz.convert(7));
Assert.assertEquals("8", fizzBuzz.convert(8));
Assert.assertEquals("Fizz", fizzBuzz.convert(9));
Assert.assertEquals("Buzz", fizzBuzz.convert(10));
Assert.assertEquals("11", fizzBuzz.convert(11));
Assert.assertEquals("Fizz", fizzBuzz.convert(12));
Assert.assertEquals("13", fizzBuzz.convert(13));
Assert.assertEquals("14", fizzBuzz.convert(14));
Assert.assertEquals("FizzBuzz", fizzBuzz.convert(15));
Assert.assertEquals("16", fizzBuzz.convert(16));
Assert.assertEquals("Buzz", fizzBuzz.convert(100));
}
While this covers the agreed examples, it doesn’t really make me feel any better for coverage.
This could easily have been:
- an array or list of data that was iterated over
- A data provider to a parameterised JUnit test with the number and expected result reported in the @Test name
Feel free to implement these as an exercise for the reader.
Pre-Canned Data
Since there are only 100 values used to check against.
It would be pretty easy to pre-generate the entire output dataset.
Instead I decided to pre-identify the values for Fizz, Buzz and FizzBuzz.
I did this manually, adding each value into an array.
As you would expect, this was an error prone process and I actually missed out two of the values in the ’test’ data. This caused the test to fail because my ‘pre-identified’ input data was incorrect.
Fortunately my FizzBuzzConverter algorithm was not fooled and did return the correct value.
@Test
public void checkAllNumbersPreCanned(){
Integer fizzBuzz[] = {15, 30, 45, 60, 75, 90};
// creating this manually was prone to error - I missed out 35 and 100 originally
Integer buzz[] = {5, 10, 20, 25, 35, 40, 50, 55, 65, 70, 80, 85, 95, 100};
// creating this array was a pain of mental arithmetic
Integer fizz[] = { 3, 6, 9, 12, 18, 21, 24, 27, 33, 36, 39,
42, 48, 51, 54, 57, 63, 66, 69, 72, 78,
81, 84, 87, 93, 96, 99};
FizzBuzzConverter fizzBuzzer = new FizzBuzzConverter();
for(int checkThis=1; checkThis<=100; checkThis++){
boolean checkedIt=false;
if(Arrays.asList(fizz).contains(checkThis)){
Assert.assertEquals("Fizz", fizzBuzzer.convert(checkThis));
checkedIt=true;
}
if(Arrays.asList(buzz).contains(checkThis)){
Assert.assertEquals("Buzz", fizzBuzzer.convert(checkThis));
checkedIt=true;
}
if(Arrays.asList(fizzBuzz).contains(checkThis)){
Assert.assertEquals("FizzBuzz", fizzBuzzer.convert(checkThis));
checkedIt=true;
}
if(!checkedIt){
Assert.assertEquals(String.valueOf(checkThis), fizzBuzzer.convert(checkThis));
checkedIt=true;
}
Assert.assertEquals(true, checkedIt);
}
}
Algorithmically Generate the Data in Advance
Rather than use a different FizzBuzz algorithm, I decided to implement the business rules as data generation algorithm.
So a list of all Fizz values - i.e.
- 3*x where the value is not also a multiple of 5
- 5*x where the value is not also a multiple of 3
- x*(3*5)
This had the advantage that I didn’t need to understand the rules, I just implemented them. Also, if I didn’t know how to use modulus then I could still algorithmically generate my data.
Also, I was able to scale this up to higher numbers than the manually created data generation.
You can see the full test code in GitHub, but I have split it into two sections below.
Data Generation
List<Integer> fizzBuzz = new ArrayList<Integer>();
List<Integer> buzz = new ArrayList<Integer>();
List<Integer> fizz = new ArrayList<Integer>();
// use an algorithm to calculate the values in advance
int maxVal = 100;
for(int i=1; i<maxVal; i++){
// For numbers which are multiples of both three and five print "FizzBuzz"
int fizzBuzzVal = i*(3*5);
if(fizzBuzzVal<=maxVal){
fizzBuzz.add(fizzBuzzVal);
}
// and for the multiples of five print "Buzz".
int buzzVal = i*5;
if(buzzVal<=maxVal){
if(!fizzBuzz.contains(buzzVal)) {
buzz.add(buzzVal);
}
}
// But for multiples of three print "Fizz" instead of the number
int fizzVal = i*3;
if(fizzVal<=maxVal) {
if (!fizzBuzz.contains(fizzVal)) {
fizz.add(fizzVal);
}
}
}
In the above generation I have shown the non-modulus data generation. This is simpler and basically says. “If I have already added this number in a list, then do not add it again.”
I could easily have used modulus code e.g.
// do not add if also multiple of 5
if(fizzVal%5!=0) {
fizz.add(fizzVal);
}
Then the actual execution.
FizzBuzzConverter fizzBuzzer = new FizzBuzzConverter();
for(int checkThis=1; checkThis<=maxVal; checkThis++){
boolean checkedIt=false;
if(fizz.contains(checkThis)){
Assert.assertEquals("Fizz", fizzBuzzer.convert(checkThis));
checkedIt=true;
}
if(buzz.contains(checkThis)){
Assert.assertEquals("Buzz", fizzBuzzer.convert(checkThis));
checkedIt=true;
}
if(fizzBuzz.contains(checkThis)){
Assert.assertEquals("FizzBuzz", fizzBuzzer.convert(checkThis));
checkedIt=true;
}
if(!checkedIt){
Assert.assertEquals(String.valueOf(checkThis), fizzBuzzer.convert(checkThis));
checkedIt=true;
}
Assert.assertEquals(true, checkedIt);
System.out.println(fizzBuzzer.convert(checkThis));
}
Other Approaches
- Output from previously checked results
- Output from other people’s implementation as input
I didn’t implement the above approaches. These would likely take the format of text files that I read in and check the results.
I have used both of these in the past. They may require updating over time and they require interactive investigation when the input and output do not match.
All have proven useful under different circumstances in the past.
Ultimately
Ultimately none of these alternative methods: increases the coverage of data or demonstrated a problem in the code that had been created via TDD.
They did:
- increase my confidence in the final code
- offer additional coding challenges
They might:
- offer additional assurance if I ever refactor my
FizzBuzzConverter
If this had been a project with multiple stakeholders:
- a programmer who believes code is working based on TDD
- a product owner who trusts programmer and wants to go live
- a tester who has not checked the full output of the app and is unsure if it covers all the data effectively
From a tester perspective this offered a ‘useful’ exercise because they are more sure of the output of the code
But from ‘programmer’ and ‘business’ perspective it offered no additional insight and consumed time and resources which we could have spent elsewhere and gone live earlier.
Real projects have to make decisions based on the value of information. So sometimes. We don’t get to conduct the experiments that we want to, which will increase our confidence.
But sometimes - they are so fast to automate or conduct - that we do it anyway e.g. in lunch hour, such is the life of a tester.
The video for this post has been released as a Patreon video with an additional more detailed Patreon explanation.
The code has been released to Github.
