Unit Testing FAQs and Lessons Learned

TLDR; Unit Testing and Code Coverage can lead to emotive discussions. These are personal lessons learned around Unit Testing and TDD. Keep things simple. Keep refactoring. View Test and Main as symbiotic.

I’ve been doing a lot of programming work recently and wrote down some of my lessons learned around unit testing.

Some general concepts in the below description:

’execution’ coverage - coverage achieved when a line of code is used in passing e.g. setup, but not directly related to the Assertions or concerns of the Test class.

Should I have one Test class per Main class?

This can make the Unit tests easy to find.
This also makes the Tests easy to build. If you haven’t got any tests, then start by creating Test classes that match the Main classes and expand them out.
If they are class focused tests and the class is simple, leading to few tests, then this can make reviewing the coverage easier as their are fewer Test classes to review.
For coverage reporting, the ‘package’ is more important than ‘one Test class per Main class’

I want to have a minimum of one Test class per Main class. I may split the Test class into multiple classes if it gets too large (Which might indicate my Main class is too large.) And I may add increased depth to the Main class hierarchy to support readability and organisation of the Test hieararchy view.

Why should I “not trust a Test if I haven’t seen it fail”?

When we add tests after code has created, but we haven’t seen the test fail, then we are not sure if the test will show an error in the future.
Failure means ‘Test has found an unexpected difference. Trigger a difference to make sure you trust the test.
- change the values in the assertion and check that they fail
- change the returned value from the code and check the test fails.

When using TDD, we see the test fail first, and then ‘see the difference’ as the code is created. We need to make sure we have enough tests, to catch as many future differences as we feel necessary.

Should I only have one assertion per Test?

For very simple classes that can work fairly well.
For more complex classes that rule might mean having more Test classes, with a common set of @Before setup, and then small @Test methods which only assert.
I don’t use that as a guideline, it feels artificial.
A guideline I do use is to avoid asserting in the middle of tests. When I review the tests that I have created with Assertions in the middle it is often because I am unaware that that condition has been asserted in another @Test or because I have missing @Test methods.
- Summary: I do use multiple assertions. I try to avoid them in the middle of the @Test class.

Note: this is a widely debated topic and people have strong feelings about it. Some people will enforce and mandate one Assertion per @Test method. That feels too restrictive for me. _I take consolation from some of the examples in Fowler’s “Refactoring” book, which have many Assertions, and they are at the start, middle and end of tests.

Should my package structure for Unit tests match my main code?

When you ‘run with coverage’, then the coverage reports for the tests report on the coverage of the Main classes which are in the same package as the Test classes run.
- if you put your Test classes in a package unrelated to the Main class they are testing then the coverage report may not show any classes covered - by default.
- the “Edit Run Configuration Dialog” shows the ‘packages and classes to include in coverage data’, by default this is the package of The tests you are running. You can amend this to include more classes. If this has nothing selected, then all packages are in scope and you can see all classes that are exercised during the test execution.
I use this as my default because by reviewing the package structures I can see if I have any major gaps in my testing.
Writing tests at the same level of the Main code means that your Test code is going to be tightly focussed on the responsibilities of a small number fo Main classes, making the Test creation and maintenance simpler.
Sometimes I find that reviewing tests, and coverage causes me to restructure my Main code to make coverage easier to assess and understand.

It is too easy to view Main code as ‘gospel’ which Test code must ‘work around’ because Main code can exist without Test code. But when taken together as ‘Production’ code, the Main code and Test code have a more helpful relationship. When I find Test code hard to write, then changing the Main can result in ‘better’ Main code (easier to understand, maintain, extend and reliably build on).

Should I Unit Test My Test Code?

Yes. When we write ‘support’ code for our Test code, e.g. a DSL, Data Generation, Reporting Classes.
- Why? Because it can save a lot of time debugging test failures e.g. Random data generation might cause a test failure if it is not generated correctly. Comparison routines for complex data might case a test failure if not Unit tested effectively.
No. When we are talking about abstraction layers for ‘Page Objects’ or ‘API calls’ - avoid testing ’external interfacing’ Test code.
- Why? Because then we would end up mocking Selenium interacting with our application or Rest Assured interacting with our API. That is too much to maintain. Rely on the results of execution for these classes.
No. When we are talking about @Test methods or classes.

Test code is part of your production system. Much of the test code is guaranteed to be ‘covered’, but is communicating with an external system we do not want to mock. Unit testing our ‘support’ code can remove sources of intermittency and make our test code more reliable.

Are code coverage reports useful?

When we control the coverage scope, to include only a few classes we are interested in. Then run the tests that are targeted at the functionality of those classes. Then they are very useful because we can see any non-covered code.
- There is a risk that future changes impact this code, and we don’t have any tests that will report any unexpected changes to this code.
- If this code is exercised by other tests, and you consider the code simple enough that ’execution’ rather than ’test’ coverage is a good enough safety net then you may decide not to add additional tests. (I prefer not to take this approach and instead write Test code to directly cover the methods in the class rather than rely on ’execution’ coverage.)
- If you leave the code uncovered by tests, then there is a risk that you have to perform low level code coverage reviews more regularly to ensure the uncovered risk status is the same. Sometimes I push my code coverage in low level tests to 100% so that they are no longer ‘flagged’ during my review processes and focus on the important non-covered areas.

I use coverage reports to help me review test coverage, and during code reviews to focus in on tests. And help me identify if my Test code is in the correct package structure and testing at the right level. e.g. if it uses too much indirection in the assertions, then I’m probably testing at the wrong level.

Can I have too many tests?

Yes. If all your tests are covering the same thing then this can increase maintenance effort.
If you have many tests that are very similar, consider adding some data driven or parameterised testing.

Are there any common code smells?

Too much setup code in tests can mean they haven’t been reviewed and refactored enough.
- Consider moving setup code to Before... annotation classes e.g. BeforeEach, BeforeAll
- For complicated setup consider creating Test Objects which can be re-used by many tests.
  - Warning. Keep these ‘Test Objects’ at the lowest level of a package hierarchy as much as possible. If you are sharing the same ‘Test Object’ in many Test classes in many different packages then you may not have architected your classes to be testable enough.

e.g. when I started working on a hobby project I had ‘case study’ tests. These use a common set of data - the Case Study - and cover a lot of ground. This allowed me to move fast and explore different implementation approaches in my prototype. When making the code suitable for production, these Case Study tests were useful in supporting refactoring to better architected code. But I needed to avoid the temptation of using the Case Study Test Objects in the new Unit Tests, because this makes the Unit Tests hard to maintain, and tightly couples the low level Units, with the wider system concerns. I wrote new Unit Tests for the new code, and expanded code coverage at a Unit Level, without using the Test Objects.

Is 100% code coverage useful?

If 100% code coverage is achieved through ’execution coverage’, where high level operations are performed in the code so that a lot of code is executed in passing, but few results are checked, then no, it probably isn’t useful.
You don’t need 100% code coverage to gain confidence that your tests will pick up many issues.
If you decide not to have 100% code coverage from low level class based tests then consider if your class really needs those methods and lines of code that you are not testing.
If you decide not to have 100% code coverage from low level class based tests because the code conditions are hard to test, then consider rewriting your class to make it easier to test.
Considering the usefulness of 100% coverage, for a specific set of code, is useful because we have to to think about the risks associated with both ‘hitting it’ and ’not hitting it’. It is possible to achieve 100% coverage, and obtain no useful change detection information when the tests are executed. The 100% number, by itself, is not useful.

I write more Unit Test code than Main Code, how is it worth the time?

Sometimes the code seems to simple to test. That is ‘at the moment’ - what about the future? Adding test code now, can save time in the future and enourage the Test code to evolve as the Main app evolves.
We may use different amounts of Unit tests, at different times.
- Prototype or early releases my prioritise high level coverage, over low level coverage. This gives us a base set of tests to catch big issues.
- When the app is being relied on by others. We write more low level tests, then review our existing test coverage for duplication, and remove the high level tests that focus on low level conditions, but leave the high level tests that deal with interaction between objects.
Time is more than main code programming time. Testing time, User time, debug time. Production down time. Release management meetings. Sales time to find new customers because we lost customers due to outage or bugs. etc. You might be saving a lot of ‘other people’ time, and ‘your future self’ time.

How much code coverage do I need?

No generic number as a target will help you.
Numbers do not provide confidence, understanding what is behind that number, and how that number is achieved will help you.
A low number tells you that there are a lot of gaps, and that you should have less confidence in the ongoing stability of your code.
A high number, without understanding, should not fill you with confidence, it should fill you with questions: What does that number mean? What are we asserting on? Is that ’tested’ or ’executed’ coverage?

The number is not relevant, the understanding and decision making behind the number is important. Your ‘feelings’ around trust of code based on coverage, are important. e.g. I have a set of 15 Test Classes in one package. When executed they achieve 65% line coverage of the core domain classes of my project. This does not fill me with confidence. I know that much of this is ’execution’ coverage, rather than ’test’ coverage, because of the style of testes in that package. But… I don’t rely on that one package.

How can I use code coverage to help me review my tests?

When you run a set of tests, edit the ‘run configuration’ to only cover the code you want to review coverage for.
Review the tests for gaps, i.e. I don’t have a test that cover invalid condition X, if you already have a high % coverage, then you might find you haven’t coded for this condition. (you don’t actually need code coverage to help with this, but it can be a useful extra step)
Review the coverage for gaps. For any gaps consider:
- Is this code hard to test?
  - Perhaps the code should be simplified and refactored?
- Is the uncovered code, so simple that it can’t ever go wrong?
  - Sometimes I add a test anyway: to ensure that it stays simple, and so that I don’t get distracted by it in future reviews.

More test code means more code to maintain, how is that worth it?

Test code needs to be refactored, reviewed, trimmed and deleted. In the same way that main code does, and because main code does.
Test code is part of the production system. If we want to write less test code then refactor the main app to require less testing.
Consider using parameterisation, and data driven approaches for Unit tests to cut down on the number of Test methods.
Consider creating abstraction layers to support your Test code to allow you to write and maintain less Test code. Apply the same coding techniques that you use on Main, on Test.

It often doesn’t feel worth it when writing the tests. But when I’m debugging and trying to fix a bug, it feels worth it. When I’m debugging and already have a lot of tests (but I had a gap in coverage), I often don’t realise how much faster the debugging and fixing time was when tests were present.

Can Unit Tests find bugs?

Yes. When reviewing code for coverage, I spot gaps. Often when I’m writing tests to cover those gaps, they fail, and I found a bug.
When we fix a defect, we write a Unit test that recreates the bug, then we fix the code and the test passes. We could have written that Unit test earlier, and found the bug earlier.

Unit tests find bugs when I make changes I don’t realise impacted some other part of the app. And I find bugs that would be very hard and time consuming to find ’externally’ through a testing process. Finding low level bugs, with an external process, can be very expensive in comparison to expanding Unit test coverage.

What are some benefits of TDD?

You create coverage as you create code. This saves time in reviews looking for gaps.
It saves future debugging time.
Your confidence about the code grows with the coverage.

TDD takes time and practice. Sometimes, when I don’t know the real shape of the code, I TDD at a high level: get some code, then use my high level TDD to refactor the code, and add additional lower level tests, then refactor and add more tests. When I can, I use TDD.

What are some risks of TDD?

The risk of creating code that is only used by the Test.
- I primarily encounter this situation when I have made my Main code too complicated, and hard to test. I find that splitting my Main code into smaller classed to make them easier to test often removes this situation. This is why the refactoring stage of TDD is important, and why ongoing code reviews and refactoring from different levels of abstraction is important.
The risk of adding methods into ’exercised’ code, which are not covered by tests.
- This more likely happens if you are not mocking classes. I tend not to mock classes so I have to be wary when adding code to other classes which support the class I’m TDD’ing. When this happens, I try to go back and review all the code created, and revisit the code coverage for @Test classes in the surrounding classes.

The risks of TDD are more related to the skill set of the programmer, than they are to the process of TDD. The risks decrease, as we become better practiced with TDD.

Edit Run Configuration dialog in IntelliJ

‘Packages and classes to include/exclude’ are very useful when reviewing Test Coverage. Only including the classes that I’m interested makes my review more accurate and I can more quickly identify gaps in coverage.
When I have ’test’ abstraction classes. I often tick ‘Enable coverage in test folders’ to help me gauge the ‘risky’ elements of my abstraction layers. Because I want to write Unit tests for any Abstraction layers that are not directly engaging with external systems. e.g. I don’t write Unit Tests for ‘Page Objects’ or classes that are composed of calls to Selenium or Rest Assured. But I do want to have Unit Tests for my domain abstractions, logging, and data generation.

Recommended Books

Refactoring by Martin Fowler

Find the book on [amazon.uk] [amazon.com]
Growing Object Oriented Software Guided by Tests by Steve Freeman and Nat Pryce

Find the book on [amazon.uk] [amazon.com]
Implementation Patterns by Kent Beck

Find the book on [amazon.uk] [amazon.com]