TLDR; Regression Testing is a Legacy term for testing when a risk of unexpected impact exists after making changes.
I want to do discuss a little bit about regression testing.
The traditional legacy view:
- what it is?
- why we do it?
- how we do it?
Then I want to look at regression testing in terms of modern system development.
I created a YouTube video which covers this blog post. Also have a look at Richard Bradshaw’s White Board Testing video on Regression Testing.
I think it is important to have more interpretations of the “legacy” or “traditional” words and phrases that we use to describe testing. I’ll probably create a few more of these posts and videos so if there are any in particular you want to see covered let me know in the comments on the blog or on the YouTube video.
Regression Testing is a common testing phrase.
What is regression testing?
Traditionally what we’re talking about is… when a system regresses to a former or less developed state. A more primitive state.
Assuming that there is a risk that the System is going to regress to a former a less developed state if changes are made.
What we mean by Regression Testing is… testing parts of the system that we’ve already tested when the system changes.
- we change part of the system
- and we test that, we test that properly
- then we go “well you know something else might happen. So let’s test the rest of the system”
And that ’test the rest of the system’ is regression test.
Why do Regression Testing?
Why would we do that? Why? We’ve just tested the change. Why would we do regression testing?
Because the change might have had an unexpected impact on another part of the system.
Systems are big. They have interconnected components.
We’ve changed one part it might have impacted the others.
So we do Regression Testing.
When do we do regression testing?
- Q: So when? When do we do regression testing?
- A: Pretty much whenever we make a change.
We make a change, if we think there might be change impacted somewhere else we do regression testing.
Just to be clear, because a comment on the YouTube video suggested that I might not have been. “A Change”. Any kind of change. Code, infrastructure, environment, libraries, browsers that users use, Java version, OS upgrade, Cache changes, Web Server version. Any change. A change.
That’s what we do.
How do we do regression testing?
The legacy view of how we do regression testing is:
“Well. We’ve already tested it. So what we’ll do is we’ll rerun those tests again.”
- When we have tests in another area we’ll rerun those tests.
- If we’ve got test cases we’ll re-run those test cases,.
- If we’ve written scripts will rerun those scripts.
- We’re doing exploratory testing? We’ll do exploratory testing in those areas again.
How else might we do it?
Because very often when we’re talking about regression testing we’re talking about test automation, so we could run the automated execution in those areas. Hopefully the automated execution doesn’t take too long, so we’ll run just run all our automated execution.
But if our automated execution takes a long time or it doesn’t work particularly well and has a lot of human intervention… Then we might have a subset and choose to run that.
Essentially what we’re talking about is a subset of testing in an area that is not directly related to the area we’ve changed.
We believe it might have some impact and that’s why we’re choosing to test it.
How did this concept arise?
I don’t like this concept and I’ll explore that later. But how did this concept arise?
When we were doing waterfall projects and we had big big projects, big big systems that were taking years to develop and we would develop a lot of stuff without actually testing it.
When we actually started testing it we’d find bugs.
When we started using it we’d find that the requirements weren’t quite right and we’d need changes.
All that change work, the new features and bug fixes was all being done at the same time.
There was very little development team testing. Remember we’d split the team up in to development team and testing team, and they were working independently of each other.
The communication was pretty poor there was very little impact assessment of changes in code, there’s impact assessment changes on a business level. “How will this impact us? What we have to do to change our process?” But we’re expected to just code and get on with it.
Very often there was very little impact assessment done there, and the change tracking was usually very weak. We didn’t always have version control systems that were particularly good.
It was a long time between releases, so we knew what was being fixed that we could then go away and adjust our test cases and scripts. Then they’d be ready when the system came but sometimes the system would have changed our scripts that weren’t quite right.
The whole approach to development was just a massive risk, and the test approach was very fixed on test cases and scripts and slow to respond to changes.
Testing and Regression Testing naturally became about running stuff that we had already defined or agreed we were going to do. So it was about rerunning stuff that we’d already identified.
The problem with Regression Testing.
The problem with regression testing is that the system does not actually regress.
The system does not move to a former state or a less developed state.
The system is advancing to its next state, but there may be unexpected changes.
When I said this at a conference someone said:
“Well what if we were doing testing of our previous version? What if we’ve decided to roll back to an earlier version. Are we then doing regression? Because we’ve regressed to a former state?”
You could call that regression testing if you wanted to.
What I would suggest you do is that you call it “migration” or “roll back to an earlier version” testing and then see what risks there are in that process and then target the risks.
Because testing is really about risk.
Testing can be viewed as being “all” about risk
Testing can be viewed as being “all” about risk. It isn’t, but we can view it like that to help re-frame our thought processes for a while.
When we use the term Regression Testing. We’re kind of obscuring the risk.
We’re just saying “well something might have changed therefore we’ll run some stuff in some other areas”.
If it’s automated “we’ll just run all that stuff again”.
Otherwise… otherwise who knows what?
If we redefine regression testing in terms of risk. Then what we’re saying is:
“there’s a risk that a change has had an unexpected impact on another part of the system”.
But that doesn’t mean that we have to use testing as our only mitigation strategy.
How else could we mitigate the risk of unexpected change?
We could change our development approach to help mitigate the risk of unexpected change.
We could re-architect it to allow us to build in smaller components. So that when we make a change of one component there is a reduced risk of change in another one because it hasn’t actually changed. And then if we do all the interface testing and integration testing around that perhaps it will work. Perhaps we can architect to avoid this.
Perhaps we could have better version control.
Perhaps you could have better change tracking.
Perhaps you could build in smaller increments.
What if we did Test Driven Development? Would TDD help? TDD means writing tests in advance of writing code. Therefore if we change the code, presumably, writing tests for that. Therefore we’ll be testing the new changes. There’s a risk that if we don’t do TDD and we just write code that we write new code that is not actually covered by the existing tests. So the existing tests only help us when we’re actually doing TDD. Also TDD is very focused, and doesn’t deal with the notion of unexpected change further down the line of using the component where some data may have fed through into it.
What about Acceptance Test Driven Development? Acceptance Test Driven Development tests take a bigger view and expanding across components. So it might help. But again what we’ve got is acceptance tests which, presumably have been automated and, cover a specific set of criteria. The changes might be outside that criteria. So it might not overlap with what we’ve already decided is covered. So we might still have to do something else. Acceptance Test Driven Development, if it’s actually driving development can help. Where we write the acceptance test and we write the criteria and then we automate the criteria for those changes; then presumably it will go to the flow and we’ve actually got automated execution across that. Which longer term will help mitigate the risk of changes impacting it.
Behaviour Driven Development? BDD isn’t supposed to be a testing process but has a side effect that we automate a lot of the assertion criteria in the same way we do with acceptance test driven development. So again it might help cover it.
Note: ATDD, TDD, BDD have ’tests’ as a side-effect. That does not mean we have automated ’testing’, it just means we automated the assertion of agreed conditions during a development process. We can use that to detect change in the implementation of the sub-set of behaviour that the agreed conditions cover, but it does not protect us from changes outside the agreed assertion scope or data used.
Automated execution, in general, might help. We might have covered some of those areas impacted by change, but it may not be covering the changes we’ve made.
So again there is a risk in there.
Any time we’re faced with uncertainty. The only thing that really deals with uncertainty, is a new set of analysis. A new set of working out what we have to test, and how we’re going to test it. A new set of ideas.
And you can either go through your formal “Let’s look at the change and let’s do test ideas, test cases, test scripts” whatever else. Or you could do exploratory testing.
Because exploratory testing does that by default:
- comes up with new ideas,
- looks at the risks,
- looks at the system,
- figures out how to test it.
When we’re dealing with unknowns, we either go back to the start of doing our test analysis, or we do exploratory testing which incorporates that in the process automatically.
But we’ve still got the issue that because there is uncertainty we don’t know if our exploratory testing is going to cover those areas.
So we still have to really properly mitigate the risk by understanding the system and its flows and the usage and the change. And then hopefully we will identify the flows that are at risk of this new area of functionality that we’ve added downstream.
The more we can understand. The more we can reduce the scope that we’re going to try and test against. The better off we’ll be when we come to this concept of Regression Testing or “mitigating the risk of unexpected change”.
The Legacy View is Simpler, but the Risk View is less Risky
The legacy view is simpler:
- there’s a risk of unexpected change,
- we pick from our agreed set of scope,
- we re-cover that scope.
- Job done.
There’s very little hassle there.
It probably doesn’t reduce the risk very much. There’s still a huge risk that we haven’t actually covered it because it’s new functionally but it makes things seem simple. So we got a nice simple term like Regression Testing.
In the modern age of software development. We have to do better than this.
We have to get back down to the notion of risk as informing our testing.
Looking at the uncertainty around that risk to try and identify how much we need to test this. Because when we test we try to reduce uncertainty. But as we test, and we learn more, we’re also increasing our uncertainty.
Because we’re identifying:
- new risks,
- new areas of uncertainty,
- new conditions that we haven’t covered,
- new data scope that we haven’t hit.
We have to constantly balance this new information and build a model of our risk and uncertainty.
- What we’ve covered.
- What we’ve not.
- What we know.
- What we don’t know.
It’s harder.
But it’s also more important.
And that’s really the essence of testing.
Put Risk back in.
Redefine your terminology, so that you’re not just relying on old legacy notions of “I’ll just rerun my stuff again”.
We constantly have to think.
Because if there’s a risk, we want to identify the easiest and fastest and best way to mitigate that risk as part of our test process.
So that’s Regression Testing
So that’s regression testing and I hope that helps.
If you’re doing this in any kind of exam or job interview then use the terminology Regression Testing because that’s what everyone’s going to expect.
But in your head. Start reducing your reliance on this kind of legacy vocabulary so that you can constantly start from first principles.
Look at risk, and adapt your process: your test process, your testing ideas and approach, to the actual needs that are in front of you at this point in time.
A constant focus on risk and uncertainty will help you do that. And the notion that “exploration is what leads to new ideas and new coverage” will help.
