The following slogans were all automatically generated by The Evil Tester Sloganizer. And I then thought about what they might mean, and posted my reflections on Instagram. I’ve collated them here because they might trigger thoughts for you.
I originally wrote this on the 8th of November 2019, but forgot to remove it from draft status, which I did on 19th March 2022.
“Have you tried Model Based Testing?”
“Have you tried Model Based Testing?” asked my sloganizer this morning.
And in the world of Software Testing, Model Based Testing normally refers to an approach to automated execution build around executable models e.g. state transition models.
And… it might refer to building models of test data, and automatically generating the data to feed into our testing, etc.
The reality is, that all of our testing is model based.
At a simple level, we build Models of the System we are testing. Then we do stuff and compare what we did, how we did it, and the results we observed, to our models. If our models do not have our observations in it, then we expand our models so that they do. And if our model and the observation doesn’t match we start investigating a possible bug.
As soon as we realise that we are working from Models all our notions of coverage change, because coverage relates to the models that we use, not some vague notion of ’test coverage'.
There are formal modelling approaches that we can use - finite state machines, markov chains, directed graphs, regular expressions. And these often come with formal and well defined techniques for traversing them e.g. spanning trees.
Model Based Testing doesn’t just mean automating.
Model Based vs Model Driven
In addition… I use two different phrases “Model Based” and “Model Driven”. I think most people use “Model Based” to refer to what I consider is “Model Driven”.
- Model Driven - create a model and then everything you do is dictated from the output of applying algorithms or techniques to that model.
- Model Based - create a model and then what you do is influenced by that model, and the results of what you do allow you to amend or expand the model.
Model Driven Automating is the use of models for automating.
Model Driven Testing is the application of formal techniques to models to create ’tests’ e.g. Boundary Value Analysis (BVA) where you would use as input the boundary value, and +1 and -1. And whatever other BVA techniques you apply.
Model Based would involve taking the model that was fed into BVA and considering things like: is this modelling the correct domain? Are the values in the equivalence classes actually equivalent? Is this boundary actually implemented as a +1, -1 boundary… perhaps it is a ‘for each’ iteration rather than a ‘for x’, perhaps I should be more concerned about concurrent amendments, is it an idempotent iterator? What data structure is this boundary represented as, is +1 really the maximum I should try?
“Let thy principles be brief and fundamental”
I will need to add some ‘quotes’ into my sloganizer. I came across this quote in Marcus Aurelius Meditations today. “Let thy principles be brief and fundamental”
I think that this is good advice. The model that we build for ourselves for testing should be as brief and fundamental as we can make it.
Look for the universals in testing rather than itemising lists of specific things.
For every specific thing you identify, can you generalise to something higher that perhaps covers more concepts.
How would you explain testing without using buzzwords and technology?
I’ve found, when I do this, that my resulting model of testing allows me to pull in concepts from psychology, math, systems theory much more easily. Because these seem to be the foundation upon which I have built my model of testing. Your model may have a different foundation when you generalise and identify its core.
“You don’t have Bugs, without Testing”
Some people equate bugs with Testing.
Do we have bugs without testing? Depends what you mean by Bug. Is a Bug a problem that someone experiences? Is a Bug a problem that someone important experiences, so we care more and try to fix it?
There are an array of words used to describe problems in software: Defect, Bug, Issues, Error, Failure etc. And there are various standards which define what these things might mean.
If flaws exist in the system, then they will exist, regardless of Testing.
The impact of those flaws will not manifest unless the system is exercised in such a way that the flaw is triggered e.g. the code is exercised, the data is such that it exposes the flaw.
A flaw might never be noticed if no-one is observing the system carefully enough.
e.g. perhaps a flaw causes the router to reboot at 02:11 every day but no-one is around to see it, and no-one checks the logs, and it reboots into a working state so no-one notices anything untoward.
Testing does exercise the system, and does so in numerous ways, which increases the likelihood of triggering a flaw.
Testing requires a lot of observation which increases the chance of observing a systems flawed behaviour. Testing frames the flaw as a Bug, to justify investigating it, and then it is reported making it visible.
Visible reported bugs, prior to a customer finding them, is a side-effect of Testing.
I wrote this because my sloganizer said “You don’t have Bugs, without Testing”.
