What is Software Testing? My working definition.

In The Evil Tester Show podcast episode 8 I described definitions of software testing and outlined my own. In this post I will deep dive a little bit more into that model.

Note: this was originally written in 20190305, but I accidentally left it in draft until a content review found it on 20200621

Posts in this “What is Software Testing? " series:

A Definition

In my podcast episode I presented my current definition of testing as:

comparing models of system with observed interactions of the system for the purpose of…. evaluating the comparison and refining the model

This is a more specific version of a general “What is Testing?” which I would phrase as:

“comparing models of the thing with the thing”

The key points for this are:

model that we have
the thing we are ’testing'
a comparison process

My definition is an attempt to identify what I perceive as the fundamental core of testing and strip away parts which I think are relationships between the context Testing is performed in, and Testing itself.

Exploring A General Definition

“comparing models of the thing with the thing”

Pretty generic.

I didn’t write:

Why we would do this: quality, assessment, information provision, stakeholder communication, risk reduction, etc.
What the thing was: System, Software, Product, Process, etc.
How we compare: exploratory, from requirements, technical, predefined quality attributes, etc.
Who would do this: Software Testers, Testers, Developers, et.c
When we do this: prior to deployment, prior to build, after design, etc.

I didn’t write very much of anything.

But, to me, this captures the essence of what I do when I test.

Models and Things

My testing starts from a model. I might even say “All Testing Is Model Based”

I certainly have a model in mind. And if I have null model then my first comparison will be perceptions of the “The Thing” which I then use to build my model.

The comparison process helps me build my model.

I will have stuff in “The Model” that I have not mapped to “The Thing”
- stuff I think might be in the System but have yet to map
- stuff in the model that is not in The Thing
  - should it be? that might be an issue?
  - should it be in my model? Perhaps I don’t understand The Thing well enough
I will have stuff that I associate with both “The Model” and “The Thing”
- stuff that I have already compared and found a mapping for
I will have stuff in “The Thing” that is not in “The Model”
- I use multiple models, perhaps it is in one of the other models
- Perhaps it is ‘waiting to be mapped’
- Perhaps my model does not cover it, and it should

When do I create my model?

I might create some of it in advance, perhaps based on requirements?
I might have some of it as presuppositions and assumptions. I assume this Thing will be like that Other Thing.

Models

What types of models?

requirements
risk
structural
physical
flows
ideas
data
usage
scnearios
what ifs
regulations
performance criteria
etc.

Comparing

How would I “compare”?

That might lead me on to identifying” Test Techniques, a Test Approach, a Test Strategy.
Depending on The Thing I might need tools or extra knowledge to perform the comparison.

What would I compare?

I might compare at a surface level.
I might go deep into technology
Perhaps I’m comparing wishes with designs.

Why would I compare?

As I compare, I would expect to have to evaluate the comparison.
I will find things that map well, and those that don’t.
Some I will absorb into my model and expand my model.
Some will cause me to amend my model because it doesn’t match the system.
- Sometimes I’ll create a new version of my model and hold on to my old model just in case.
Some I will communicate to other people who might be interested or are better able to evaluate the comparison.

When would I compare?

That depends on The Model and The Thing
Some Things have to be physically delivered to compare.
Some Things exist in people’s heads and we can question them.

A Software Testing Definition

comparing models of a system with observed interactions of the system for the purpose of…. evaluating the comparison and refining the model

System, to me can contain sub-systems so System is pretty encompassing, I didn’t feel the need to make it plural.
What is a System?
- the software, the environment, the users, the hardware, the databases, the libraries used,
- the system under development (what we build)
- the system of development (the way we build it)
- pretty much everything
- and I will have multiple model of it
Observed interactions
- this is really a model
- I don’t actually work with the real system
- I work with my perceptions of the system. This is a relationship.
- Everything is a relationship: coverage, bugs, requirements, risks. Everything.
- There are some things I won’t see, those things are in the System, but I might not observe them, they might not make it into my model.
- I work with an abstraction of the system
- I have to use tools to observe parts of the system e.g. look in a database
- Perhaps the tool has bugs. I am limited by my tools. I am observing what the tools are observing, not the full System.
I added a purpose, but the purpose was internal to the process, not the reason for the process existing
Communicating the evaluation to other people sits outside this definition, I have to evaluate before I can communicate this to others, and I evaluate to help me build and refine the model

Warning

I was reticent about explaining my definition. So I’ve tried to address some possible objections below.

Is it complete?

I doubt it. I only wanted to get at the heart of what I do. To give me focus.

I didn’t need it to cover everything I do.

Some of what I do will also be covered by:

“What is Business Analysis?”
“What is Project Management?”
“What is Programming?”
“What is Automating?”
“What is Hacking?”
“What is Deployment?”
“What is Environment Provision?”
“What is Consultancy?”
“What is Mangement?”
“What is Risk Analysis?”
“What is Communication?”
etc.

But Other People Might Not Understand It

That’s OK.

This was for me.

Most people I meet don’t want to know “What Is Software Testing?”

They might want to know:

“What do you do?”
“What does a Software Tester do?”
“What do you do on this project?”
“What do you do on this team?”
“What can you do for me?”
“What value do you add to me?”
“What can you tell me that is useful?”
“Who cares about what you do?”
“Who do you report to?”
“What do you report to the people you report to?”
“What do people do with what you report to the people you report to?”
etc.

I really wanted to know what Software Testing is to me.

But I Disagree!

Possibly.

If you’ve created your own definition and model. Then we agree. Because that’s what I did too. And hopefully we agree that we should choose our own definitions and models.

Unless you think I should use your definition to the exclusion of all others?

If you haven’t created your own definition yet then

Consider…

The person who created the definition you are using, do they still use it?
Perhaps they learned things that caused them to adapt and change their definition.

And consider that a first step to creating your own definition is also the first step in fully understanding the definition you currently use:

Can you fully explain it in your own words?
- Or do you use a call to authority? e.g. “Bob says…”
Are there any parts of the definition that you don’t fully agree with?
Are there any things you do that you think of as Software Testing that don’t quite fit the definition you use?

And, because when we create our own models we are influenced by the models and definitions we have already seen, so honour that by thinking about:

What in the definition you currently use really resonates with you?
What parts of the definition really provide you with insight into your work?

Knowing the above will help you when you iterate over your own definitions.

But your definition doesn’t cover X!

Possibly not.

But that doesn’t mean I don’t do X, or don’t think X is important.

I might identify X inside the definition.

And, I might not.

I might identify X as something that is a relationship between the Contenxt and Software Testing, rather than as something that should be part of the definition.

Definitions can’t really be large enough to encompass everything and still make sense.

And Lewis Carroll Wrote…

… And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!”

“Have you used it much?” I enquired.

“It has never been spread out, yet,” said Mein Herr: “the farmers objected: they said it would cover the whole country, and shut out the sunlight! So we now use the country itself, as its own map, and I assure you it does nearly as well.”