There exist test ‘concepts’ which, while seemingly simple, have a tendency to confuse.
- Black Box/White Box Testing
- Functional/Non-Functional Testing
- Positive/Negative Testing
- QA
Years spent studying hypnosis and revelling in the ambiguities of communication have left me with an inability to parse language the way I did as a child.
I used to have the ability to read, and use, these terms without blinking. Now I throw a ParseException.
I realise the above concepts may not seem strange to the majority of people reading this blog post. So I will attempt to describe how I respond to reading these terms, in the hope of spreading a contagion against them.
First some generalities
First some generalities across all the ‘concepts’.
I think that I think, and test, in terms of Systems. More specifically in terms of models of systems.
Yes/No models jar my sensibilities. Testers lives in the grey areas and find treasures in the shadows. Absolutes do not help me explore the System. I do not want guidance by absolutes, I want guidance towards flexibility.
All of these concepts encourage me to think and communicate in ambiguities, while pretending specificity and authority. If I say White Box testing, you still have no idea what I mean. If I describe the models that I use to guide my testing then you have a better idea of the type of tests I might create, and the type of gaps I may encounter, regardless of the colour that I paint that model.
These concepts all encourage me, as a tester, to ignore the model and think in vague conceptual terms. They categorise techniques out of context and far removed from the model that the techniques apply to.
They encourage debate about irrelevancies. e.g. Should we do White Box, or Black Box testing? Developers do White Box Testing, testers do Black Box Testing? Where does Grey box testing come in?
The amount of time wasted on projects when people ‘debate’ if the requirement should form part of the ‘functional’ or ’nun-functional’ specification, instead of getting on with communicating what they need shocks me. (I estimate this in terms of man-years for some projects)
etc.
Black Box / White Box Testing
Ah, the most pernicious of them all.
How comfortingly I viewed these terms in my early years. I could ignore the relationships between my test models. I remember times when I made no effort to learn the technology or architecture of the system. I could simply focus on inputs and ignore flow, because I did my testing all “Black Box” like.
Then over time I grew uneasy. I started to apply White Box techniques to ‘Black Box’ingly understood systems.
This concept started to seem like the devil on my shoulder whispering “Relax, don’t do the work to view it as a system, treat it as a ‘black box’, see how much simpler it all becomes”.
I started to feel as though the constant repetition of this phrase in the testing literature stopped me hearing the siren call of the technology.
I weaned myself away from these terms by using them as insults. I engaged in a form of aversion therapy.
I labelled all testing I did without understanding the technology, or without having made an effort to observe the architecture, or without building a structural model as ‘Black Box’ testing. I imbued the term with an association of laziness.
I view all my areas of superficial modelling, as having greater risk of defects unfound.
I associated my use of ‘White Box’ testing with over-confidence. I felt nervous if I only read the code, or only based my testing on a superficial understanding of the structure. For me, the phrase warns me about my complacency. I try not to plan for complacency.
This helped me move towards a more Systems focussed approach to my software testing. I no longer feel a pull towards exclusively the Darkness or the Light.
I try to concentrate on models and identify techniques to apply to models.
I ignore the categories and partitioning of techniques presented by the Canon.
Functional/Non-functional Testing
I periodically write non-functional automated test scripts… Sometimes my scripts simply don’t work and they remain non-functional until I fix them to get them functional.
In the past, I have communicated my testing in terms of “I built tests to cover the ‘Non-Functional Requirements’”. Which in context could have read “Speed Monster Funky Requirements” and I would have just as happily used that phrase in my test communication.
When we mean requirement based testing, then we should say “requirement based”. Removing the word “requirement” suggested to me that “Herein lies magic”. That the “Functional” tester does something different from the “Non-functional” tester. That we can phase the testing differently and do “Functional” testing and then later do “Non-functional” testing.
I removed my unease towards this phrase by always appending/inserting the word requirement.
But … I don’t like the distinction between “Functional” and “Non-Functional” because of the ineffectual processes and discussion that I have seen result from the distinction. Sadly Software development has a history of considering functionality as separate from the other “ilities” that might relate to that functionality. A tradition or an old charter, or something.
And yet I don’t see why I should let this influence my testing. I’ll do whatever ‘it’ takes, and if that means merging the functional with the non-functional to get more advantages in my testing or tooling then I will jolly well do so.
Positive and Negative Testing
I assume “positive” and “negative” as testing terms, must have come from a more objective diagnostic context than I get involved in. From a context where a negative test, rules something out completely, and a positive test rules it in.
I classify much of the testing I do as exploratory or experiment based. I do things to try and rule out certain possibilities and I could possibly count that diagnostic. But I don’t think my tests demonstrate the certainty that “positive” and “negative” testing suggests to me. I find more value in thinking in terms of a general experimental process rather than a diagnostic process.
But even the above description doesn’t seem to map to how the testers I meet describe positive and negative testing.
Testers seem to describe negative testing in terms of exercising error validation and reporting. And positive as “stuff we think will work”. I tend to view the existence of error handling functionality in applications as a positive thing.
I find it hard to use the terms –ve and +ve testing. When I try to show that the system does not work, I talk in terms of trying to ‘break’ it – clearly I don’t break it, but it works as a term for me. I also view this as a positive thing for me to do.
Yes, I freely admit to misusing words, particularly if the misuse of the word frees my thinking. I try to stop using words that constrain me.
QA
I dislike the term QA when applied to testers in general. I dislike it more when applied to my team. I dislike it even more when applied to me. If you read this, and you use the term QA to mean testing or tester, then it doesn’t.
Unfortunately if I graphed the trend for verbal usage of this term I would see draw a line heading upwards. I used to hear it more from the American staff I worked with, but now this is used world wide.
I hear it more than I read it.
Quality Assurance does not mean investigating the product for the purposes of finding more information out about the product from a multi-tude of different dimensions. Testing maps on to that more than QA.
QA means - did we do the do the things we said we were going to do to the standards we said we woul do them? That means all the things. Done by everyone. Which would be checked by the people doing them.
I test things, I don’t QA them.
The Team QA’s them. The process needs to be designed to learn and adapt and have feedback loops to help Assure Quality. QA is not the responsibility of a single role, it is the responsibility of the System (Process, Team).
Use of the term QA distorts thinking.
Even if just for the second that you say it.
Say “tester” or “testing” or “performance testing” or “review” or “unit testing” or insert whatever phrase actually covers the meaning that you have in mind instead of the generic, highly abstract, and incredibly open to interpretation (and abuse) QA
I’ve been accused of being overly pedantic, or ‘but they are just words’.
Clear Thinking, and Critical Thinking, require effective modelling and effective use of words. Muddled thinking can be aided by ambiguous or misused words.
End Notes
For all of the above ‘concepts’ I can read the definitions and almost understand why those names fit those descriptions. But I try to find other, more exact, or more contextual, or more personal, ways of describing the testing that I do.
You may not need to go as far as I did to de-programme their effect.
My brain still finds a ‘reasonableness’ in some of them while at the same time each of the ‘concepts’ strikes me with dread. Each of them seems to proudly state that we can easily view the System as two simple partitions and we can relax about modelling or understanding the complexity.
When you speak to me using these terms, I may have to ask you what you mean.
