Software Testing and Spies in The Art of War

TLDR; The Art of War Chapter 13, on spies, is directly applicable for interpretation in terms of Software Testing.

I interpreted some of Chapter 13 of The Art of War, in terms of Software Testing. I encourage you to read The Art of War, and make your own interpretations.

I was collating the transcriptions of the 33 videos in my “How I Test Web Apps” Micro Course and noticed that my ‘recon’ section was numbered 007.

This synchronicity had passed me by when creating and numbering the videos.

007 - James Bond - Spy - Reconnaisance (recon)

And while I have read the Bond books, and various other spy thrillers over the years.

My mapping of Spying on to Testing wasn’t a result of Bond. It was a result of the 2000+ year old Art of War - specifically, Chapter 13.

Reconnaisance in Testing

In reference to Exploratory Testing I refer to a Recon session as one which I carry out for the purpose of learning about the system.

I do this in advance of any hands on testing
I do find ‘bugs’ as I do this, but that is not my main aim
I am building my knowledge of how the system functions
I identify ‘risky’ areas
I identify Questions that I might want answered before I start

Essentially, I’m getting hands on with the System to build a model of the System to support me in my testing.

The Art of War

I started associating Spies with Testing, years ago, when reading Sun Tzu’s “The Art of War” and directly correlating the text to my model of testing.

The most commonly encountered translation of The Art of War is from Lionel Giles and long out of copyright, so re-used everywhere.

Here are some online sources to read it:

Sun Tzu dedicates the entirety of Chapter 13 to “The Use of Spies”

Spies and Testing

I think it is worth reading, and mapping your own interpretation on to Software Testing, but I’ll write up some of what I have mapped below.

“Raising a host of a hundred thousand men and marching them great distances entails heavy loss on the people and a drain on the resources of the State”
…
“Thus, what enables the wise sovereign and the good general to strike and conquer, and achieve things beyond the reach of ordinary men, is foreknowledge.”

Sun Tzu describes throwing a lot of people at a project, at great cost and time. But if we develop foreknowledge then we can operate more quickly, and more cost effectively.

When I perform exploratory testing, I try to gather foreknowledge through such processes as:

asking questions of people involved in the project
conducting recon sessions
making notes of questions, and finding answers to these in the system early
making notes of assumptions, and determining the validity of these in the system early
building my model of the application as I test
expanding wider, and expanding more deeply, my model of the application as I test

I try not to build too much on ‘assumptions’ - this was what we used to do when we conducted a lot of ‘structured’ and ‘scripted’ testing. Having a lot of people build a lot of Conditions -> Cases -> Scripts from Requirements and Wireframes, leading to a lot of waste and rework, for a long time.

“Knowledge of the enemy’s dispositions can only be obtained from other men.”
“Hence the use of spies”

We can use tools. And we do use tools. To do ‘spying’ for us. Observation, Monitoring etc. But:

Alerts are usually processed by people to determine their validity and seriousness
We rely on people interpreting the Data in tools, to convert it into Information, to Communicate in a way that results in Action.

e.g.

When the assertions in the automated execution fail. We investigate.
When our static analysis tool flags a problem. We investigate.

Approaches to Testing

“there are five classes: (1) Local spies; (2) inward spies; (3) converted spies; (4) doomed spies; (5) surviving spies.”

I might think of Local Spies as gaining access to the system itself:

observing logs
looking in the database
monitoring the memory on the machine

I might think of Inward Spies as harnessing the official external interfaces and APIs for our own purposes.

observing HTTP traffic
looking in the DOM
using the API
using the GUI

I might think of Converted Spies as harnessing anything created for supporting development:

debug facilities,
increased logging information,
read.me files,
tools created during the development,
access to source code,
version control history
etc.

I might think of Doomed Spies “doing certain things openly for purposes of deception” as augmenting my testing through tooling:

static analysis
dynamic analysis
automated execution
debug tools
tactical automating
strategic automating
test data combinations
etc.

“Surviving spies, finally, are those who bring back news from the enemy’s camp.”

And “Surviving spies” as the process of testing. People Testing who are engaged with the system, and the tooling, modelling the system to find new new sources of knowledge, and who report it for the benefit of the project.

“When these five kinds of spy are all at work, none can discover the secret system… It is the sovereign’s most precious faculty.”

How often have you been asked:

“How did you find that?”,
“How did you know to look there?”

Perhaps you are already harnessing this secret system.

The “divine manipulation of the threads.”

Perhaps there is more you can do?

Finding Value in Testing

Sadly not everyone takes Sun Tzu’s words to heart.

“None should be more liberally rewarded.”

Given that Testing doesn’t seem to be ‘more liberally rewarded’ this does not seem to be the case.

“In no other business should greater secrecy be preserved.”

However, that might be due to the Testing discipline not communicating its techniques, skills, coverage and technical processes as effectively and widely as it could.

And, in the world of Independent Security Testing through bug bounties, the bug findings often are ’liberally rewarded’

Test Management

Sun Tzu goes on to describe the use of Spies.

“Spies cannot be usefully employed without a certain intuitive sagacity.”

Realising the value of Testing in relation to the implemented development processes and contextual deployment.

And Sun Tzu continues by hinting at effective management techniques

“They cannot be properly managed without benevolence and straightforwardness.”

Testing doesn’t play well with company politics and the ‘games’ of management.

Testers don’t like hiding information, or mis-communicating status.

Sun Tzu even hints at the issues associated with trusting testing and not blaming Testing for reporting defects.

“Without subtle ingenuity of mind, one cannot make certain of the truth of their reports.”

Generalists

“Be subtle! be subtle! and use your spies for every kind of business.”

Testers very often come from diverse experience backgrounds and can often offer multi-dimensional insights into the product, project, and process.

Some people in testing will develop the fortitude to comment on topics outside their direct sphere of influence and responsibility.

Improved relationships between teams can build when practitioners of Testing are viewed as having insight into multiple topics ranging across the business.

Improving Testing Skills

Sun Tzu describes the continual expansion of Testing Skills and how we build knowledge upon knowledge as we test.

“The enemy’s spies who have come to spy on us must be sought out, tempted with bribes, led away and comfortably housed. Thus they will become converted spies and available for our service.”

By working closely as a team we are able to share knowledge that people who Test can harness for the purposes of testing in ways that other people might not realise.

“It is through the information brought by the converted spy that we are able to acquire and employ local and inward spies.”

By learning more about the System Under Test, the Technology in place, and increasing our ability to Observe, Interrogate and Manipulate the system. We improve our ability to test in ways we might not fully understand before we have access to those sources of knowledge.

“It is owing to his information, again, that we can cause the doomed spy to carry false tidings to the enemy.”

The more we understand, the more we can use appropriate tooling and tactical solutions to help us test more effectively.

“Lastly, it is by his information that the surviving spy can be used on appointed occasions”

The more we treat: automating, technical knowledge, exploration, going beyond the obvious; as part of our general model of Testing. The more contextual value our testing can provide beyond the conditions that can be automatically evaluated.

What is the Purpose of Testing?

A long debated question for Testing:

What is it for?
Why do we test?
Is it all about bugs?

Sun Tzu answers:

“The end and aim of spying in all its five varieties is knowledge of the enemy;”

Not Data. Not Process. Not Tests.

Information:

contextual
usable
actionable
understandable

Knowledge.

Is Testing Still Important?

Sun Tzu answers:

“Hence it is only the enlightened ruler and the wise general who will use the highest intelligence of the army for purposes of spying, and thereby they achieve great results. Spies are a most important element in water, because on them depends an army’s ability to move.”

I think Sun Tzu said Yes.

Next Steps

I recommend reading The Art of War.

A quick glance at my bookshelf reveals that I have read, consulted, and revisited, at least eight different translations.

The Art of War has impacted my Test Process, and my Management Process. I thoroughly recommend studying The Art of War by Sun Tzu.

If you are interested in “How I Test Web Apps” then I provide a case study with meta analysis and interpretation as a course for the $5 Patreon supporters. And those on the $1 tier have access to a 30 minute summary and notes on my modelled process.