I was asked a question over email, and I’m paraphrasing the essence as, “given our project issues what tool can we use to automatically pull out all this data and automate the app to let us test it properly?”.
Unfortunately most of the time, when I’m asked this, the answer isn’t what people want to hear.
What they want to hear is “use tool X”.
What I often have to tell them is… “no tool will save you, I would do these things instead”.
When can tools help?
Tools can help quickly when you are using standard technology and you need to interact with it in an obvious way e.g.
- query a database you have access to
- send HTTP requests to an API
- compare files
- check links on a web site
- etc.
A tool works quickly and easily when used for well defined tasks, implemented by out of the box functionality, on standard technology, used as you would expect.
When don’t tools help?
Tools cannot help as well when:
- we need to access the database but the development team won’t give us access
- we want to see the data in the application but don’t know the database
- we want to see the data but it is stored in a custom format
- we’ve been told there is an API that the GUI uses but it isn’t documented
- we want to scrape data off the GUI because we can’t get access to the storage
- we want to know the versions of files installed on the server but they won’t give us access to the server
- etc.
None of the above are really technology problems. They are attempts to bypass social dynamic issues.
Tooling as workarounds
Sometimes we look for technology solutions to help us bypass people problems because we have work to do.
I’ve certainly done that, but it requires knowledge and experience prior to finding a tool. And… really it’s hacking as a replacement for effective managing.
e.g.
- Because the external vendors would not give me access to the database of their tool to create custom reports and analysis. I accessed their tool database using MS Access and the default admin password that I found in a document in a network share that I scanned with some custom scripts to find documentation that might have the information I needed.
- Because the external vendors would not give me the password to open the excel spreadsheets that contained the formulas for the calculations of the reports they were using and which I was sure were incorrectly calculating the information they were passing on to us, I found a way to bypass the password using OpenOffice to read the data and the code used to calculate it.
- Because the external vendors would not explain their unit testing approach or show us the internals of the application to see if there were any architectural risks or API access, I used decompiling technology to give me visibility into the application and make a risk assessment and review the code.
- etc.
Note that I rephrased these so that the actual problem was at the front.
Because of …some social interaction and communication issues… I used my existing technical knowledge to use tools to bypass the situation and get the information I needed.
And…
- This could have backfired and made the social interaction and communication issues worse. These were not my first approaches to solving the issue, and I was working on the social interaction and communication issues in parallel to this ‘workaround’ activity.
- I had to have the knowledge of what to do prior to using tooling. I had a choice of tooling because I understood the technical details of the problem I was working with.
- This would not work with all the problems I’ve faced. When I haven’t been given access to production environments, I didn’t use tools to hack into production to gain access. There is a legal and ethical dimension associated with tooling workarounds.
It would have been far easier if we had managed to solve the communication and social interaction issues.
But what if there really was a tool?
Let’s say that you don’t have the technical knowledge to hack your way around the social interaction.
If there really was a tool, and I told you what it was, you would now have additional activities and risks:
- you need to learn how to use the tool
- you need to learn the technology of the tool and system to understand the results
- you may create a new maintenance problem
- you may need to find a budget
You now have to spend time learning to incorporate the tool into your process. Chances are that you need the tool because of time limitations and were hoping that the tool can speed up your process, but, it just slowed you down.
Without the technical knowledge to fully understand the tool, how it works, and its capabilities you might not identify technical limitations with the tool and report information back as false positive. This frequently happens when incorporating new scanning technology into your project e.g. security scanning software. But without the knowledge to assess and interpret the results, we raise issues which are not actually issues.
If we don’t pick a tool carefully, and don’t use it effectively, then we might create a maintenance issue for ourselves that may not give us issues immediately, but later, after we have come to rely on it.
Without a budget we often only look at free and open source tools, which may not have the polished interface we need to get started quickly or have support to act as an offset mechanism for our lack of experience.
Tools require knowledge and experience so that we understand them, what they do, how they do it, and what alternatives we can use if the tooling itself starts to fail us.
Feature Requests
Many of the questions I am asked about over email don’t require tooling as the solution.
Some of them could require new features in the application under test to support the testing.
e.g.
- additional reports
- data exports
- an admin interface to control data
- etc.
Additional reports can often remove the need for someone to access the database or file system directly. These can often prove useful in a live environment as well, since often the information we need during testing will also be required for reconciliation in live.
Data exports can often remove the need for scraping data from the GUI. This can often prove useful in a live environment to support adhoc requests and reporting later.
And an admin interface to control data can help configure the system into controlled states and limit the amount of data in the environment to make the testing more controllable.
But all of this requires fairly good communication and social interaction otherwise there may be no willingness or motivation to implement the features.
Tooling Isn’t The Only Solution
When tooling is the solution, the right tooling can be a marvellous addition to your process.
Tooling isn’t the only solution.
Tooling can introduce risks, so we have to be careful when we adopt tools.
One risk of tooling is that it can make the social and communication issues worse.
Tooling often seems easier to introduce. But I’m not sure I’ve ever seen tooling act as an effective replacement for the resolution of social and communication issues. But the resolution of those issues is often outside the realm of influence of the people asking for help, so they seek a technical tooling solution in the hope that one exists.
Social and communication issues can be one of the hardest things to resolve, but when resolved they lead to a better work environment and an improved long term situation.
I am often brought on site as a consultant to help with technical issues. And when I’m onsite, I also help solve problems longer term by working on the social and communication issues. You can find more about my consultancy work at eviltester.com/consultancy
Alan Richardson works a Software Development consultant helping teams improve their development approach. Alan has performed proof of concepts for clients for automated execution, and has worked with teams to help them implement fast and effective experiments to improve their process. Alan consults, performs keynotes and tutorials at conferences, and blogs at EvilTester.com. Alan is the author of four books, and trains people world wide in Software Testing and Programming. You can Contact Alan via Linkedin or via his consultancy web site compendiumdev.co.uk and testing web site eviltester.com.
