TLDR; Question: Can selenium be used for real world testing? And does it take longer to create different scenarios/permutations than just testing the system manually? A: Yes, Probably
I received the following Question via email from a reader, and since the answer expanded, I moved it to the blog.
Question
Question:
Can selenium be used in testing real world scenarios? And would spending time programming selenium to create different scenarios/permutations take longer than using the system manually creating scenarios through exploration?
Short Answer
Short Answers:
- Yes it can be used for Real World Scenarios.
- And probably takes longer than interacting manually.
- But… if you need to continually check that the assertions coded in the scenarios still pass, and you the time involved in maintaining them and writing them is less than the time involved in manually checking this each time, then automating might be a good fit.
Longer Answers
“Question: can selenium be used in testing real world scenarios?”
Yes. I, and teams I work with, use Selenium WebDriver in the real world for production work and to implement user scenarios.
“Question: And would spending time programming selenium to create different scenarios/permutations take longer than using the system manually creating scenarios through exploration? "
Generally, Yes. Exploratory testing of the system app and interacting with it will likely take less time than automating that interaction.
At least in the short term. But how many permutations are you talking about?
- 10?
Probably still faster interacting manually.
- 100?
- 1000?
- 10000?
At some point, the physical interaction would probably take longer than the automated interaction.
And where are we considering adding permutations? Data? Flow?
We have a bunch of “What ifs” to consider:
- What if we coded the automated interaction badly? And it fails because of synchronisaton issues that we don’t understand. Then we might not get value from it because we don’t trust it and we have to keep fiddling with the code to make it work.
- What if the system kept changing and we didn’t abstract the different layers of the system correctly so with every system change we have to amend so much code that the amendment takes longer than interacting physically with the system.
- What if the permutation was ‘just input data’ and ’expected output data’ - then the automated scenario might add value almost immediately because the same path is executed but with massive data permutation. Automated tactics can excel in these circumstances.
- What if the permutation is subtle tweaks to the interaction e.g. different orders of inputs, different input mechanisms (copy/paste, typing), different form navigation methods (mouse click, tab, keyboard). When the execution path changes then the automated interaction becomes harder to code for and synchronisation becomes essential - not impossible, just a different approach to the work.
- What if the result checking was incredibly time consuming for a human to perform? - poring through long reports, calculating lots of different totals for different permutations, pulling results out of the database and merging with information from a REST interface and comparing to a GUI representation - perhaps an automated comparison routine would take less time to create and execute?
- What if … ? Feel free to add your ideas here…
It can be done
I’ve successfully automated:
- lots of data combinations across single execution path
- permutations of input order e.g. surname, firstname, address; address, firstname, surname.
- different navigation paths through a gui
- different submission approaches for a form - submit, keypress, mouse click
And all while the system is changing regularly.
Sometimes
Sometimes the automated approach has taken longer than the manual effort, but over time, had we continued to interact with the same scenarios manually we would either have had to not cover those scenarios, or perform those scenarios at the expense of covering other areas of the system. So automating it seemed like the right thing to do.
Sometimes the effort of automating the system seemed to take far too long given then short term nature of the product and the automated code was dropped in favour of manually interacting with the system.
We have to make choices when we choose our test approaches. And hopefully, with experience, we can learn to choose the most effective balance between manual interaction or automating without pursuing the wrong path first, wasting time, and then changing approach.
