Lessons learned and tips for testing from public bug bash events

TLDR; Tips for external public bug bash testers. This is not testing as you know it. Change your beliefs. Get ready in advance. Hit the system hard, hit it fast. Report quickly and well. Demoralise the other teams. “Congraturation! You Sucsess!”

Stop the Express - ZX Spectrum - from RZX Archive youtu.be/Xze0SGFH_9s?t=1m57s

Lessons learned and tips for testing from public bug bash events

There seem to be a few public ‘Software Testing Cups’ and competitions. Which all seems like jolly good fun. The closest I’ve come to taking part is when I attend public bug bashes. I’ve been to a few in the past. So I’ll list some pro-tips and experiences.

Adopt a different belief model for testing

One thing I particularly enjoy about the competitions and bug bashes is that ’testing’ is very hard to judge as a competitive sport. And unfortunately I love the irony that we fall back on the ’thing’ that we are all told not to do in ‘real’ testing. Which is judge the testing by the bugs we raise. We all know that ’testing’ isn’t ‘all about bugs’ and the value of testing is in the ‘information’ not the ‘bugs’ etc. etc. But not in a bug bash [insert evil laugh here].

So the first set of pro-tips embody that:

This is a competition, that means ‘winners’ and ’losers’
- The losers are going to be the people who hang on to ‘beliefs’ about what testing means
- “Bug Bash” doesn’t imply “Testing” it means “hit the software hard and fast and find as many bugs as you can”
  - Think like a bargain basement testing warehouse
  - Pile em high, sell ’em cheap
- The winners will be the people who can:
  - find bugs fast
  - document bugs into the tracking system (because you’re going to be judged on the bugs you raise in the system, not those in your notebook)
  - have bugs that are well supported with evidence: steps to re-trigger, screenshots, succinct and clear descriptions

How to also go deep

Now, you don’t have to embody that. You can go into the ‘competition’ with an attitude that:

I’m going to learn, and
I’m going to test it properly
Quality over quantity

I’ve done that. It can be a good personal exercise. I was involved in one bug bash on a test environment where I didn’t race to the bottom and pick up everything I saw. Instead:

I scouted around,
observed,
found some good risks,
followed the risks to their functional source,
infiltrated the functionality,
went into deep cover
and found a massive security issue that gave me access to all the customer details on the production system
and no-body cared [insert laugh of incredulity here]

I learned a lot doing that which helped my personal test approach, and about the company in question.
So this can be a good personal strategy to adopt in a bug bash. But when you adopt this attitude - don’t join a team that wants to win.

Often when I follow this strategy I end up finding ‘big’ bugs, so it is generally the strategy I use for private bug bashes when we conduct them internally. The hard part about this strategy is that you often stumble over bugs on the way: during your scouting, reccie, pursuit, infiltration prior to your hard contact. In a team, a good strategy can be too pass those on to other team mates while you continue on your deep infiltration mission.

Don’t…

Based on experience, here are a few “Don’ts” - assuming you want to ‘win’, if you want to ’learn’ then these might be suitable tactics, but you won’t find the winning team using these.

Don’t try and use tools you’ve never used before
- You will lose a lot of time learning and configuring the tools.
- Do use tools you already know how to use.
Don’t assume you’ll configure your machine when you get there
- Make sure you know how to configure your machine prior to the event
  - for different wifi networks, proxies, VPNs
  - to connect your mobile decide to your laptop via cable, or wifi, or bluetooth
- You normally use your laptop with an external monitor, mouse and keyboard?
  - make sure it works without those connected
Don’t check you’ve got the tools you need when you get there
- that USB stick with all those portable tools that you used last year - check it works and has the tools you need.

All of the above relates to a lack of approach planning, so don’t NOT plan.

Do…

You won’t have all the information you need to create a detailed plan, but you should have enough information to to think through the problem.

If you’re testing a web app then:
- make sure your browsers are up to date
- have a range of browsers available to test on
- know how to use the dev tools for the browsers you’re going to use
- make sure you can configure your proxies with the browers
- make sure you have the proxy https certificates installed on your laptop and browser
- make sure you know how to use the proxy
- make sure you know how to toggle between proxy and without proxy
- have link checkers available
- have a mix of local and remote tools
  - e.g. if your link checker is cloud based but the test environment is locked to a local ip range or wifi network then you betterknow how to create a tunnelto allow the remote client to connect, or have a local tool that will work so long as your machine can connect.
- know how to use your proxy functions in unconventional ways e.g. Fuzzers for automating over a massive data set, rather than writing code to automate the system.
If you’re testing a device based app then:
- make sure you know how to toggle developer mode on your device
- make sure you know how to side load apps on to your device (just in case)
  - this also allows you to rip the app off the device for local decompilation and binary artifact investigation
- make sure you know how to take screenshots
- if you choose to do all your testing and bug reporting from the device itself then I recommend being able to hook up a keyboard to it
- if you can, have the ability to take movie
- I recommend using your device in conjunction with a laptop
  - make sure you can transfer screenshots, files etc from device to the laptop
  - connect the device to the laptop
  - show the screen of the device on the laptop
  - take screenshots on the laptop
  - capture movies in parallel on the laptop
  - use the laptop to raise defects rather than the mobile device
Have the essentials
- screenshot tools
- screen movie capture tools
- text editors
- file compare tools
- data generators e.g. counterstrings
- learn to touchtype fast prior to the event
- etc.
Parallelise (is that word? it should be? I hereby declare its existence thus!)
- use tools that can statically analyse as you test
  - e.g. static analysis plugins on tools to check traffic for issues
- scan http traffic logs looking for 4xx and 5xx status codes
- use tools that can work in parallel to you
  - link checkers that spider and scan the site
  - spiders that pull out comments from HTML pages
  - standard compliance tools that spider sites
  - binary analysis tools for common vulnerabilities

Any tool that can work in parallel and alert you to issue that you might not notice, and are a good way to pick up low hanging bugs for free:

spelling errors on pages
missing images
missing include files
server errors on ajax requests
broken links

If you aim to win then you are going to do more in a short period of time than you have ever done before, and you’re going to use a range of tools. Make sure you startup all the tools you use before you go and that they are up to date and functional.

Work as a team

This is essential.

It is way too easy for everyone on a team to hit the same area, and then follow someone else into that area when they find a problem.

Working in pairs can be useful:

particularly if it is a tricky area that neither of you understand fully.
your partner might observe something that you’ve missed.

But… I’ve found that:

sending everyone out on their own,
- to different areas of the app,
- to the same area, with different observational and attack focus
concentrating on each persons strengths
- ‘you like to do accessibility’
- ‘you like to get technical’
- ‘you know how to use tool X, so you can follow behind Bob when he does Y’

A more organized wide dispersal coverage in parallel seems to work better than a self organizing swarm, if you want to win.

And… the team needs to communicate:

when you find a problem then communicate it to your team,
- not so that you can swarm and see it, but more so that you can generalise to a root cause or pathology and see if a similar fault exists in the other areas of the app that other people are looking at.
sometimes it is useful for someone to find a problem, then pass on their notes to another team member to raise the defect, while they continue to sweep the area for further issues when they are in the groove
quietly.
- The other teams (for the period of time that the competition is on) are your enemies
- They can, and will, steal your intel if you shout it out (you might not be able to exploit the intel, but they might)
- They can prioritise, if they know you are working in an area of the app, they could hit it hard and fast to find low hanging fruit that you waste time raising when they have already raised it.
- they could steal your defects if you are slow to raise them, but discuss them loudly and carelessly
- “loose lips, sink ships”

Other tips

Depends on the rules for the bug bash but if the bug bash has a rule that “First to find the defect claims the defect, and duplicates don’t count”

Get efficient in raising defects

This means you need to be efficient in raising defects. I recommend copy and pasting into a defect tracking system rather than writing it directly into the browser. Why?

defect tracking systems can crash
create your own ’template’ to copy and paste commons stuff e.g. priority, environment, version into mandatory fields with no default
you have a local record that you can scan without having to use their system
if it crashes or messes up when you raise the defect you just create a new defect and paste from your notes rather than typing again

Low hanging fruit

Use low hanging fruit to demoralise the enemy.

Early on in the competition, have at least one person monitoring your parallel tools and report spelling errors or broken links and images. Raise these into the bug tracking system quickly and make a big deal of it. Then it looks like your team is raising Bug, Bug, Bug after Bug. This can panic the other teams into losing site of their strategy and following you into the areas you have already cleared.

When you do raise bugs, don’t announce the content, just celebrate loudly the raising. You might get lucky and the other team might waste time going into the defect system to read your reports.

If you are lucky then periodically the organisers will announce how many bugs have been raised, and if you have dominated the bug count, even with low severity issues, you can gain a psychological edge on your opponents, aided for free by the Progress Propaganda Department of the organisers.

Summary

A public bug bash is not your normal testing exercise. Particularly if you want to win.

target bugs, pile ’em high, and pile ’em fast
get ready before you go, install what you think you’ll need, make sure you know how to use it, and make sure it all works before you go
harness parallelism
demoralise your opponents by announcing defects
organise as a team
Hit the system hard, hit it fast
Report quickly and well

This post is in response to @ncbaral’s twitter request for “have you got any #protip” for Bug Bash attendance.