This page is the supporting page for the talk that I gave at EuroStar 2003 on becoming a better tester, specifically by beta testing software.
You can
- download the paper, or the presentation.
- Find links to all the tools and web sites mentioned.
- Read Additional Points and information not in the paper.
The Paper and Presentation
[Paper .pdf (975kb)] 
[presentation .pdf (227kb)]
Additional Points
Some of these points have come out from comments made by the reviewers of the paper (thanks: James Lyndsay, Robert Sabourin, James Bach) and through my continued use of beta testing and thinking about beta testing since the paper was written.
I haven’t included everything that the reviewers pointed out, and I am responsible for any and all of the mistakes, inaccuracies and inadequacies of the paper, but I did so want to thank them for their comments.
They probably make more sense if you have read the paper or attended the talk first, but feel free to browse them if you haven’t.
Only Technique Sessions?
Am I saying that when you practise you should only practise techniques?
Heck No (see below), the paper does use Technique Session as an example though, and it does have a list of possible practise sessions (under the title ‘How to Practise Techniques’!). Here are some other sessions that you might want to do:
- Relax: Have a stress free time learning the software. Learning is good.
- Enjoy: What can you do to have more fun with the software before you? Have you ever seen a child break a toy through exuberant play? Well then…
- Become Aware: As you are testing, what are you noticing? What are you not noticing? What are you not paying attention to? Ask yourself those questions in a session and see what happens as you do.
- Look Inside: Pick a testing book, any testing book. Pick a Page, any page. Pick a line, any line. (Stichomancy) How does what it says relate to the software now?
Maturity
It seems to me that some of the mature processes that I have seen are judged more mature because they are more bureaucratic. Maturity when we apply it to humans is about learning, specifically learning about relationships, about evidencing that learning and about experience. Practising testing is a good way to become a mature tester.
Technique Sessions
One thing I have found about ’technique sessions’.is that these are among the least effective (in terms of bug finding) sessions that you will do.
Technique Session: Sessions where you set out to practise a technique exclusively, and you try to use that technique on every applicable area of the software in order to learn the technique, and learn where the technique is not applicable.
i.e. You are likely to find more bugs in a short period of time by using your exploratory testing powers than you are by applying a specific technique exclusively.
This doesn’t negate the practising of techniques, in fact, to me, it means quite the opposite.
What I think is going on is that when you are practising the technique, you are building up a model of the techniques applicability, usefulness, and how to actually apply the technique. When you are testing exploratorily, you are taking all the information you have learned (all those technique and experiental models) and applying them to the context model that you build up by observing the system before you.
It is kind of like improvising on the guitar. When playing guitar we learn our scales and chords and we practise them, listening to them in different keys so that when we improvise, we can take the knowledge that we gained and apply it to the situation we are improvising in.
A Good Beta Test Bug Report
One thing that all the reviewers picked up on, and this is a good thing. Is for more practical examples. I could make the excuse that that would make the paper even bigger! Or I could include them here.
But before I do that, what is a good bug report?
In the context set out by the paper it is a bug report that:
- is focussed,
- is useful.
- is quick to read,
- is relevant to the person reading it
- allows the reader to make the correct decision about it (fix/ignore/defer)
- makes a really bad defect stand out (even if we don’t realise that it is really bad)
- and makes a pathetic winey little defect look that way (even if we think it is a showstopper)
And with that in mind, the challenge was do I actually write good bug reports. Well I had a look.
It was interesting for me to look back and see just how many things I’ve beta tested and the range of software.
- Graphics Software: I would never class myself as knowledgable on Graphics algorithms, but I can say “oooh”, “pretty”, and “ugh”.
- Zip Utilities
- File System Utilities
- Diagrammers
- Games
- Email Utils
- XML Utils
- And Misc. Strange one off utilities that are hard to categorise - and interestingly this makes them harder to sell. Something to consider if you are planning on writing any software.
And I don’t limit myself to software. I’ve tested books and teaching cds too. Why not? Testers read documentation and we get taught stuff.
