Back in 2001? 2002? Back whenever I noticed the ISEB certification starting, I thought “Hmm… how strange, I wonder why they would want to do this”.
I read an early draft of the syllabus online and thought “Well this seems fairly simple, but misses out a lot of stuff that I do in the real world, but what harm could it cause?”
After the unleashing of the foundation certification, and training and examining began in earnest, I remember reading a trade press advert that offered a “High paid career in testing after one week of training and gaining the ISEB certification” (or words to that effect). I still had enough idealism to feel personally affronted and feeling that ‘what I did’ had some how become demeaned.
In that moment I vowed to ‘Save the world!’, or at least the small part of it relating to Software Testing Certification.
Ultimately I failed. I relate my story here.
…But lords and knights and other noble and worthy men that con Latin but little, and have been beyond the sea, know and understand, if I say truth or no, and if I err in devising, for forgetting or else, that they may redress it and amend it. For things passed out of long time from a man’s mind or from his sight, turn soon into forgetting; because that mind of man ne may not be comprehended ne withholden, for the frailty of mankind.
The Travels of Sir John Mandeville
Idealism
I had vowed (and failed) to ‘Save the world!’ on a number of occasions before:
- when I built my first CASE tool (a JSP diagrammer with a built in COBOL interpreter),
- I built an ER diagram of ‘Testing’ (to understand and finally get a handle on Software Testing),
- when I built a configurable entity management system (to model any testing process).
- when I generated test scripts from graph models
Introspection: Perhaps, this idealism and failed attempts at Super-heroism led to the development of the lampooning Evil Tester comic character?
At any rate - I resolved to engage in the certification process and change it for the better.
Infiltration
At one of the BCS Sigist meetings (Special Interest Group In Software Testing), a call went out for people to help with the examination process (writing questions, marking, etc.). My chance to infiltrate had appeared, so I applied…
…Only to receive a rejection on the basis of not having passed the foundation certificate.
You see I had let the certification process proceed apace such that ISEB had added an additional layer of certification, called the ‘Practitioner’. And now to receive consideration of competence to allow me to mark and write questions for said ‘Practitioner’ exam, I had to have made the grade of ‘foundation’.
So I duly went out to pass the ISEB Foundation. Eschewing any form of training for such a simple and easy test. I downloaded the syllabus and began preparing for a public sitting of the exam. (see my notes on how to prepare)
Sadly this was not my only motivation. The ISEB Certification had already taken hold on the UK Job market and I had already had my CV turned away from two job applications because I did not hold the certification. So driven by the ‘fear’ of never working again, I sullied myself, but justified my actions on the basis of ‘Save the world!’.
Making the grade
I found sitting the exam a strange experience. The courses that people sit through in preparation for the exam must train them in such a black and white manner, that some of the ambiguous and vague questions seem obvious to them.
But for me, trained in the real world, having had my black and white orientation knocked out of me, I wanted to answer with a “well, it depends, do you do X or …?”. And that led me to choosing the wrong answer on at least one occasion because, and I confess this with a shamefully lowered head, I did not achieve 100% on the ‘foundation’.
But still… I made it!
I passed!
I achieved certification!
I can describe myself as a tester!
I can now wave my bit of paper and say… “I, am a tester!” (I violated the General Semantics ‘is’ of identify for dramatic effect)
“Nah nah, ne nah nah! I got a bit of paper, you don’t. I got a bit of paper, you don’t.” Cough, back to the seriousness.
A second Infiltration
Armed with my certification, I wrote a second letter requesting admittance to the hallowed realms of ISEB examination… and received an acceptance in return.
The ISEB certification community contains many ‘big hitters’ from the UK Software Testing world so it acted as a surprisingly useful networking opportunity. Although this post may result in the ’throwing away’ of all those hard won networking moments, who knows?
I made a point of attending the exam panel meetings as much as possible. While I could have involved myself more remotely, I thought that I would stand a better chance of ‘changing’ things if I actually attended the meetings in person.
I had never written or marked any exam papers before so I expected to receive a little tutelage to prepare me for the coming travails. But as with many things, I learned by doing, and I learned by making mistakes.
On duty my vision dissipates
My duties included at least elements:
- I wrote exam questions - foundation and practitioner
- I marked practitioner exam papers
- I reviewed questions
All the judgments that we took as an exam panel used the mighty syllabus, as a constraining tool (to not go beyond it), and a driving tool (to cover it). That sanctified document which, if ’twere not in, then we could not ask thereon.
Changing the syllabus did not happen lightly and I do not remember a syllabus change driven by the exam panel. All changes occurred when the ‘higher’ panel decided. For two panels did exist - one which did the marking and examining, and one which engaged in more lofty pursuits; deciding what teaching should occur and who should teach it.
Very quickly my idealised vision of change dissipated. And I got down to the business of writing questions and marking exam papers.
On the art of writing exam questions
I did not master the art of writing multiple choice questions. They either read too easily, making the answer obvious to even the most unschooled and inexperienced candidate. Or required such a long description of the context underpinning the scenario I wanted the candidate to unambiguously address with a multiple choice answer.
I quickly gave up writing multiple choice questions.
I did not consider my attempts at writing the longer, essay based, ‘Practitioner’ questions a greater success. These questions took a long time to write, for a number of reasons:
- Trawling the syllabus to find areas of sufficient complexity to challenge the candidates thinking process
- Constraining the question to the limits of the syllabus
- Writing the question, then writing an example answer, and iterating around that process took a long time
The shortage of questions resided solely in the Practitioner exam pool. So every question brought forward got bashed into shaped, tweaked and tucked by the panel - unless withdrawn by the writer.
I think I remember the butchering of one of my questions where the judgement passed on it meant that we combined it with a subquestion on the Belbin personality types - which personally, I abhor its inclusion in the syllabus. The inclusion of Belbin represented a far worse sin than the exclusion of ’exploratory testing’ or ‘model based testing’.
While as, an exam panel we did our best to understand the effect the question would have on the candidates, you only saw the results when the candidates tried to answer it.
On the decipherment of chicken scrawl
Candidates please… use your best writing. But sadly time pressure, and all that, typically means that the best analogy for reading of an exam answer paper became the extraction of meaning from a paper which chickens had run over with ink covered feet.
Marking these papers took a long time. Typically two hours (sometimes more) per paper. Then we met with our exam marking partner - for each paper had two markers. To examine variance between our grading. If the variance in question seemed high then we conferred to see what one marker had seen that the other had not. For example, had one marker managed to work out why the candidate kept writing the word ‘chicken’ in their answer, what did they really mean?
We tried to mark fairly and consistently, using the marking guidelines, and with our knowledge of the syllabus. This did result in disappointments where people who you felt should not have passed, did manage to pass.
At the end of the whole process we collated the marks together to see if we needed to apply weightings to the final marks. For example if ’everyone’ failed a question and we deemed that question at fault then we might apply weightings to ‘be fair’ to the candidates.
Until I learn to accept my reward
It seemed that the only ‘change’ I could effect involved:
- commenting on questions that did not ‘stress’ the candidate well - although always constrained by the syllabus
- writing questions to test the candidate effectively - but again constrained by the syllabus
At one point, after a long marking session I examined my vision statement.
- Had I made any inroads into my goal of “Save the World!” - answer no.
- Had I at least received compensation for my actions - answer no. We could not consider the monetary remuneration for all this work as ‘worth it’
- Instead, I helped to perpetuate a system I did not believe in, nor did I see it changing for the better.
- Instead the trainers and ISEB received the monetary rewards while I had managed to voluntarily enslave myself for their benefit, to the detriment of my own time.
I made the decision to leave - I forget exactly when, I think I engaged in this process for about two years.
A Few Lessons Learned
Simple lessons. Probably already learned. But obviously easy for me to forget.
- Do not perpetuate systems that you disagree with
- I don’t know how to stop this certification industry, but I do know that I can starve it of my involvement and my support.
- Writing exam questions requires experience and training
- Part of an effective certification process must involve effective questioning and evaluation. I have my doubts as to the degree that the process, as it existed when I helped with it, met that requirement. Given that we received no training in, or had no experience of, writing or marking questions.
- The certifications do not represent an assessment of ability
- I read many exam papers. Very few of these ‘Practitioners’, exhibited in written form, a degree of knowledge sufficient that I would want to employ them.
- Having this stuff on your CV really does help
- Having ‘past member of ISEB exam panel’ on your CV helps. If you support the certification process then I encourage you to join the exam panel. That way you help to perpetuate a system that you support and you do your CV some favours.
- My hidden agenda in the encouragement: I hope that you experience a growing unease and distaste for the process the longer that you involve yourself in it.
- Keep your vision in mind and track progress regularly
- In the heat of battle, you can forget the ultimate strategy all too easily so keep this in mind and track your progress towards it.
- Always charge your full complement of expenses
- Since I didn’t join the panel for the money, I sometimes didn’t claim all the expenses or follow up demanding payment of all the money after writing a question. (At least this allows me to not feel too bad about taking their money - I didn’t take enough of it)
Postscript
And thus, taking solace in my wretched rest, recording the time passed, I have fulfilled these things, and put them written in this book, as it would come into my mind…
The Travels of Sir John Mandeville
My involvement summarised, to the best I can remember.
