How to write a keyword driven test framework - an historical perspective

I don’t really like keyword driven frameworks. For a whole bunch of reasons that I won’t go into now, since you came here to learn how to write them.

Well, here’s the secret…

This is old tech. Older than old.

In fact, when I see it being promoted as ‘state of the art’ I’m amazed, and a little disappointed.

The first book I read that relates to this was “Write your own adventure programs for your Microcomputer”, back in 1983, and it teaches the basics of how a keyword driven framework works. I was using a ZX Spectrum 48K at the time. I loved these little Usborne books, they probably set me on the road to programming better than any other resource.

I don’t think I managed to get the listing in the book working, but I did manage to write my own crude adventure game parsers.

In fact, pretty much any tutorial on coding text adventure games will cover the basics.

And this is partly why Keyword driven frameworks exist.

Keyword Driven Frameworks exist because they are easy to write

Keyword Driven Frameworks exist because they are fun and easy to write.

Not because they boost productivity, or are easy to maintain or anything like that. Simply because they are easy to write.

And as much as I denigrate them, if you write your own, then you may well learn lessons about parsing input that will help you improve your test automation abstraction layers in the future, so I shouldn’t be too hard on them.

And yet, I caution you to be cautious.

The basic principles you have to address for Keyword Driven Testing are:

Decide on a vocabulary and phrases
Decide on a parsing mechanism
Decide how to map your custom code to the verbs you parsed

Decide on a vocabulary and phrases

With an old text adventure you used a basic Verb/Noun parser. e.g.

Take Book,
Open Door

We might relate that to:

Open_Browser Firefox
Goto_URL https://www.eviltester.com
Assert_Title “Home | Evil Tester.com”

One of your first tasks is to decide on the Verbs you want to handle in your application and your domain.

Decide on a parsing mechanism

Then you have to decide how you are going to parse them. csv? tab delimited? plain text?

When writing text adventures you have the disadvantage that you have to handle user input, dynamically.

What we did in the old days, with a verb noun parser was split the string looking for the first space. Everything before the space was the verb, everything after the space was the noun.

And that’s pretty much all you do with keyword driven testing. You take the line of input and split it based on whatever delimiter you choose.

This is why a lot of people use Excel format as input, you use a library that reads Excel, and treat each cell as an input. The first cell on the line is the verb, the rest are the nouns or parameters.

You can get more complicated, and indeed, most home grown keyword driven test frameworks seem very crude given the type of parsing we eventually learned how to do when coding adventure games.

But if you want to do this, and you want to get started now, then keep it simple.

You might want to split your input verbs and nouns with tabs, then you can accommodate spaces in words and parameters. Up to you.

Decide how to map your custom code to the verbs you parsed

Assuming you have parsed the input and stored the verbs and nouns in some variables and some sort of collection, then you have to decide what to do with that data and map it on to custom methods.

There are lots of ways of doing this, and your level of programming experience will pretty much dictate the one you choose.

In the game above, they tokenised the input and associated each word with a number, to make it easy to decide what to do. This was also done because of memory constraints. Its actually a pretty good way of working, but most people writing keyword driven frameworks don’t do this, they tend to do something like the following pseudo code:

If uppercase(keyword) = "OPEN_BROWSER" 
     then open_browser(params_array);
If uppercase(keyword) = "ASSERT_TITLE" 
     then assert_title(params_array);

And that’s fine, create a function for each keyword, then call it when you find it.

Job Done

And what do you know. You’ve done it. That’s it.

When writing an adventure game you have to maintain state, create maps, handle lots of objects, track location, handle random events, etc. etc.

But with a keyword driven test framework you just map a keyword to some code you’ve written and voila.

It’s ugly. It’s crude. It’s basic. And it’s still what many people do.

If you want to get ‘fancy’ then you could read a ‘how to write your own interpreter’ or compiler book. And there are loads of those. I’m not up to date on them because I studied this stuff years ago, although one of my favourites [Aho, Sethi and Ullman]’s “Compilers: Principles, Techniques and Tools” is still well regarded

Dragon Book I prefer the Dragons cover, which is also much more fitting with the text adventure theme.

I still have my copy of “The Dragon” book, it seems to be one of the few compiler/interpreter book I kept. (In addition I also have “Introduction to Automata Theory Languages and Computation” by Hopcroft and Ullman, which isn’t quite as simple to pigeon hole )

Why not use a simple and more CV friendly approach?

However, we have learned a lot now about Domain Specific Languages, and writing maintainable abstraction layers. So instead of writing a simple text adventure parser, why not use something like Cucumber.

Cucumber essentially implements the parsing for you, and the execution, and the reporting, and the running, and all you do are implement the methods which map on to the commands, e.g. the ‘open_browser’ method above.

Yes your code and scripts will remain pretty horrible because you’ll have abstracted at a verb keyword level rather than a logical action level, but that’s exactly what you were going to do with a keyword driven approach anyway.

At least this way, you get a head start with a ‘modern’ tool so you don’t have to re-invent the wheel.

And you can gradually refactor your test abstractions to something readable and maintainable if you start to see issues with a keyword driven approach.

This also has the benefit that it is in vogue, and therefore easy for other people to understand as an addition to your CV. No-one really cares if you can do verb/noun parsing anymore, but if you can use Cucumber, they will look more closely at your CV.

Warning: Dragons within

But… here’s The Thing.

I loved writing parsers. I did. I wrote text adventure game parsers, AI parsers, Natural Language parsers, Interpreters and Compilers.

I loved it, they were great fun.

I learned a lot.

And that is part of the danger.

The people writing the framework are going to have a blast.

The people using it, are going to create long unmaintainable scripts that are ugly, confusing and brittle, and all in the name of Keyword driven testing.

So I warn you.

Your techy testers, when just starting out, are going to recommend Keyword Driven approaches because:

The ‘industry’ still loves to promote them
The frameworks are fun and challenging to build from a technical perspective
Your testers don’t need to learn anything technical, just a ‘small’ vocabulary of verbs (or so the promotional material goes)

I caution you. Here be Dragons.

Rather than write your own Keyword parser for production use, consider the Robot Framework instead.

Resources

Usborne Books

I’ve been buying up old Text Adventure game books on ebay and now own most of the Usborne computer books.

You don’t have to since you can download the 80s Usborne computer books from their website for free.

Writing Adventure Games on the Amstrad

I can recommend Simon Price’s book “Writing Adventure Games on the Amstrad” from 1985. Most books of this type have BASIC code listings. This is the first I’ve seen that describes the development and implementation of a Domain Specific Language.

A Brief summary of the Adventure Kernel System DSL is available on Simon’s site people.cs.bris.ac.uk/~ecsnp/games/#aks

Simon has released his book as a free PDF . I think it is worth reading.

When we work with BDD, Cucumber, Keyword Driven Testing, Abstraction Layers, etc. we are building Domain Specific Languages. This book explains and demonstrates some of the concepts behind that approach.

When we conduct exploratory testing, we build models of the application. Sometimes we create diagrams. When we play and write adventure games we do similar things (this book covers ways to do that).

Keyword Driven Testing can be viewed as a Verb/Noun parser and is often implemented as such e.g. String []verb_noun = input.split(" "); The parser in this book shows simple ways to go beyond that.

This book is a really good introduction to the design an implementation of interpreters in general. Basic knowledge which has served me well when writing code to automatically execute other applications.