Lessons Learned Testing the House of Test contest app

Lessons Learned Testing the House of Test contest app

Martin Nilsson and House of Test wrote a small ’test tool’ which they released as a competition app. I’m sure they’ll release the app later.

This is a mix of ’testing notes’ and ’thoughts on how to test’ this type of app. I spent about an hour or so looking at the app, so the notes are a little scrappy. I spent about 40 mins or so tidying them up for publication, in the hope that they help someone.

GO!

Martin said:

“A small challenge for all you testers out there: Test the Crappy Little DataGenerator: https://t.co/6vqix2AgZv”

A small challenge for all you testers out there: Test the Crappy Little DataGenerator: https://t.co/6vqix2AgZv

— Martin Nilsson (@MartinNilsson8) February 22, 2016

Quickstart

• Download .jar
• run .jar
• see/use GUI

Obvious errors:

• truncation of text
• spelling errors in tooltips
• spelling errors in options
  • e.g. “Captial Ch…”

I could at this point, resize, hover for tool tip text etc. And I could test it blind, but what if I get more information first?

How can I observe more about what this app is doing?

Has logging built in.

Try that first.

Hint: Always identify how to use the thing to test the thing

Turn on logging. Do something. Hunt for file. Look in running directory first. Found File.

• tail file

tail -f DataGenerationLog.txt

Generate “Captial”

• I see chars
• is that random enough? don’t know - how could I test that from the GUI?
• is auto copy into clipboard same as copy paste? (same length)

Generate Small Chars

• I see chars
• is that random enough? don’t know
• is it 200? need to copy and check (into my Test Tool Hub , is 200)

International C…

• no generation
• no log output
• Obvious bug

I could carry on like this, but I really want to be able to interrogate and manipulate the system effectively.

Interrogate and Manipulate by working with the source

I really want to work with the source.

Maria said:

“Give me the source code and I can assure you I’ll break it! "

Give me the source code and I can assure you I'll break it!
Seriously, testers don't miss this cool challenge! https://t.co/eMe1nLSYrE

Maria Kedemo (@mariakedemo) February 22, 2016

But we don’t have to wait for the code. We already have the code.

Hint: Most Apps come with the source. It is hard to hide the source if it is on your machine.

• Web Client - the source is in your browser. We need to learn to read it and debug it using the built in browser tools.
• Java applications are pretty easy to decompile. There are tools to do this e.g. jd.benow.ca (But I use IntelliJ, as you’ll see later)
• Ruby executables often are a self extracting zip with a ruby interpreter that runs source from a temporary directory - use a file monitor and you’ll probably find where it runs the source from.

Basically it is very hard to protect your source. Assume you always have it. Assume your users and hackers always have it.

It has ever been thus. Back in the olden days we used disassemblers and monitors with debuggers to gain access to the assembly version. Now it is just easier.

If you want to protect your code then you run it on a server. And have a client server system.

To look at the source

• Open IntelliJ
• Create blank maven project.
• Add crappy_little_datagenerator_v_1.0.jar to project as a dependency using the IDE.

OK,

Now we can really see what this is about.

I can see there is a GUI by default, but also a command line tool version.

I can run that from the command line. We can have multiple main methods in a jar file. The manifest file controls which runs by default. But we can start any one of them we want.

java -cp crappy_little_datagenerator_v_1.0.jar yhtest.tool.controller.TestTool

There may well be more bugs in this interface, since it is not the one promoted but, it might make it easier to automate for someone who does not know how to code, but knows how to use the command line well.

But, we won’t do that.

We’ll work from the IDE.

Because in the IDE I can write @Test methods

I can write unit tests to explore the Data Generator library e.g.

public class QuickTest {

    @Test
    public void canIRunDataGeneratorHere() throws IOException {

        DataGeneration td = new DataGeneration();

        Assert.assertEquals(3, td.generateCounterString(3).length());

    }
}

I now have a way of checking if the data generation is statistically valid. I could call the data generator say 10000 times, and count each occurrence of the characters and see if they level out. If not then there might be something wrong with the generation code.

Note that:

• we could view this as unit testing
• or treating the app as a ’library'
• or view the ‘app’ as a client server with the TestDataGUI as the client and the DataGeneration class as the server.

I often do this with web applications. There are the simple models of “client -> http -> server”, and we use a proxy to mess with the messages. Or send messages direct to the server. This is typically done with an API.

But I also use the concept of App as API. Where I send through the HTTP form messages that a GUI would POST. As though it were an API, even if there is no API. And simulate GUI testing without the GUI.

Even though I’m targeting the DataGeneration class, I’m not thinking of Unit testing, I’m thinking of testing the server without going through the GUI.

This means that I will jump back and forwards between @Test code, and the GUI while I’m testing.

If I was just doing Unit testing then I probably would not have the app running at the same time.

It might look superficially like I’m Unit testing, but the thought processes are different - try it and see.

Now we know we have all the tools we need to test this thing properly, and observe how it works.

Why not just decompile

I could have just decompiled the app e.g.

Bill Mathews said:

“just download JD-GUI and decompile the jar you can see the junit ’tests’ they ran”

@mariakedemo just download JD-GUI and decompile the jar you can see the junction 'tests' they ran @houseoftest @MartinNilsson8

Bill Matthews (@Bill_Matthews) February 22, 2016

A few people on twitter mentioned this when the competition was running.

I have decompiled the app, but since I have it as a .jar in my project, the decompiled version is essentially a source view so I can do more than if I had simply decompiled it.

From the IDE I can now:

• run the gui
• debug the gui so we can breakpoint it and fiddle etc.
• run the CLI
• debug the cli so we can breakpoint it and fiddle etc.
• breakpoint and evaluate code while running
• write @Test methods to explore functionality and test it

A quick investigation to reveal ’tester thoughts’

But, what we’ll do is quickly investigate the “International C…” error we found earlier.

A quick code read and we can see:

String internationalChars = "";

Now we could have seen that in the decompiled code as well.

While we are looking at the DataGeneration code we can see that lowercase chars set will not generate a statistically valid set of chars since there are multiple ‘o’s. This bug would have been very hard to find from the GUI.

I might have been able to find it from the CLI had I scripted something to analyse all the output.

But sometimes a code review will reveal things. Certainly I can scan the code now and find all the spelling errors quite easily without having to scan the app.

(if you are working on windows then there are usually tools to help you analyse the resource files (do windows apps still have those?)) (Or you can hex dump the binary and you’ll often find the strings tucked away at the end of the file.)

String lowercaseCharacters = "abcdefghijklmnoopqrstuvwxyz";

Now I can see what the special chars and password special chars and email friendly chars are etc.

Looking at this list of chars I can then go and search the web to find out the rules for ’email friendly’ and compare them to this set of chars and see if any are missing, or we have any extra.

I could probably do the same for special chars.

I don’t know what password Special really means, so I assume that is a ‘valid’ set.

Since I can see more than 26 chars in the lower case set, I have to wonder if the random generation routine uses the full length of the string, or might there be an off by one error?

A quick code review later and …

I get distracted immediately because I can see that generateRandomString uses a contains to decide what string set to use, and concatenates them together.

Which means that I wonder if all string flags are passed in by the GUI, interestingly I can see that there is an ’email friendly’ flat in datagenerator that is not used by the GUI. Not a bug, but I can see that the GUI uses a generateEmail method, which doesn’t use the email friendly string as part of its generation, it just uses ’l’ which is lowercase - that is an interesting choice and means that the email generation doesn’t create random emails that stress the email validation rules.

OK, get back on track.

The strings of ‘valid’ data to choose from for each of the categories are concatenated together. This will also skew the statistical ‘randomness’ of the generated output. Perhaps that was intended?

The loops all use pre increment, which confuses my code reading since I’m really not used to that.

for(int e = 0; e < length; ++e) {
    returnString = returnString + 
	   String.valueOf(
	    stringToChooseFrom.charAt(
		    randomGenerator.nextInt(var10)));
}

Is this doing what it is meant to?

Let me think it through:

• nextInt is 0 -> var10 (not including var10)
• charAt indexes from 0, so would end at upper level
• e=0; while less than length, increment

OK, seems OK.

But I might have wrapped it with a unit test if I couldn’t read it. Something I would be unlikely to do if I had just decompiled it.

And I still might because I haven’t ruled out that I have read it incorrectly, but I’ll do that later.

A lot of string concatenation. I wonder if the counterstring does that or if it uses a string builder?

Note: a side thought idea because I’m in ’tester’ mode and I’m still learning the app.

OK, it uses a string builder in

generateCounterString

Which is good because that is more efficient.

In theory, we know exactly how long the string should be so StringBuilder could be initialised to the length, rather than 100, that would reserve enough memory for the full counterstring at the start of the process and might be faster, it might also trigger an out of memory error early in the generation rather than late.
Hmmm, jumped back to GUI and tried 20000000000 as the counter string, and it dropped to 200.

Look at code to see why. OK, code seems to use Integer, so will be restricted to max length of Integer:

2147483647

Compare with an other oracle

James Bach has a counterstring app. I could compare Martin’s to that. But I have one too, so I’ll compare it to mine, since mine uses a different algorithm to James Bach’s.

Gonna try a quick counter string race with my algo.

Let us try 2000000 - that should be visible time

Mine is about 16 seconds faster.

I’ll do a manual timing, with the apps running individually so they don’t clash.

• Martin’s 17.10
• Mine: 1.44

Martin’s GUI controls render the text better than mine. I’m using JavaFX his is not.

But, interesting speed comparison. I wonder if it is the GUI that slows it?

Tried from the command line and Martin’s without the GUI is comparable speed to mine with GUI, that’s interesting but probably of no value. And his GUI will be released, and mine probably won’t.

There might be more to investigate there but I’ve been at this for about an hour.
OK. Time to finish and summarise some stuff.

1. Looking at the code makes it easier to spot:

• spelling errors
• simple gaps in code coverage

2. Having the option of running the app via the IDE with the decompiled code, and investigating it while running opens up more ways than just the GUI. Which opens up more testing possibilities.
   • I can use @Test methods to explore an app, not just unit test it.

This is a practical approach

I’ve had to take this type of approach on projects before when I was working with applications where, because I’m a tester, I was not allowed to see the code.

Ho Ho.

Some examples:

• A metrics excel spreadsheet that I was sure was wrong, but it was using ‘macros’ and they wouldn’t let me see the code. It was password protected so I couldn’t access it. Did you know that open office would open a password protected spreadsheet without forcing you to use the password? They didn’t. I did.
• Android apps are basically a zip file. Then you can decompile it. I did this to help derisk testing an app, because we were able to see how it worked, and could make a decision on what to test based on the technical risks we identified.
• When I’m faced with a library I don’t know. I’ll do a similar thing. Look at the code. Build simple @Test methods to explore the library. These also act as ‘documentation’ for me later to refer back to. (This is basically the learning approach I recommend in Java For Testers)

This was more of a ‘how to test’ rather than ‘see the testing I did’. But I hope it is useful.

Note also, I mentioned I used my app as an oracle. I wrote a similar app. I recommend everyone who tests and who codes, to wrap up their utility code into an ‘app’. You’ll learn a lot about structuring your code, using a GUI etc. And you’ll have a project you can work on in your spare time that will grow out of control, but prove very handy.

Video

I know that some people will have read, and been able to implement the notes I published after testing the House of Test Data Generation contest application.

I also know that a video can help people overcome issues trying to implement the notes because you’ll be able to see my testing in action and the steps I take to create the project and add the jar to the project etc.

So I have turned the case study notes into a set of free videos.

I have uploaded the first video to YouTube, if you want to watch it prior to signing up to watch the videos for free on the training site.