What could possibly go wrong? And what to do about it?

TLDR; We test based on risk. If we don’t identify risk we don’t test for it. Automated tools can reveal risk that our technical knowledge can not.

True story:

A timeline of a true story of software development:

15:00 I get an idea for an ‘app’ so start making notes
15:30 decide it will be faster if I just start coding it
17:30 basic version of “The Evil Tester Sloganizer” done
- it is unstyled, but generates slogans
21:30 add some styling and a ’tweet this’ slogan button, and a hacked up navigation bar
10:30 what the heck, I’ll just release it
10:45 what the heck, I’ll just announce it as beta, because then it doesn’t really matter if it works, right?

Whoa there!

Did you even test it?

What testing did you actually do?

no unit testing
I was hacking out an idea, and I haven’t incorportated JavaScript Unit testing into my personal ‘hacking an idea out’ process (I do with Java, but not yet with JavaScript)
I wrote code that tested it on each release… kinda
- I added some code at the bottom of the page to generate 100 slogans into the console each time it loads
- because sophisticated test harnesses are us

 // only for testing
 for(var x=0;x<100;x++){
     console.log("-" + random_sentence());
 }

I had the ‘app’ running in Chrome the whole time
I checked for ‘responsiveness’ using dev tools and resizing the browser
I even ran it on different browsers: Firefox, IE, Edge (admittedly I did this after releasing to live and after tweeting of the apps existence)

But hey, come on. Look at the app ("The Evil Tester Sloganizer"), it was just a ‘fun thing’.

Look at the app from a technical risk perspective

Yes. Let’s look at the app.

So what could go wrong?

single page app
no external CSS
JavaScript
internal CSS
Dom updating for the slogan

Actually, that’s not completely true.

I added a navigation bar pretty late in development. My EvilTester.com site is generated using Hugo, but I didn’t put this under Hugo generation, I just added the .html file into static and ftp’d the final .html onto the site. So the navigation bar isn’t the real navigation bar. And it meant I had to bring in two external CSS files.

Sigh. OK.

Oh. And there is another div in there that is shown randomly about 4-8% chance of reveal. Shown through DOM manipulation of the style attribute.

OK. Sigh.

single page app
external CSS
JavaScript
internal CSS
Dom updating for the slogan
Dom updating randomly for a link to the book

Obvious risks:

JavaScript might not work cross browser
HTML might not work
CSS might not render properly

And what happens if the risks manifest?

Impact of JavaScript not working

The slogan might not appear
The tweet link might not work

Wait a minute. You didn’t say anything about the Tweet link earlier.

Eh yeah. There is a ’tweet this’ button and I amend the href on that on every click

Sigh. Ok.

So stuff might not work.

What did you do to minimise that risk?

used pretty simple JavaScript for DOM manipulation

innerHTML seems the most risky. I know we are supposed to use innerText on some browsers, but I’ll add some wrapper code that detects which method the element supports later.

TODO: Amend code to mitigate against innerHTML not working

What Else?

used pretty simple JavaScript for string manipulation and generation
- basic core Javascript: functions, arrays, hashes, strings, Math (floor and random)
- I didn’t even use trim() because some browsers don’t support it

OK. Risk that custom string formatting code doesn’t work.

I didn’t say that, I said I didn’t use trim

And did you unit test your custom string formatting code?

OK. Risk that custom string formatting code doesn’t work.

TODO: Create JavaScript Unit tests for string formatting and generation

What else?

I added calls to the methods directly in the onclick attribute to minimise chance of hooks not hooking. So really if a click triggers on one browser, because its standard, it should trigger on them all. Really. Right?

Anything else?

Oh
- I used encodeURI to generate the url for ’tweet this’
- well, mostly encodeURI
  - encodeURI didn’t seem to encode the # in #EvilTesterSloganizer properly
  - I found that in my testing 🙂
  - so I added some string concatenation

Unit Tested?

OK.

TODO: Create Unit tests for tweet this href generation

Anything Else?

Impact of HTML not working

If the HTML doesn’t work properly then the page might not render.

come on, look at it! It is so simple.

<h1 class="title">The Evil Tester Sloganizer</h1>

<div class="title">
    <button class="title" onclick="changeSlogan()"><h2>I Want A Slogan</h2></button>
</div>

<p class="slogan"><span id="slogan">How can I help you?</span></p>

<div class="tweetthis">
    <a id="tweetthis" class="twitter-share-button"
       href="https://twitter.com/intent/tweet?text=Evil%20Tester%20Sloganizer">
        <button>Tweet This</button></a>
</div>

<div id="bookinfo" class="book" style="display:none">
    <h3>If you like this, you'll probably enjoy the book 
     "<a href='/page/dearEvilTester/'>
      Dear Evil Tester
     </a>"
    </h3>
</div>

It does look simple.
Did it work?

erm
almost?

@eviltester Is there a label missing from that button? pic.twitter.com/IkvnMf7J5d

Danny Dainton (@DannyDainton) September 8, 2016

@eviltester @DannyDainton Looks like an issue with certain iDevices. https://t.co/HTOuYI5r3E

Neil Studd (@neilstudd) September 8, 2016

OK.
- When Danny tried it on his device the “I want a Slogan” button might not have rendered properly, but it did click and generate the slogan, so that’s good right?
- And when Neil ran it through BrowserStack Screenshots he might have demonstrated that there were rendering issues on some iOS devices.
But I fixed that.

@neilstudd @DannyDainton Cool. Seems they don't like an `h2` inside a `button` I have amended my html to use more css and less html

Alan Richardson (@eviltester) September 8, 2016

What did you do?

Well, I assumed that it was the h2 inside the button so I took out the h2 from the button and styled with CSS instead
And when I ran it through BrowserStack Screenshots I couldn’t see the problem anymore

And why did you add the h2 in the first place?

Well, to make the text bigger

Was it a heading?

Well. No. But it makes the text bigger.

Sigh.

Lesson Learned: Use HTML for structure and layout. Use CSS for style
TODO: Go through every HTML I have ever written and apply this rule

And you didn’t find this in your testing?

No. This was device specific. I viewed the site responsively on a desktop browser.
I thought the HTML was so low risk that nothing would go wrong there.

So what could we do in the future?

Test the page on every physical device?

That’s an option, but sounds expensive. What else?

Use BrowserStack Screenshots

That’s an option. But it only does screenshots, what if we want the page/app in a specific state and view that state.

Make the app testable so that the URL puts it into a state?

Or a test harness that renders the app in that state using the same GUI code, remember we are only testing the GUI rendering by looking at screenshots, not the functionality, we don’t need the full app.

(Note: I was working just last a few weeks ago with a client where we did exactly this - created a harness, that rendered the GUI, using the ‘real’ code, but injected data and state, to allow responsive testing to be performed for visual rendering and support reviewing the rendering for issues via screenshots.)

TODO: Make ’testable’ by controlling state through URL params

Any other options?

Would an HTML validator pick this up?

Did you try?

No
Erm. Yes. I have now. And the answer is… yes
https://validator.w3.org
- reported an error that “Element h2 not allowed as child of element button in this context”

OK. That’s good.

But it also said “Error: The element button must not appear as a descendant of the a element.”

OK. That might be bad then?
What are you going to do? Remember it is really a link. It doesn’t really ‘do’ something.

I’m going to make it a link?

Good.

OK. Done.

Great.

And I was able to use target="_blank" to open in a new window so we got extra functionality for no cost

And you added the rel tag to prevent the opener.location exploit?

What?
I’ve been using target="_blank" for years. It just works.

OK. Have a read of:

Did you read it?

Yes.
target="_blank" rel="noopener noreferrer"

Great.

So rather than ’testing’ we can pick some of these errors up with a linter or a validator prior to testing.

TODO - validate all my pages and apps against https://validator.w3.org
TODO - find other validation software and linters

Impact of CSS not rendering properly

And if the CSS doesn’t work?

well, it might look worse than it already does.
but since we now have standard validated HTML, it should render. It might just look a bit more bland.
I don’t think we have any particularly complicated CSS so I think the risk of functional issues is low, it will just look a bit rubbish

Summary & Lessons Learned

We decide on the testing that we perform based on the risk we identify and accept.

My Risk Analysis was wrong

I thought it was a pretty low risk app, with very simple JavaScript and HTML.
I thought releasing in ‘beta’ was a viable (I don’t have to do as much work) strategy

As it turned out my HTML choices were high risk and triggered a cross platform issue, but I didn’t have enough technical knowledge of HTML to recognise that. Perhaps a more technically focused HTML developer would have seen that risk earlier and not have written code like that anyway.

If I relied on testing I could only find on real devices. Fortunately running this on a single iOS device would have found this - as Neil demonstrated with his use of BrowserStack Screenshots.

Even though I did a ’tiny’ amount of responsive manipulation I would never have found this through testing unless I used a real device.

But ’testing’ isn’t always the best way to find issues.

I didn’t use any validation tools

Make use of validation tools for automated feedback prior to testing.

In this case the W3 Validator would have provided actionable feedback. And now I believe that “Error” in the validator means - “some device is going to render this badly” I will act on errors (and probably warnings).

Validation tools can provide technical expertise that we might not have. and can point out risks that we have not identified.

Use of validation tools doesn’t mean that something won’t go wrong on devices you haven’t tested on, but it reduces the risk and you have less ’technical’ reason to test cross platform.

Use technical knowledge to identify risks and minimise its chance of manifestation

I did accept some risk

The lack of unit testing makes me uneasy about some of the JavaScript - I mitigated that with ’lots of’ execution
I did jump back and forwards between code and rendering when assessing the HTML, CSS and JavaScript
I didn’t embed it in the site properly using Hugo - so that is still a risk

Be aware of the risks that you are accepting

I did not anticipate mobile issues so quickly

Because I hardly ever use Twitter on mobile. It never occurred to me that people following the link on mobile was a high risk.

But I’ve seen that before.

I should assume that pretty much everything linked to on twitter will be viewed on mobile.

I’ve been improving my web sites to make them more responsive because of this. And I’ve been extending my ‘responsive’ testing repertoire.

But ‘responsive’ doesn’t mean ’test cross device’, it means respond to certain attributes reported by the device, and you can simulate these attributes.

I need to derisk ‘cross device’ at a deeper technical level than I already do. Because this wasn’t a ‘responsive’ issue. This was an HTML issue.

While I am planning on putting this app on mobile, and I build it with that use case in mind. I didn’t think we would hit it so quickly.

There is a risk that we will miss risks during our Risk analysis

We choose to go live

I chose to go live, knowing I had risk. And knowing I had work still to do.

I didn’t choose to go live because:

testing was finished
it was ‘done’ ‘done’

I chose to go live because I thought it was ‘good enough’ and ’low risk enough’ and went with a ‘beta’ tag because I hoped for early feedback if something was wrong.

I chose to go live.

Projects choose to go live, unless they automate the ‘go live’ decisions as part of their automated workflow.
Thanks Neil and Danny for providing the early feedback on the app that led to this retrospective.