On the Black Ops Testing Webinar of 11th May 2015 we tested QA Mail. You can find a write up and watch the webinar replay over on BlackOpsTesting.com
This post, expands a little on how I approached the testing and what I learned.
A long time ago, on a job far away, my team and I had to test website registrations and emails on various account events.
For our interactive testing this was fairly simple, we could use our company accounts or gmail accounts and then reset the environment and then we could re-register again. For the automation we needed a different solution. We didn’t want to use gmail ‘dot’ and ‘plus’ accounts because we felt that the number of emails might put us against the gmail terms and conditions.
We started using mailinator.com creating adhoc email addresses for the automation, and I created some abstractions to read and delete the emails. But mailinator introduced a captcha part way through our testing and our automation failed.
I created a fairly simple appengine tool which acted as an email sink, and I wrapped a simple API around it for our automation. AppEngine has changed so much since then that the code I wrote will no longer work. But QA Mail does a very similar job to the AppEngine code that I wrote.
It provides a simple GUI and simple API wrapper around a polling daemon which reads a mailbox folder and brings in emails.
Great.
I approached the Black Ops Testing Webinar as a learning experience.
- I didn’t know too much about SMTP or email
- I wanted to experiment with API Automation from a zero to ‘working’ state as fast as possible
- I wanted to experiment with sending emails from Java
- I wanted to know what tool support I would need for interrogating and comparing emails
Automation Abstractions
I started with the automation. And first off wanted to de-risk it by making sure I could send emails.
I had a quick try of the Javax mail libraries, and quickly decided to find an abstraction library to cut down on the time I required to get up to speed and sending emails fast.
I started using Simple Java Mail https://github.com/bbottema/simple-java-mail
With a few basic emails sent, I started to work on the API abstractions for QA Mail. You can see the various twists and turns I took via the history on github
https://github.com/eviltester/qamail_automation
I created abstractions at a few different levels:
- A QA Mail Rest API Call abstraction
- A QA Mail Domain abstraction
- A Mailbox abstraction
These all work at a similar level so they overlap a little.
This allowed me to create basic automation fairly simply.
They lack a way of conducting automation to twist the API calls i.e.
- make the REST call with a POST instead of a GET
- add null parameters
- use badly named params
- add new params into the calls
- re-order the params in the calls
- etc.
I could achieve the above with direct calls using RestAssured, but since they are fairly common requirements when testing an API, I need to identify a different way of building abstraction layers which support ’testing’ and not just ’exercising’ the API.
In this particular instance I didn’t need that flexibility since the API would have thrown 500 errors on invalid calls. QA Mail was written to meet the needs of a single user and graciously released to open source in the event that it might help others.
I did use the API abstractions to test for different combinations of routings e.g. no ’to’, multiple cc, and bcc etc.
Unfortunately we were using an older version of the app which did have a bug in this area, so I didn’t pursue this line of enquiry long. The bug has been fixed in the main code branch and on the QA Mail demo server.
Testing Emails
After sending a few emails, it became quickly apparent that I didn’t really know what I was looking at in the raw email view.
I’m used to looking at HTTP headers, I’m not used to looking at email headers.
Was the email I was seeing correctly rendered?
How could I know?
A lack of oracles for domains we don’t know well can make testing harder in the initial stages of building domain knowledge. One strategy I use involves me finding alternative sources of rendering the information via complementary or competing renderers.
In this instance I used similar tools: mailinator and temp-mail.
Both of these accept emails to anonymous mailboxes and render the email as raw text so you can see the headers.
I saved these as text files and compared the output through winmerge.
I found differences in the headers and had to go look them up to try and understand them. Oft times, many of the headers are actually added by the routing servers the mail winds its way through, so what I’m seeing is not what I actually created as the email.
So I needed to find a way to observe the emails I was sending out in the first place.
For the automation, I found a debug flag on Simple Java Mail which output the raw email smtp session to stdout so I can see the original message, headers and encoding. I was then able to compare this to the output and see what the routing had added, and what might have been added by QA Mail. In the final analysis, nothing was added by QA Mail, it simply sucks out the message after it has flowed through postfix.
For my interactive testing, I discovered the ‘Show Original’ menu item in gmail. This lets me see the ‘raw’ email sent to me, and which I’m sending out.
Very handy - I’ve actually become addicted to looking at email headers now, and most of the emails I receive I check the headers. I find it amazing how much information is contained in these about the machines that the email has passed through in its travels. I encourage you to have a look for yourself. Never again will I send an email direct from machine - I’ll always try and use a server based tool to avoid giving away my machine ip addresses.
Observing the System Under Test
One of the interesting challenges I faced testing this was isolating where issues were introduced.
QA Mail runs on a server, so I can only observe it via logs.
It provides one log by default the log for the import daemon that polls the mailbox folder.
I ssh’d into the test environment and could ’tail -f’ the log file.
This way, when I started testing different email address formats I could see if they were actually hitting the system.
I found that many were not. valid emails such as 1234-@ and
()<>[]:,;@\"!#$%&'*+-/=?^_`{}| ~.a
were not reaching the mailbox that QA Mail polled. They were being rejected by postfix, making it impossible for me to test how QA Mail would handle extreme email formats.
Identifying where processing occurs in an end to end system is a common challenge and one we should be aware of when testing. So I recommend trying to understand the architecture of the application under test and trying to add observation points in as many positions in the chain as you can.
Summary
Normally when testing emails, I’ve been more focused on:
- was an email sent?
- Did the email render correctly?
When testing QA Mail I had to focus on:
- Was it pulling in the appropriate information from the mailbox?
- Was it routing the emails to the appropriate database tables?
And this forced me to consider new ways of observing and interacting with the system.
- You can read my raw testing notes over on the Black Ops Testing site. Also a write up that includes information from Steve, Tony and James.
- You can see the code I wrote on github.
- Also - watch the webinar replay via the Black Ops Testing video page
