TLDR; Automating is not limited to “Test Automation”. With the ability to code, we gain the flexibility to approach problems in multiple ways. We can refactor tactical code to become strategic code.
I needed to migrate some data from one system to another. To do it manually would have taken me about 1.5 to 2 days. I automated tactically and was done in 2 hours.
I have a full video report of this work on Patreon
I needed to automate migration of URLs from one system to another.
Since I was only ever going to do this once, I consider this tactical automating.
Getting Data Into A System
The System I needed to move data to did not have:
- a bulk import function
- a formal API
But, it is a web application, and it has a GUI.
This means that if I can model what the GUI does then I might be able to automate it. And I don’t mean automate the GUI, I mean automate the messages that the GUI sends.
In Automating And Testing a REST API I referred to this as “App As API” and the book contains a longer write up of this approach.
Basically:
- I use the GUI
- I have all web traffic going through an HTTP Proxy (I used BurpSuite)
- I determine what are the minimum necessary HTTP messages to replicate
- I copy and paste those messages into an IDE
- I automate those messages directly
- I refactor them to have the paramaterised data that I want to upload
- I create the data in an Array
- I loop over the array and send the messages
- I send all messages through a proxy so I can see the outcome and the content
- Initially I run everything in debug mode so that I can check if it is working and review the outcome in the System GUI.
Note:
- The App doesn’t have an API, I sent through the HTTP messages that the GUI sends
- I don’t attempt to replicate all the messages, just the ones that make a functional difference
- with many apps you’ll find logging calls, diagnostic calls, health check calls, etc.
- I just used the ones that trigger functionality
The code is simple:
- I have hardcoded strings
- I refactor those into parameterised Strings that use a
String.format - I did not create any payload objects because I’m working tactically rather than strategically
- If I was automating something complicated then I might use payload objects, but I didn’t need to.
- I needed an object to represent migrated data
- I used a private inner Class
- I often do this for initial work, then refactor to a public Class when other test classes need to use it
- I created a data class, a POJO with a constructor, where all the fields are
public finaland set by the constructor- because I didn’t need the overhead or protection of methods for the fields
- I only needed a single constructor
- e.g.
class ShortUrl{
public final String path;
public final String url;
public ShortUrl(String path, String url){
this.path = path;
this.url = url;
}
}
- I sometimes start my strategic automated code with inner classes and then refactor to public classes, and refactor the public fields to private fields with public accessor and setter methods.
Getting Data Out of a System
I tried to use the “csv export” from the System, but it didn’t have all the fields I needed.
I raised this with the vendor, and they confirmed it was a bug that they will fix, but I’ve already worked around the issue. Tactical automating is often used to workaround bugs and issues.
This is a JavaScript heavy GUI, so I can’t use an HTTP request to parse the data from the HTML.
In these circumstances I automate in the Dev Tools Console.
Useful JavaScript constructs to learn:
- document.querySelectorAll(“tr”)
- row.querySelector(“td:nth-child(4)”).innerText;
Also a grasp of CSS Selectors can help.
For Parsing data I do like Regex:
let regexp = /.*\/(.*)\/(.*)/ig;
let matchOne = regexp.exec(notes);
var project = matchOne[1];
Very often when I’m working with JavaScript from the console I end up building a string as I loop over lots of data:
var reportLine = '"'+url+'" => "'+shortUrl.trim()+'",';
console.log(reportLine);
output = output + reportLine +"\n";
I often console.log it so I can see it for debugging, but by building a string output it is easy to output it to the console and then copy and paste into another System. If I console.log every line then copy and pasting the output is a pain.
Working tactically means:
- I write code that works
- it may not do everything at once, so I have to change variables, tweak conditions etc. when I want to run against different data sets
- I will probably run it multiple times
- it is supporting what I am doing rather than trying to automate a workflow reliably from start to finish
More Regex
I also used Regex find and replace in Visual Studio Code. This is becoming my favourite Markdown and text editor.
I find this useful as a tactical way to convert code from one language to another
find: (.*)=> (.*),
replace: new ShortUrl($1,$2),
The above allowed me to convert a PHP Array to a Java Array of objects.
Mix of Skills
To automate like this I have to use:
- Technical Web and HTTP skills and tools that I have built up from testing
- Java and Libraries that I use for automating strategically
- JavaScript which I frequently use for in browser automating and adhoc GUI querying and manipulation
All of the code was tactical.
- I haven’t saved it
- It wasn’t intended to be maintained
- I refactored it only as far as I needed to make my work easier
- It wasn’t intended to run reliably, sometimes I have no synchronisation and execute it in debug mode to have synchronisation points
- It has minimal error handling and checking, it relies on me spotting problems to stop execution
Tactical is often a first step to creating strategic automated execution, and with more refactoring I could have created a long term strategic automated migration. But I didn’t need to.
No doubt some of the code snippets I’ve included here seem like ‘bad practices’. And they might be if this was strategic.
If you want to see this in action then I have a video in Patreon, but hopefully there is enough here to encourage you to learn some of the tactics and approaches that I used for tactical automating.
