That moment where you should have automated but didn't

TLDR; I migrated blogs over to Hugo and I didn’t automate because I was only doing it once, I should have automated because I actually migrated 450+ times (at least once per post. Find results at testerhq.com

On Risk Aversion

Because I’m slightly risk aware I have the following concerns about my main blogs:

blogger.com is free so at some point Google will cancel it and I will lose everything
wordpress uses php and a database and at somepoint someone will take advantage of a security hole and I’ll get hacked and lose everything

When I post to linkedin I have the concern that:

there is no RSS feed
linkedin might go down and I’ll lose everything

And if I ever start posting to medium or other platforms I will have the same issues.

Risk Mitigation

Therefore I decided to ‘archive’ everything into a static site so that if everything goes down I have a backup.

I chose to use Hugo for this.

Subtle Wins

This has the advantage that:

all posts in Hugo are written in markdown
I already write all my posts in markdown (and covert into html for pasting into the other platforms)
I experimented with my testerhq.com domain
I get to try out the multiple site hosting on my vidahost cloud plan
much harder to hack a static site (and if you do, I just re-upload)
I can eventually put all posts on github and have an offsite source back up as well
I can repurpose all the posts later
stuff I post to slideshare
instagram
tweets I think are useful
youtube videos etc.

But I didn’t Automate the migration

I looked around for tools that could easily migrate from Wordpress to Hugo and blogger to Hugo but didn’t find a lot that fit my workflow.

I thought I should really write my own but since I saw so many failed attempts out there I thought it might be too hard and take too long.

I really should have done

I thought that since I’ll just migrate once, that I really couldn’t justify the time to automate it.

I might migrate each blog once (4 of them), but there are over 450 posts, that might be worth automating.

I found a solution

My final solution was:

Use wordpress blogger import
Use a jekyll export Wordpress plugin
import from Jekyll to Hugo

I used a Wordpress bitnami VM and imported my blogger blogs to that, then exported to Jekyll and then imported to Hugo.

I exported my wordpress blogs directly to Jekyll and imported to Hugo.

And then the pain started.

I should probably have automated this part

I then had to check every post for mistakes during the various migration.

Sometimes it was:

  unconverted
html for iframes of slideshare and youtube messed up
images not convered to markdown properly
mad html not converted properly due to a missing tag somewhere
various special chars not converted properly

Pretty much every post needed to be hand checked and amended with find and replace in some form or other.

It took me ages. I suspect it would have been faster for me to automate this last part.

I did automate something

I dug out my old HTTP Testing Web Crawler and amended it to check for characters on the page this made it easy for me to check for:

images that hadn’t converted properly e.g. and left ![]( type chars on the html

I still have a lot of broken links to stuff I’ve linked to over the years, and I’ll slowly fix that up over time.

I almost repeated the same mistake

I was looking at an old Choose Your Own Adventure Game that I created a few years ago in Twine with my Son. And thought about converting it into more of a CYOA game book format e.g. walk out of the door (goto 8) listed on the page rather than a link.

I started hand converting it from a twine archive into markdown because I looked around and could not find any tools available that did this, and I saw a few failed attempts and I was about to say ’this is hard’.

But I knocked up a quick converter using Java and JSoup and it didn’t take long at all.

The benefit is, that because I have automated: