TLDR; Apply MVP principles when coding. Code to the API first. The API is internal before it is external. Unit tests with classes. In code testing with classes in combination. In code API testing. External HTTP API Testing. And then if necessary -In memory and process HTTP API testing. GUI.
A long time ago, in a town which I no longer live in, I wrote a tool called Compendium-TA
Commercially that was a disaster: it was self funded, it took a long time to write and I made some poor technology decisions.
I learned MVP and API First Thinking the hard way. I’ll try and explain within.
One of the (many) things I learned was the need to create a Minimum Viable Product. Build the MVP and sell the MVP. This is now well known due to books like The Lean Startup, and The Startup Owner’s Manual. Sadly I was building my tool in 2003 and the lessons were not as widely propagated then, so I had to learn this from experience by not writing an MVP and failing as a result.
I think most people now know about MVP. But I’m not convinced that people incorporate the MVP philosophy into their development approach.
Build the API first
As an example. I go to a lot of conferences. At the conferences I speak to the tool vendors. I inevitably ask the vendors if they have an API.
Often they do, but it doesn’t cover all the functionality of the product. Sometimes they will say that “the API is coming later, the functionality is available in the GUI”.
I get nervous at this point because an API is easier and faster to explore from a testing perspective, and most of these vendors are test tool vendors. So they aren’t building their tools in a way that makes it easy for them to test the tool.
And… that isn’t even the main point about MVP and API first thinking.
The API you build first isn’t a User API
When I talk to people about building the API first they often think in terms of a REST API, or an SDK.
One of the lessons I learned while building Compendium-TA was to build an API first.
In Compendium-TA I did this by adding the Microsoft Scripting Engine into the product. This allowed me to create internal classes and code that were exposed as an API. I could then use those from the Scripting interface to experiment with the functionality from an end user perspective without writing the full GUI. It also allowed the few actual customers I had, to extend the tool for their needs.
The first API supports building
I still carry this concept through to this day. Many of my custom tools are actually quick sets of code, and they are triggered by an @Test method, and the IDE is my GUI.
Some of my tools never move beyond this level of MVP. And I’m the sole user. They are almost pure API at a code level where the API are a set of classes.
Lessons learned revisiting Compendium-TA
I look back on Compendium-TA with fondness. It did things I still have no tool to replace.
But I built it on dying technology VB6 and when I ran out of funds I didn’t have time to rebuild it or carry it forward.
I looked at it recently and with hindsight I can see that most of the tool is GUI. Much of the complexity was making the GUI scalable for different resolutions and responsive to events. I had to create my own event publish and subscribe mechanism to make it work well.
The actual core code was quite simple.
So I’ve started to rewrite portions of it - using MVP and API first thinking.
MVP API thinking
Compendium-TA had the following main functional areas:
- user defined entity and relationship management
- graph based diagramming and modelling using the entities
- scripting engine
- custom reporting and templating engine
I’ve started with the user defined entity and relationship management since that was the core of the app.
In essence it was a limited implementation of E-R modelling.
Class Level APIs
I explored this in my new code, with code:
- TDD of simple classes to represent the main concepts
- entity,
- relationship,
- entity definition,
- relationship definition
- etc.
The basic classes I create are the initial API that I worked with.
I was able to make sure that I could:
- create entity definitions,
- instantiate the entities as instances
- define relationships
- create relationships between instances
And get a feel for the viability of the approach. See if I needed to have a complicated set of base classes. I experimented with some methods and approaches that I didn’t use. And I cut out a lot of code that seemed too complex for an MVP going forward.
e.g. at the moment I only support 1:M relationships so if I want a M:M relationship then I create two relationships. This seems like a valid thing to do because I cut down on the complexity of the code and can still create viable E-R models.
@Test methods to explore the classes in combination
Because I was experimenting and not sure what I wanted to do. I used a loose form of TDD for the next exploration.
I wrote @Test methods to use the classes in combination and essentially built a DSL that helps me create a scenario in code.
This isn’t always elegant, and I can see that longer term I’m going to have refactor the interfaces on the objects, but as an MVP - the API allows me to get the job done and explore the concepts in more detail.
As an example, here is an @Test I used to explore entity instance creation.
@Test
public void createAmendAndDeleteATodoWithAGivenGUID(){
Thing todos = todoManager.getThingNamed("todo");
int originalTodosCount = todos.countInstances();
String guid="1234-12334-1234-1234";
ThingInstance tidy = todos.createInstance(guid).setValue("title", "Delete this todo").
setValue("description", "I need to be deleted");
todos.addInstance(tidy);
ThingInstance foundit = todos.findInstance(guid);
Assert.assertEquals("Delete this todo", foundit.getValue("title"));
todos.deleteInstance(guid);
Assert.assertEquals(originalTodosCount, todos.countInstances());
foundit = todos.findInstance(guid);
Assert.assertNull(foundit);
try{
todos.deleteInstance(foundit.getGUID());
Assert.fail("Item already deleted, exception should have been thrown");
}catch(Exception e){
}
}
It isn’t particularly good TDD e.g. I have asserts in the middle and this does a lot of stuff.
But it allows me to explore:
- the objects I need
- the methods they have
- what methods do I use in sequence to ‘do’ something
- do I need to throw exceptions or should I rely on returned objects?
- etc.
In Code API Testing
I realised that with a basic E-R model I could create a CRUD application fairly easily and if I was able to support an HTTP API then I might be able to automatically generate a simple REST API system from a very basic model.
To experiment with this I started to create “In code API tests”
i.e.
I don’t have an http server I have the basic concept of PUT/GET/POST etc. I have the concept of a url path and a json ‘body’
And I can write in code tests to explore the CRUD functionality as though it were driven by an HTTP API.
I started with GET because I knew that if I got that working I would be able to create an app with canned data that could be explored with GET requests and it might be possible to use the tool as a front end to “other people’s data”.
So I created relatively ugly code to drive a querying engine:
// project/_GUID_/todo/category
query = todoManager.simplequery(
String.format("project/%s/todo/category",
officeWork.getGUID()));
Assert.assertEquals(1, query.size());
Assert.assertTrue(query.contains(officeCategory));
Having done that. I then wrapped the query engine in an api object to help me explore the API concept:
e.g.
ApiResponse apiresponse = todoManager.api().get(
"todo/" + paperwork.getGUID());
Assert.assertEquals(200, apiresponse.getStatusCode());
I gradually expanded this to cover GET/PUT/POST
And while the tests might be ugly, they do provide a lot of coverage that allows me to make changes in the back end without the overhead of amending a lot of TDD based tests.
// DELETE a Relationship
// DELETE project/_GUID_/tasks/_GUID_
numberOfTasks = myNewProject.connections("tasks").size();
apiresponse = todoManager.api().delete(
String.format("project/%s/tasks/%s",
myNewProject.getGUID(),
paperwork.getGUID()));
Assert.assertEquals(200, apiresponse.getStatusCode());
Assert.assertEquals(numberOfTasks-1,
myNewProject.connections("tasks").size());
Assert.assertNotNull("Task should exist, only the relationship should be deleted",
todoManager.getThingNamed("todo").
findInstance(
FieldValue.is("guid", paperwork.getGUID())));
Once I’m comfortable with the interfaces at this level of testing, I drop down to TDD and design the underlying classes in more detail.
Experiment with a User focussed API
It was surprisingly simple to create a REST API for this.
Since I already have a DSL for the actual ER Model. I was able to automatically create the routings for this model to create a REST API
Part of a model
Thingifier todoManager = new Thingifier();
Thing todo = todoManager.createThing("todo", "todos");
todo.definition()
.addFields(Field.is("title", STRING), Field.is("description",STRING),
Field.is("doneStatus",FieldType.BOOLEAN).withDefaultValue("FALSE"));
Thing project = todoManager.createThing("project", "projects");
project.definition()
.addFields(Field.is("title", STRING), Field.is("description",STRING),
Field.is("completed",FieldType.BOOLEAN).withDefaultValue("FALSE"),
Field.is("active",FieldType.BOOLEAN).withDefaultValue("TRUE"));
todoManager.defineRelationship(
Between.things(project, todo),
AndCall.it("tasks"), WithCardinality.of("1", "*"));
I was able to generate the routings in Spark
e.g. here’s an excerpt
for(Thing thing : thingifier.getThings()){
final String aUrl = thing.definition().getName().toLowerCase();
// we should be able to get things
get(aUrl, (request, response) -> {
ApiResponse apiResponse =
thingifier.api().get(justThePath(request.pathInfo()));
response.status(apiResponse.getStatusCode());
return apiResponse.getBody();});
// we should be able to create things without a GUID
post(aUrl, (request, response) -> {
ApiResponse apiResponse =
thingifier.api().post(justThePath(request.pathInfo()),
request.body());
response.status(apiResponse.getStatusCode());
return apiResponse.getBody();});
options(aUrl, (request, response) -> {
response.status(200);
response.header("Allow", "OPTIONS, GET, POST");
return "";});
delete(aUrl, (request, response) -> {
response.status(405); return "";});
I have notes in a // TODO comment to push this back a level and have the routings generated at the “in code” api, rather than the Spark level (the HTTP API) as it will make it simpler to test at the code level in the future.
But, I’m experimenting and by pushing it to HTTP it allows me to interactively explore it via Insomnia immediately and get a feel for what it is like to work with as an REST API.
In memory and process HTTP API testing
My next ’testing’ step would involve me spinning up the Spark app in memory and from within the same @Test sending through HTTP requests.
i.e. I would do HTTP Integration testing without packaging up or deploying the application.
GUI
Then I would start on a GUI based around the HTTP messages.
But I might not
I’ve gone quite far with the exploration quite quickly and now I have a much better idea of what I can use this for.
I might not take it to the GUI level.
There already exist other tools which do this better for GUI work:
- Jeddict seems like a very capable tool that I will investigate to see if the MVC stack allows me to create GUI based apps quickly.
Part of the reason for building an MVP is to help you investigate and explore options without committing too much time and money into it. And help you evaluate your MVP against other products.
MVP and API Thinking Changed How I Code
MVP and “API First” thinking has influenced how I code.
I do try to write expressive code so that I can ‘see’ how the design of my classes will work together.
And I don’t always stick to a rigid class based TDD approach. I jump between low level TDD and higher level BDD/TDD/“in code” design based work.
This helps me when I’m writing applications but also when I’m writing code to automatically execute applications. Often I’m designing the code and abstractions through usage, because they are automating other apps that would require too much work to ‘mock out’ so we execute against them directly.
Hope this helps.
All the code, in its uglified beauty is available on Github. It is a work in progress. It is an MVP.
https://github.com/eviltester/thingifier
Interestingly, I note that although I’ve written a main method. I do run this from the IDE. I haven’t written any manifest instructions into the pom.xml. And yet I have tested it as a REST API over HTTP from Insomnia. MVP thinking.
Summary:
- The API for an MVP does not have to be aimed a the end user.
- Treat the code and classes that you are build as an MVP and iterate
- Use your initial code to explore a problem.
- Use your Tests in your TDD process to help you design and explore the solution, as well as the code.
- Build your API incrementally:
- class focussed unit tests
- in code testing with classes in combination
- in code API testing
- external HTTP API Testing using Insomnia or Postman
- in memory and process HTTP API testing
- then add an end user GUI using the API
