Blog

Side Project: Arduino Harmonograph

A small project I developed over xmas 2018 to produce this Arduino-based device. [Source Code (GitHub)]

Work Related: PetaGene scores $2.1 M in Funding

My current employment victim, PetaGene, has just scored $2.1 M in funding. Great news for an amazing team!

[TechCrunch story], [Cambridge Independent story]

Cover Design: Sulfone so good for COF catalysis

A cover I designed for the Cooper Group’s work on COF catalysis has been published in Nature Chemistry (article link, screengrab).

Jobson 1.0.0

After many weekends and evenings of fixing little bugs, cleaning up the codebase, and polishing the build, I’ve finally managed to publish v1.0.0 of jobson.

I open-sourced jobson late November 2017. The version I demoed here was already close to release-grade in terms of implementation (the server had >200 tests, was used in prod, etc.). However, the deployment, installation, documentation, and maintenance needed work.

For the open-source release, I wanted to make sure that jobson was OSS-grade before putting a 1.0.0 badge on it. The main changes over the last year are:

  • Stabilized all user-facing APIs (CLI, configuration, HTTP). No known breaking changes since Feb 2018.
  • Added more systemtests to ensure the above
  • Reimplemented UI in Typescript
  • Refactored and cleaned up server code
  • Fixed various bugs (race conditions, etc.)
  • Added various features into the specs (better output collection, templating, etc.)
  • Added more datatypes (float, int, long, etc.)
  • Added a lot more documentation, including API documentation
  • Significantly improved the build, which now builds the full stack into Debian packages, Docker images, etc.

I plan on patching 1.0.0 slightly with some little annoyances I spotted (immediately after deploying, of course), followed by another round of YouTube videos and other media. After that, it’s time to start slowly chipping away at 1.1.0.

Side Project: Live Demos

After several days of faffing around with Docker and build systems, I’ve finally managed to launch a demo’s page here. I’ll eventually integrate these into my about page, but they’re at least a useful resource for showing some of the technologies I’ve worked with.

One useful side-product of this work is that Jobson now has a basic docker image, which enables users to boot a pre-integrated Jobson UI + Jobson stack.

The Demos:

Side Project: Rust: fo2dat

tl;dr: I used Rust to make this CLI utility for extracting Fallout 1+2 DAT files.

I love the occasional playthrough of Fallout 1 and 2. They were some of the the first “serious” games I played. Sure, DOOM/Quake/Command & Conquer were also “mature”—I played them around the age of ~10, which marked me as doomed (heh) by the kind of adults that would also think eating sweets is a surefire path to heroin addition or something—but F1+2 included prostitutes, drugs, slavery, and infanticide: irresistibly entertaining topics for a teenager.

You might think that, with Bethesda buying the rights to Fallout over 10 years ago, F1+2 would’ve had a commercial re-release by now, much like what happened with Baldur’s Gate 1/2, or that such a popular franchise with so many fans would create enough momentum to get an open-source engine ported, but those things haven’t really happened. Probably because of various historical hitches.

F1+2 were originally developed by Interplay, which had a meteoric rise in the 90s followed by a precipitous fall in the early 00s (details). Bethesda excavated Fallout from Interplay’s remains in 2007. However, Interplay’s zombie, adopting a 90s zombie movie strategy of having a character bitten right before the credits roll, decided to give away F1+2+Tactics for free just before the IP passed over. As a consequence, Bethesda probably sees a F1+2 reboot as unprofitable. This assumes that the source code is even available to Bethesda. Interplay may have only handed them the sales rights + binaries, which would be a big shame.

I’ve always wanted to make a F1+2 source port, but it’s an incredibly difficult task for several reasons: the literature for open-source RPG game engines is thin, which means an engine will need to be built from first-principles; F1+2 uses a bunch of bespoke file formats, which means deserializers will need to be coded for each of them; and the game logic—what characters do, actions, etc.—is held as a binary opcode language, which requires a small virtual machine to handle.

The Falltergeist developers are the closest to surmounting the F1+2 Everest. They’ve created something that is close to a complete engine, which is amazing. I decided to chip away at the smaller problem of F1+2’s file formats. The result was fo2dat, a CLI utility for unpacking F2 DAT files, which might help any developers wanting to view the game’s assets in a file explorer.

The micro-project was also a chance to try Rust, the new hot thing in systems programming languages. I enjoyed the experience quite a bit: it feels like the Rust developers really understood the strengths of languages like C (non-OO, simple procedures handling data arguments), C++ (powerful compilers), Scala (pattern matching and abstractions), and Lisp (code generation). They combined those ideas with an excellent build system and package manager, which has resulted in a very nice platform.

Cover Design: Core–Shell Crystals of Porous Organic Cages

A cover I designed for the Cooper Group’s latest breakthrough. Modelled in plain Blender 2.79b. Post processing done in GIMP 2.8.18. [raw high-res render] , [cover official link], [journal article]

Integrating Software

tl;dr: If you find you’re spending a lot of time integrating various pieces of software across multiple computers and are currently using a mixture of scripts, build systems, and manual methods to do that, look into configuration managers. They’re easy to pick up and automate the most common tasks. I’m using ansible, because it’s standard, simple, and written in python.

Research software typically requires integrating clusters, high-performance numerical libraries, 30-year-old Fortran applications by geniuses, and 30-minute-old python scripts written by PhD students.

A consistent thorn in my side is downloading, building, installing, and deploying all of that stuff. For example, on a recent project, I needed to:

  • Checkout a Java (Maven) project from svn
  • Build it with a particular build profile
  • Unzip the built binaries
  • Install the binaries at a specific location on the client machine
  • Install the binaries at specific location on a cluster
  • Reconfigure Luigi to run the application with the correct arguments
  • Copy some other binaries onto the cluster’s HDFS
  • (Sometimes) rebuild all the binaries from source, if the source was monkey-patched due to a runtime bug
  • (Sometimes) Nuke all of the above and start fresh

Each step is simple enough, but designing a clean architecture around doing slightly different permutations of those steps is a struggle between doing something the easy way (e.g. a directory containing scripts, hard-coded arguments in the Luigi task) and doing something the correct way.

The correct way (or so I thought) to handle these kinds of problems is to use a build system. However, there is no agreed-upon “one way” to download, build, and install software, which is why build systems are either extremely powerful/flexible (e.g. make, where anything is possible) and rigid/declarative (e.g. maven).

Because there’s so much choice out there, I concluded that researching each would obviously (ahem) be a poor use of my valuable time. So, over the years, I’ve been writing a set of scripts which have been gradually mutating:

  • Initially they were bash scripts
  • Then they were ruby scripts that mostly doing the same as the bash scripts
  • Then they were ruby scripts that integrated some build parts (e.g. pulling version numbers out of pom.xml files), but were mostly doing the same as the bash scripts
  • Then they were a mixture of structured YAML files containing some of the build steps and ruby filling in the gaps
  • Then they were a mixture of YAML files containing metadata (description strings, version numbers), YAML files containing build steps, and Python filling in the gaps because Python’s easier to integrate with the existing researcher/developer’s work

After many months of this, I decided “this sucks, I’ll develop a new, better, way of doing this”. So I spent an entire evening going through the weird, wonderful, and standard build systems out there, justifying why my solution would be better for this problem.

Well, it turns out this problem isn’t suitable for a build system, despite it having similar requirements (check inputs, run something, check outputs, transform files, etc.). Although my searches yielded a menagerie of weird software, what I actually needed was a configuration manager. Ansible being a particularly straightforward one.

This rollercoaster of “there probably isn’t a good solution already available to this problem”, “I’ll hack my own solution!”, “My hacks are a mess, I should build an actual system”, “oh, the system already exists” must be common among software developers. Maybe it’s because the problem isn’t actually about developing a solution: it’s about understanding the problem well enough. If the problem’s truly understood, it will be easier to identify which libraries/algorithms to use to solve it, which will make developing the solution a lot easier. Otherwise, you’ll end up like me: Keeper of the Mutant Scripts.

(Not so) Fancy-Pants new Website

So, I just spent an evening + morning refactoring the site into a, uh, “new” design.

I only ocassionally work on this site these days—I now see it as the sparse journal of a madman that also likes to distribute mobile-friendly versions of his CV—but I thought it would be a nice and easy blog post to reflect on how the site has changed over the last 3 years.

The original version of adamkewley.com was launched in May 2015 and was the fanciest design:

It makes me feel a bit ill. The first version was modelled off of the kind of erudite landing pages you see 3-man startups use to try and sell IoT soap bars or something. By May 2016, I clearly had gotten sick enough of my own bullshit to remove a bunch of that cute stuff:

By May 2017, there were a few regressions:

And now, in March 2018, I’ve finally decided to just throw almost all fancy tricks out the window and use the simplest HTML + CSS solution I could create:

The previous versions of this site required a bunch of javascript libraries and jekyll gems to build. There were also subtle bugs that would pop up if you used it on a phone. Development/maintenance time was dedicated to fixing that - I also couldn’t help but tweak with the code.

This new site is HTML + a ~50 line long CSS file. It felt strangely liberating to make. Maybe because after working on ReactJS (e.g.) and angular sites I came across this absolute gem that satirically makes the point that barebones sites are: a) fast, b) easy to make, and c) responsive. I couldn’t argue with the logic and immediately wanted to just rip out all the complexity in my site, so here we are.

I wonder how quickly regressions will set in ;)

State Machines in ReactJS

I’m currently implementing job resubmission in Jobson UI and found that state machines greatly simplify the code needed to render a user workflow.

Background

A large amount of Jobson UI’s codebase is dedicated to dynamically generating input forms at runtime.

Generating the relevant <input>, <select>, <textarea>s, etc. from a Jobson job spec is fairly easy (see createUiInput here) but became increasingly complex after adding job copying because extra checks needed to be made:

  • Is the job “fresh” or “based on an existing job”?
  • Was there a problem loading the existing job?
  • Did the existing job load OK but can’t be coerced into the live version of the job spec?
  • Did the user, on being informed of the coercion issue, decide to start a fresh spec or make a “best attempt” at coercion?
  • etc.

Each of these conditions are simple to check in isolation but, when combined, result in delicate state checks:

render() {
  if (this.state.isLoadingSpecs)
    return this.renderLoadingSpecsMessage();
  if (this.state.errorLoadingSpecs)
    return this.renderSpecsLoadingError();
  else if (this.state.isLoadingExistingJob)
    return this.renderLoadingExistingJob();
  else if (this.state.errorLoadingExistingJob)
    return this.renderErrorLoadingExistingJob();
  else if (this.state.isCoercingAnExistingJob)	
    // etc. etc.
}	

These checks were cleaned up slightly by breaking things into smaller components. However, that didn’t remove the top-level rendering decisions altogether.

For example, the isLoadingSpecs and errorLoadingSpecs checks can put into a standalone <SpecsSelector /> component that emits selectedSpecs. However, the top level component (e.g. <JobSubmissionComponent />) still needs to decide what to render based on emissions from multiple child components (e.g. it would need to decide whether to even render <SpecsSelector /> at all).

State Machines to the Rescue

What ultimately gets rendered in these kind of workflows depends on a complex combination of flags because only state, rather than state and transitions are being modelled. The example above compensates for a lack of transition information by ordering the if statements: isLoadingSpecs is checked before isLoadingExistingJob because one “happens” before the other.

This problem—a lack of transition information—is quite common. Whenever you see code that contains a big block of if..else statements, or an ordered lookup table, or a switch on a step-like enum, that’s usually a sign that the code might be trying to model a set of transitions between states. Direct examples can be found in many network data parsers (e.g. websocket frame and HTTP parsers) because the entire payload (e.g. a frame) isn’t available in one read() call, so the parser has to handle intermediate parsing states (example from Java jetty).

State Machines (SMs) represent states and transitions. For example, here’s the Jobson UI job submission workflow represented by an SM:

From a simplistic point of view, SMs follow simple rules:

  • The system can only be in one state at a given time
  • There are a limited number of ways to transition to another state

I initially played with the idea of using SMs in ReactJs UIs after exploring SM implementations of network parsers. I later found the idea isn’t new. A similar (ish) post by Jeb Beich has been posted on cogninet here and contains some good ideas, but his approach is purer (it’s data-driven) and is implemented in ClojureScript (which I can’t use for JobsonUI). By comparison, this approach I used focuses on using callbacks to transition so that individual states can be implemented as standard ReactJS components. In the approach:

  • A state is represented by a component. Components can, as per the ReactJS approach, have their own internal state, events, etc. but the top-level state (e.g “editing job”) is represented by that sole component (e.g. EditingJobStateComponent)

  • A component transitions to another state by calling a callback with the next “state” (e.g. transitionTo(SubmittingJobStateComponent)).

  • A top-level “state machine renderer” is responsible for rendering the latest component emitted via the callback.

This slight implementation change means that each component only has to focus on doing its specific job (e.g. loading job specs) and transitioning to the next immediate step. There is no “top-level” component containing a big block of if..else statements.

Code Examples

A straightforward implementation involves a top-level renderer with no decision logic. Its only job is to render the latest component emitted via a callback:

export class StateMachineRenderer extends React.Component {

  constructor() {
    const initialComponent =
      React.createElement(InitialStateComponent, {transitionTo: this.handleTransition.bind(this)});

    this.state = {
      component: initialComponent,
    };
  }

  handleStateTransition(nextComponent) {
    this.setState({component: nextComponent});
  }

  render() {
    return this.state.component;
  }
}

A state is just a standard component that calls transitionTo when it wants to transition. Sometimes, that transition might occur immediately:

export class InitialState extends React.Component {
  componentWillMount() {
    const props = {transitionTo: this.props.transitionTo};

    let nextComponent;
    if (jobBasedOnExistingJob) {
      nextComponent = React.createElement(LoadExistingJobState, props, null);
    } else {
      nextComponent = React.createElement(StartFreshJobState, props, null);
    }
    this.props.transitionTo(nextComponent);
  }
}

Otherwise, it could be after a set of steps:

export class EditingJobState extends React.Component {
  // init etc.
  onUserClickedSubmit() {
    this.props.api.submitJob(this.state.jobRequest)
      .then(this.transitionToJobSubmittedState.bind(this))
      .catch(this.showErrors.bind(this))
  }
	  
  transitionToJobSubmittedState(jobIdFromApi) {
    const component = React.createElement(JobSubmittedState, {jobId: jobIdFromApi}, null);
    this.props.transitionTo(component);
  }
}

Either way, this simple implementation seems to work fine for quite complex workflows, and means that each components only contains a limited amount of “transition” logic, resulting in a cleaner codebase.

Conclusions

This pattern could be useful to webdevs that find themselves tangled in state- and sequence-related complexity. I’ve found SMs can sometimes greatly reduce overall complexity (big blocks of if..else, many state flags) at the cost of a little local complexity (components need to handle transitions).

However, I don’t reccomend using this pattern everywhere: it’s usually easier to use the standard approaches up to the point of standard approaches being too complex. If your UI involves a large, spiralling, interconnected set of steps that pretty much require a mess of comparison logic though, give this approach a try.

Cheeky Hackers

I’ve been running several webservers behind custom domains for a while now (plateyplatey.com, textadventurer.tk, and this site) and it never ceases to amaze me how cheeky bots are getting.

For example, certbot recently started complaining that textadventurer.tk’s TLS certificate is about to expire. That shouldn’t happen because there’s a nightly cronjob for certbot renew.

On SSHing into the server I found an immediate problem: the disk was full. Why? Because some bot, listed as from contabotserver.net decided to spam the server with >10 million requests one afternoon and fill the HTTP logs. Great. Looks like I’m finally going to implement some log compression+rotation.

Then there’s the almost hourly attempts to find a PHPMyAdmin panel on my sites. That one always surprised me: surely only a small percentage of PHP sites are misconfigured that badly? Lets look at the stats:

Percentage of websites using PHP

Even if 1 % of them are misconfigured, we’re doomed.

Jobson: Now in 2D

I recently made screencasts that explain Jobson in more detail. The first explains what Jobson is and how to install it. Overall, Jobson seems well-recieved. The first video seems to be leveling off at around 2700 views and Jobson’s github repo has seen a spike in attention.

Will other teams start adopting it or not? Only time will tell.

Jobson: Webify CLI Applications

This is a post about Jobson, an application I developed and recently got permission to open-source along with its UI.

Do any of these problems sound familiar to you?:

I’d like my application to have a UI.

I want to trace my application’s usage.

I want to share my application.

They’re very common problems to have, and are almost always solved by building a UI in a framework (e.g. Qt, WPF), sprinkling on usage metrics, and packaging everything into a fat executable.

That development approach is becoming challenging to do well nowadays because clients are more likely to use a mixture of OSs and 3rd-party resources. Therefore, new projects tend to need to choose between several options:

  • Develop a CLI application. The easiest thing to develop in many situations. However, they can be challenging to use and have the usual client-side distribution problems (installers, etc.).

  • Develop a UI application. Easier to use use. However, they’re usually less flexible for developers to use and are more challenging to develop.

  • Develop a web API. Has similar flexibility to client-side CLI applications but doesn’t have the distribution issues. However, web APIs are challenging to develop because you need to deal with networking, sysadmin, setting up a domain, authentication, etc.

  • Develop a full (API + UI) webapp. Easy to use and flexible (because of the API). However, webapps have all the complexities of the above and more: you effectively need to develop a full web API and a working UI.

When scoped in this (biased) way it’s clear that webapps are ideal vehicles for delivering the full “product” but CLI applications are ideal for ease-of development and flexibility.

It’s clear that developing a method for turning CLI applications into webapps would be valuable. It would enable developers to rapidly develop and roll out applications. That’s what I explored with Jobson: a web server that turns CLI applications into webapps.

Jobson’s Approach

Jobson has now been working in a production environment for a few months now and has proven to be an effective vehicle for delivering new platforms. However it cannot turn any application into a webapp. That would be tough: the potential “inputs” (applications) and “outputs” (webapps) are far too varied.

Jobson’s primary limitation is that the only applications it handles are batch applications that: a) start, b) take typical inputs, c) write typical outputs, and d) end. Most applications follow this model (e.g.echo, gcc, nmap, and ls).

With that limitation in place, Jobson could then be designed with simple principles in mind:

  • Input applications are explained to Jobson via “specs” (at the moment, YAML files). The applications themselves should not need to be changed in order to suit Jobson.

  • Jobson uses the specs to host a standard HTTP REST API which can be used by all general programming languages and utilities (e.g. curl jobson-server/v1/jobs)

  • The API is regular and predictable: job specs should always create the API you’d expect.

  • Jobson’s authentication uses standard protocols (e.g. HTTP Basic) that can be integrated into a larger systems easily.

  • (Default) persistence uses plaintext files on the filesystem, which allows for transparent debugging and administration.

  • Persistence can be swapped for other implementations (e.g. a MongoDB implementation)

  • Job execution itself uses the underlying operating system’s fork(2), and exec(2) syscalls, which allows Jobson to run any application the host OS can run.

  • By default, Jobson has no external dependencies beyond Java. It should be bootable from a barebones linux server with minimal configuration.

Overall, this design means that Jobson can webify almost any application very quickly. I could webify a major CLI tool in less time than it took to write this post, resulting in a web API and UI for that tool.

Why It’s Useful

Here are some typical development problems Jobson could help out with:

Problem: You’ve got a cool application idea you want to share

Without Jobson:

  • Develop the idea
  • Develop all the crap necessary to share that idea (download links, websites, databases, etc.)
  • Share link

With Jobson:

  • Develop the idea
  • Write a Jobson spec
  • Share link

Problem: You’ve got a toolchain (e.g. a pentesting toolchain) that has hard-to-remember commands. You want to streamline that toolchain such that you can run and queue multiple commands.

Without Jobson:

  • Develop a UI around the tools, dealing with the usual development headaches
  • Use the tool

With Jobson:

  • Write a few small scripts around the tools (if necessary)
  • Write a job spec
  • Use the Jobson UI

Problem: You want to automate data requests at work, but there’s a lot of flexibility in the requests (FYI: this is why Jobson was made).

Without Jobson:

  • Write an application to handle typical data requests. The application will have to deal with request queues, request structuring, etc.
  • Find that end-users suddenly want to make different requests
  • Redevelop, etc.

With Jobson:

  • Write short scripts for handling a data request
  • Host it with Jobson, which automatically deals with all the other stuff
  • Find that end-users suddenly want to make different requests
  • Add a new script + spec into Jobson

Overall, I believe this approach to developing a job system is extremely flexible and much easier to work with. Jobson abstracts all the typical job system faff (auth, APIs, queues, etc.) away from what’s actually important (an application that does something), resulting in a much cleaner system.

The next steps with Jobson are to streamline installation (by implementing installers), add a landing+documentation page, and start making tutorial videos for it. Watch this space =)

Textadventurer Has Actual Games Now

I’ve been distracted by other things going on, but I finally managed to spend an evening or two adding actual games into textadventuer.

The following games were added:

  • Droids (a text adventure game), python3, I found here
  • DUNGEON (Zork 1), native C application, One of the earliest interactive fiction computer games ever made (wiki). I compiled the open-source port of it for Textadventurer
  • Descent Into Evil, python2, I found it on Tyler Kershner’s blog
  • Zork-py, python3, A fan’s interpretation of Zork (above) that I found here
  • Python Text Adventure Game, python2, someone’s collage project that I found here
  • little-dungeon, python2, posted on reddit with accompanying source

One thing the game search did for me was find similar platforms. I should’ve looked before hacking away at textadventuer because some of those platforms (e.g. http://textadventures.co.uk/) are very well made. Some even use clever tricks like emulating DosBox in the browser. However, I’m yet to come across a platform that can run any application via a browser (textadventuer’s “USP”, if you will), so hopefully other developers will find the server and UI source helpful if they have an idea along those lines.

Text Adventurer

One of the first things people learn when they start programming is how to write a text prompt. It’s a decades-old exercise that teaches new programmers input-output.

In order to inject a little excitement, learners are normally encouraged to write interactive games using standard IO. This helps them learn programming by interactively - they will need to learn conditional logic to handle the “Will you stab the monster with your sword or run away?” prompt.

This creative learning process is great but, unfortunately, it’s hard to show the creations to other people. Any players will have to undergo the nuisance of installing an interpreter, libraries, and the game in order to play. These deployment problems are alleviated on the web: javascript can be distributed and executed remotely in a browser with no effort required from the client.

If people wrote their text adventures in javascript, they could be easily be shared with a URL. However, javascript is not necessarily a good teaching language. Enforcing people to write their text adventures in it for the sake of distribution detracts from the learning experience.

An ideal system would allow text adventures (console applications) to be written in any language but also be distributed on the web. This is what my latest project, textadventurer, tries to achieve (gallery).

textadventurer keeps interaction in the browser while moving execution onto the server. Communication between those two layers is achieved with websockets. This makes it easier for people to play the game. The frontend focuses on presenting games (CLI applications) to the players and provides a basic UI for sending input (STDIN) to the server and recieving output (STDOUT) from the server.

Using process forking server-side affords a lot of flexibility: the server is completely agnostic to the language or framework that the game is written in. This means I can use textadventurer to distribute any standard interactive CLI application. With that in mind, I plan on deploying historic text adventure games to textadventurer when I get the chance so that people can enjoy those games once more without having to faff around with installers, legacy interpreters, etc.

Keep Side Projects Small

I’ve officially thrown a deployment of PlateyPlatey (PP) up onto plateyplatey.com (galley) along with its server and frontend source code. I previously explained PP in a blog post.

PP is missing many features I would of liked to have. However, a few weeks ago I took a long look in the mirror and decided to just tidy it then throw it up on the net. This let me start sharing it on sites such as reddit to gauge if anyone’s interested. After about a week of PP being up, I concluded that no, no one is interested and yes, I spent way too much time coming to that conclusion.


This is my attempt to articulate general some general rules of thumb I have picked up over the last few side-projects. It was written just after PP, one of my bigger projects, didn’t live up to my expectations. It is not a definitive guide nor something you should take seriously without reading other developers’ thoughts.


I used to approach software side projects as an opportunity to write full-blown applications that I might someday commercialize or turn into a big hit. This, I’ve learnt, is a silly approach. Side projects should be approached as side projects, not as full-blown projects.

If a side project requires a light scripting engine, command controller, custom UI elements, a fair amount of interactivity, servers, etc. then it’s doomed to failure. This isn’t because these features can’t be implemented—I believe any developer can implement any feature if given enough time—but because big projects don’t suit only getting “and hour here” and “an evening there”.

The problem with PP is that I specced my minimum viable product way too ambitiously. The spec included several headline features:

  • The user should be able to visualize a plate in the browser
  • That plate should be interactive, allowing the user to click wells and selectors on it
  • Interaction with the plate should provide visual cues in the table, so the user can visualize the link between plate and tabular data
  • The table should be interactive, with features resembling Excel tables
  • The UI should have commands that let the user manipulate the table and plate layouts
  • Those commands should have disable logic and carry metadata such as tooltips
  • There should be light support for scripting so the user can write/record macros
  • etc. etc.

In isolation, each of these features are easy to implement. However, implementing all of them into one system requires more care and, crucially, architecture.

If you write software full-time as a job then you can afford to spend a month or two making prototypes, designing hierarchies, and, ultimately, implementing a system that’s quite large. However, side-projects won’t get more than, say, ~20 % of the time an at-work project will get. Because of that, those few weeks of prototyping and planning will stretch out across months.

In principle, many months isn’t a problem for a side-project with no time limits. However, spending many months without making anything requires a lot of discipline: no one finds planning and prototyping as fun as diving into the heart of an idea. The same goes for boring old architecture documents, automated testing, and design specs.

Those things mostly exist to make working on bigger projects more predictable and to make the maintenance of a working system easier for other developers. A small side project is only going to be worked on by one, maybe two, developers over a relatively short period of time; therefore, it isn’t worth worrying about the theoretical overhead of maintaining and sharing it. If a small side project becomes an overnight sensation then—because it’s small—there is scope for dropping the initial codebase and starting again with all the knowledge and expertise the first runthrough gave you.

Dropping those things for a bigger side project, though, is dangerous. Not having plans, specs, architecture, or tests in place for a bigger project will result in development grinding to a halt under its own technical debt. When that happens, you’ll need to do a big refactor. This happened roughly twice with PP. First, to move all the javascript code into separate hierarchies with a makefile build system. Second, to translate all the javascript into typescript and use webpack to build everything. The time taken by those reorganization steps really sucked some of the momentum out of PP’s development. Especially because the actual functionality of PP didn’t change after each step - they just made working on the codebase easier.

The main take home is that side projects get developed much more slowly than real on-the-job projects because of time constraints. That makes working on bigger side projects especially painful because the boring planning, speccing, and testing steps—which tend to come in handy for bigger projects—last too long to keep the momentum of a side project going. By contrast, smaller projects work well with a “code and fix” development model which gives immediate feedback and a semi-working system very quickly. This helps maintain momentum and interest; however, “code and fix” gets treacherous once projects start growing, so don’t let them grow.

Keep small. Keep agile. Try things out. Get the audience’s opinion. Then grow.

Low Level Programming

I haven’t posted in a while because I have been busy with a few random studies and projects.

Low level programming has always, to a high-level programmer like myself, felt like a mysterious playground where all the real hackers write graphics engines, password crackers, and operating systems. I have always had a fascination with people who can write something hardcore like an emulator. Sure, I can open a window, draw some graphics, and update the UI to reflect state changes. However, an application that does all that and some low-level wizardry is in an entirely different league of complexity.

Many prominent programmers recommend learning something like C because it is a good way of learning how computers really work. Apart from the occasional toe-dip into C, I have almost exclusively coded in higher level languages.

I want to get better at programming. I’m getting bored of the high-level: there’s only so many ways classes, functions, currying, type systems, can be repackaged before you’ve seen it all. I like the idea of mastering the low level. However, deciding on where to start was challenging.

Last weekend, however, I decided to set myself what I thought would be a simple exercise:

Write a websocket server that capitalizes and echoes any text it recieves. You cannot use a websocket library.

I eventually hacked something together that works (github). I took a shortcut and used the nodejs http-parser and a base64 encoder written by Brad Conte. However, the websocket implementation was derived directly from RFC6455.

Even though it’s pathetically basic, is blocking (I don’t fork or do any fancy event queueing on the connections), and uses compile-time allocated buffers for most of the messages (so it can’t stream big websocket frames), this was still a very eye-opening first step into some low-level programming. I somewhat regret writing the process forking for text adventurer in clojure (with the help of websocketd) after seeing how straightforward a lower-level implementation would be.

Engineering Flexibility

It’s tempting to believe that having a flexible file format for a system’s configuration is enough to ensure forward-compatibility and flexibility as the system grows. The thought process is that with a suitable data format—for example, XML or JSON—extra information can be added without breaking existing systems. For adding new fields into the data, this is true. However, if you don’t engineer additional flexibility into the fields themselves then you may end up needing to make breaking changes anyway or, worse, end up with a much more complicated data file.

Imagine you are implementing an ecommerce server that exposes products. The spec for the server stipulates that there will only be a very small number of products. It also stipulates that rebooting the server to refresh the system with new products is fine. With that in mind, you add a products field to the server’s configuration:

port: 8080
products:
 - name: Acme Explosive
 - name: Fake Backdrop 

One day—and this day always comes—a change to the specification comes: some products should be hidden on the site. To implement that feature, you add a hidden: field to the product. It is assumed that if a hidden: field is not defined for a product then it should default to false. This is backwards-compatible and easy to implement given the extensible YAML format being used:

- name: Fake Backdrop
  hidden: true

Later on, it emerges that product-hiding should not be binary. It should depend on the group that the user belongs to. To implement that, you add a hidden_to: field. However, to allow for backwards compatibility, it is enforced that hidden_tos entries take priority over the hidden state. Side-rules are beginning to creep in:

- name: Fake Backdrop
  hidden: false
  hidden_to:
    - roadrunners
    - unregistered-customers

As the system gains popularity, it becomes increasingly necessary to be able to add products to a running system. It is decided that products should be held in a database rather being loaded from the configuration file. The configuration file, therefore, should support providing database details as an alternative to products. However, backwards compatibility should still be maintained to prevent legacy deployments and tests from breaking:

port: 8080

use_database: true
database:
  url: //localhost/database
  user: productapp
  pw: password123

A heuristic is adopted that if use_database has a value of true then the server should load products via the database defined in database; otherwise, it should load them via products.

The sysadmin then mentions that it would be useful to allow the server to listen on multiple ports, some of them being HTTP and some of them being HTTPS. Marketing also wants to integrate email updates into the server. However, one week into developing email updates, they also mention that some systems may need to send tweets instead of emails. Your implementation strategy follows the same logic as previous development efforts:

port: 8080

other_port_configurations:
  - port: 8081
    protocol: http
    port: 8082
    protocol: https
	
use_database: true

database:
  url: //localhost/database
  user: productapp
  pw: password123database:

send_tweets_instead_of_email: false

email:
  host: smtp.app.com
  port: 442
  username: productapp-email
  password: thisisgettingsillynow

This process of introducing additional data that is handled by special-case heuristics is a common pattern for extending configurations without breaking existing fields. However, it can become quite difficult to keep track of all the different rules that arise out of this gradual growth in complexity. Not only that, the resulting configuration file can be difficult to work with.

What would make this process a lot easier would be to extend the actual fields themselves. However products, for example, is defined to contain an array of products. The only actions that can be carried out on products are to add or remove products.

The solution to this is to to ensure that there are extension points at major parts of the configuration hierarchy when it’s initially designed. Relegate primitive values (which can’t be extended later) deeper into the configuration hierarchy and have clear extension points.

An example of adding extension points would be:

products:
  product_data:
    - name:
        value: Acme Explosive

In this design products, product (within product_data), and name could all be extended later with more type switches or other information to allow them to be loaded/handled differently. Applying this nesting concept to the configuration at each step of the above scenario would result in something like this:

server:
  application:
    port: 8080
    protocol: https
  admin:
    port: 8081
    protocol: http
  internal:
    port: 8082
    protocol: http
	
products:
  source: database
  database:
    url: //localhost/database
    user: productapp
    pw: password123database:database:

product_notifications:
  type: email
  settings:
    host: smtp.app.com
    port: 442
    username: productapp-email
    password: thisisgettingsillynow

There would still be some heuristics involved with handling missing keys, defaults, type switches, etc. However, the hierarchy between features is maintained and there are now no sibling heuristics. As a result, the config-handling code can be modularized to then read each key in a configuration in isolation. The products in this improved example could be handed to .handleProductsConfiguration(ProductsConfiguration conf), a function that now doesn’t also need to receive sibling use_database and database keys.

A strict tree hierarchy maps cleanly onto polymorphic systems and is surprisingly easy to implement using libraries such as jackson (java):

@JsonTypeInfo(
  use = JsonTypeInfo.Id.CLASS,
  include = JsonTypeInfo.As.PROPERTY,
  property = "source")
interface ProductsConfiguration {
  <T> T accept(ProductsConfigurationVisitor<T> visitor);
}

class DatabaseProductsConfiguration implements ProductsConfiguration {
  @JsonProperty
  private String url;
  @JsonProperty
  private String user;
  @JsonProperty
  private String pw;

  @Override
  public <T> T accept(ProductsConfigurationVisitor<T> visitor) {
    visitor.visit(this);
  }
}

The above is compile-time verified, type-safe, and involves no conditional logic. Completely new product sources can be configured by defining a configuration that implements ProductsConfiguration and updating the relevant visitor. Clearly, this implementation requires more up-front engineering than the original design. However, for projects with changing specification (i.e. most projects) allowing for flexibility pays for itself.

The Cycle of Platform Maturity

In programming, “mature” is a funny word. When I first started programming, it had negative connotations. A “mature” programming language was boring, had complicated tools, over-complicated design patterns, and would likely die off soon. By contrast, non-mature languages were exciting, free-form, and carving out a new frontier in software.

My opinion has evolved since working as a programmer rather than studying as one. Mature now implies that a language will have have good debuggers and profilers. It will have an established build and testing systems for the language. Its core libraries won’t change midway through a critical project. Most importantly, though, mature languages (tend to) have moved past the same boring old problems that every language ecosystem encounters in its early days.

JavaScript’s history is a good case study for this. I started on JavaScript in its early (IE4) days. Back then, I would open notepad, write a few HTML tags, a <script> tag, and voila I could see stuff happening in my browser that my code put there. Simple.

Unfortunately, it’s unlikely that my code would work in any other browser. The JavaScript language platform was horribly fragmented. Internet explorer’s implementation of browser functionality was non-standard. Even the most trivial DOM manipulation or standard library call would behave differently between IE and Firefox. It was not unusual to have a lot of if(browser === IE) tests throughout a codebase. This made working with JavaScript less simple.

jQuery stabilized JavaScript development somewhat but it wasn’t officially supported as a language standard. Because of that, a large amount of JavaScript libraries ended up with a tangled mess of hidden dependencies on different versions of jQuery. This sometimes resulted in dependency hell. Also, adding jQuery to a website project was considered the “done” thing at this stage in JavaScript’s history. Because of that, you couldn’t just open notepad and write a few tags and some JavaScript, you would also have to import jQuery and become familiar with that. The complexity was creeping in.

Later, Firefox, chrome, and—because of the blockbuster release of the iPhone—safari began to eat away at IE’s market share. They were standards compliant, IE wasn’t. Web developers started to make sites that wouldn’t work in IE, which prompted IE users to install IE’s competitors. The overall effect of this was that a growing share of users started using standards-compliant browsers. JavaScript, the language platform, became (mostly) stable and predictable.

Because of this stability, web developers could build bigger, more complex, sites. Animations, realtime updates, and dashboards became more popular. As a consequence, tools for debugging this complexity became more valuable. In an effort to secure their market share, Chrome and Firefox began to include developer tools which enabled developers to set breakpoints and profile their websites. Having these tools made web development a lot nicer; however, the workflow was now more complex.

The next issue developers encountered was managing libraries and dependencies. Traditionally, libraries would be manually downloaded and included in <script src="..."> tags in the page. However, it was tedious to do this once projects became split over many files. It was also inefficient because the HTTP/1.1 protocol doesn’t suit downloading many small files.

Backend frameworks, notably Rails, began to offer asset pipelines for combining, minifying, and including scripts in websites. This enabled developers to build bigger codebases fragmented over many files; however, they would also need to learn how their build system links files and use appropriate strategies to debug minified files.

Once using a build system with JavaScript became the norm, developers continued to up the ante, creating more complex web applications. However, the JavaScript ecosystem was beginning to crack at the edges under this new complexity. Managing and installing the ever-increasing glut of project dependencies became tedious. bower and npm raced to fill the gap, allowing developers to automatically acquire and install dependencies. It became much easier to install JavaScript libraries; however, you’d have to learn bower or npm.

Now that developers could install many libraries easily and painlessly, they installed many libraries easily and painlessly. The JavaScript language didn’t support declaring dependencies on a module-by-module basis. Consequently, dependency hell ensued. Libraries such as requirejs came along to fix the problem, allowing developers to declaratively state a module’s dependencies. Developers no longer had to reorder <script> tags to make the application work; however, they did need to learn requirejs.

At this stage, JavaScript was beginning to become mired in libraries for doing stuff that the language ought to support. ES6 (/ ES2015+) was specced out to deal with some of those issues. It offered standardized higher-level language abstractions for building bigger systems (classes, getters, setters, proxies), language-level protections (const, Symbol), and declarative library loading (import). However, it also made the language a little more complex.

Today, JavaScript is a (mostly) standardized platform with good debugging and profiling tools, batteries-included libraries for standard operations, a popular dependency manager (npm), and module-level dependency management to prevent namespace pollution (import). It finally has all the features every other old-fogie language has.

However, if I was starting out today, I wouldn’t use JavaScript. Clearly, the future is akewley-lang. It’s amazing. You can open a text editor, write plot my data as a bar chart please and see stuff happening right away. It’s so simple. Now, all I need is to generate that bar chart and have a whizzbang animation when it loads! Luckily, there’s a library for that on the akewley-lang community forum I can download. akewley-lang is so much simpler than JavaScript, ruby, python, C#, F#, java, C++, and C. The designers of those complicated languages (and associated ecosystems) didn’t know what they were doing.

Just Make Stuff

Software development can be a bottomless well of languages, build environments, platforms, coding standards, and cat analysis algorithms. For someone who likes to study and learn new techniques, this is great. A bottomless well of stuff to better at. However, after almost ten years of doing just that, I’m willing to make a bold statement: curiosity can be maddening and knowledge can be hindrance.

Curiosity is maddening because the more I learn about assembler, linux, garbage collection, compiler theory, engineering techniques, software principles, functional programming, networking, etc. The more I’m convinced that I know almost nothing about computers.

Every time I turn over a figurative “small pebble” in the software world I end up discovering that there’s entire industries dedicated to what’s under that pebble. Even a statement as innocuous as “I’d love to know how my CPU actually sends input requests to this USB hard drive” - can yield a very long (but interesting) explanation.

Complexity isn’t unique to software development. However, combined with curiosity, software is one of the few fields where there is practically no inertia to ping-ponging. If I like the look of writing client side 3D games in C++, I can be doing it in under 20 minutes for free. 30 minutes later I can be writing a plugin for firefox that makes it purr whenever I visit a website with “cat” in the URL.

You’d think that healthy curiosity is a good thing: over time, it grows knowledge. However, knowledge can become a hindrance if you leverage it a little too much. A simple “Hello World!” application can scream out with potential pitfalls - do we have standard IO? What about cross-platform? What about internationalization? Is there a more flexible design for delivering “Hello” and “World” to the console? What character encoding is being used?

This, I think, is why there’s a significant amount of very young, inexperienced, software developers making interesting—sometimes very valuable—products in the field. They don’t know that their servers will implode once they have a few thousand concurrent connections. They don’t know that it’s going to be very painful to refactor that UI they hacked together with four bleeding-edge UI frameworks. They don’t know that there’s a big competitor in the market with a lot of money. They shoot first and ask questions later, which makes them one of the few demographs doing the shooting.

We’re all aware of this phenomenon. What I’m saying isn’t news. However, here I am, writing heavily designed, SOLID certified, high unit-test coverage software that few people use. Meanwhile, there’s teenagers out there hacking together VR experiences in which I get to fly around space as Thomas the tank engine.

What’s missing? Isn’t the reason we get curious, study, and train so that we can make even cooler stuff? Yes. However, as presented, curiosity is maddening and knowledge can be a hindrance. The trick, I’m slowly convincing myself, is to have a cool and interesting problem or project to apply that curiosity and knowledge to and not get hung up on the details. Otherwise, you may end up spending hours analyzing the random noise of a bingo game.

So, with that in mind, over the next year, I’m going to be making a conscious effort to just “make stuff” and care alot less about how philosophically pure its software engineering is or what amazing 0.5 % speed bonus I’ll get if I change to some slightly newer framework half way though.

Now, all I need to do is write a design document, get people to approve on it, do a technology review, select a programming language, ah yes, mutability is evil! It’s the reason I haven’t created a decent app yet, I should look into Haskell! Ok, now possible test harnesses, oh I wonder what’s the difference between this test harness and this other very similar test harness, that could really have a big impact on my project and I don’t want to make an error before I even start…