Just as a little fun, I decided to rewrite textadventurer in Rust
The server was initially written in ~600 LOC of Java with a basic
(source). The Java
implementation worked fine–it was essentially just a tiny tech demo
to demo a jobson feature–but
I decided to rewrite it in Rust so I could understand where the
pain-points are in an application like this.
This is an interesting standalone Rust project because:
It’s small and standalone. A minimal implementation would only
really require a basic HTTP + websocket implementation.
It requires multithreading and appropriate IO handling, because each
game that’s launched in a browser effectively launches a live
process server-side. The live process can write to a client at
arbitrary times and visa-versa for the client writing to a process.
It has several interesting edge-case conditions that need to be
handled correctly in order for the server to not leak:
Clients can disconnect at arbitrary times. Subprocesses must be
Processes can exit at arbitrary times. The server has to ensure it
waits on the process to prevent zombies. Clients need to be
Clients behave slightly differently in how they handle websocket
connections. For example, Firefox seems to keep TCP and websocket
connections open even when a user closes the browser tab, so the
server can’t rely on sessions being closed appropriately.
The reimplementation used
actix-web, an actor
system. Actor-based architectures are really interesting for these
types of servers because they greatly simplify the mental juggling
required to understand concurrent processes.
The Rust rewrite was actually smaller (<600 LOC) than Java original
version and had more features (command-line flags, configuration
options, etc.) and used significantly less RAM and CPU than the Java
version. The resource usage drop is especially nice because I like
paying less for my cloud servers :)
So my latest interest has been trying to squeeze performance out of
simple algorithms - mostly so I can understand the impact of branch
misses, lookup strategies, etc.
I spent Sunday writing an optimized solution to the language
challenge. I ended up doing all kinds of hacky things I’d never
recommend doing in prod, like writing a custom vector and writing
tricksy algorithms. Repo
Well, for all my hard work, I managed to come… Second! To, of
course, a much tidier Rust implementation (❤️). Why? Not because the
Rust solution is a more efficient (it’s not: it takes at least 2x
more cycles and memory than my single-threaded C++ implementation),
but because the the Rust implementation throws threads at the problem,
which is the true power of Rust (in addition to the fact that the Rust
version can be just as efficient as the C++ one by adding some SIMD
and unsafe code).
This kind of underlines an important trend that’s likely to shape
software development over the next decade or so: processors aren’t
getting faster, but they are getting more cores. This means that there
is an upper limit on how fast single-threaded software can get in the
future. Rust has the dual advantage of being extremely fast and
easy to multithread. It’s well-positioned for the future. All we need
to wait for is when IT departments around the world start realizing
how much it’s costing them to scale-out their ruby webapps ;)
As is tradition for many developers stuck at the family home over
xmas. I decided to go hack something.
Asynchronous programming is becoming more popular in all major
languages. C++20 is going to get co_await and friends, python 3.7
now has async, and Rust has async / .await. Rust’s implementation
of Future<T> is quite unique. It uses a “polling”-based interface,
where the listener “polls” for updates but–and this is why I am
making judicious use of quotation marks–polling only occurs when the
asynchronous event source “wakes” the poller, so polling only actually
happens when a state change occurs, rather than continuously.
This pattern can be used to decouple the waiter from the event source
(if necessary) which is important for a high-perf language with strict
threading logic like Rust. By contrast, almost all other
implementations I’ve seen in the wild tend to use callbacks
make it difficult to figure out which thread executes the callback and
how backpressure should be handled.
This polling design can make implementing async Rust from scratch (as
in, without using something like tokio to do it for you) quite
complicated. There’s Future<T>s, Wakers, RawWakerVTables,
etc. etc. to have to implement, and these would be fairly easy to
implement in a GCed language, but Rust also requires that the
implementations are suitably safe, which typically requires a little
unsafe code. I implemented an async system with the help of
this book (high-level
overview) and the source code for futures and tokio. The
experience was eye-opening, but much more complicated than expected.
I went to Houston for ASHG 2019 with PetaGene to demo PetaSuite
Protect, one of the products I’m helping to develop.
Giving tech demos is always a daunting task, especially because we
gave our tech demos completely freeform - typing shell commands in
front of clients is always fun ;). The demos were delivered without a
hitch, though, so there’s something to be said about the effectiveness
of writing bash scripts during a long-haul airplane journey.
One of the more interesting projects I’ve worked on recently is using
emscripten to port PetaGene’s
high-performance decompression suite to wasm so that it can run in a
browser with no installation.
It required figuring out how where to draw the line between having a
emulate synchronous IO (ideal for standard C/C++ applications). It
also required an ill-thought-out
igv.js, which prompted a much
better fix by the maintainer. This is why I like the OSS model: even
bad ideas can prompt a discussion about better ones.
In a previous post I
demoed fo2dat, which can be
used to unpack Fallout 2 DAT2 files. I used the venerable
flate2 crate for that
project, but I’ve since learnt about
libdeflate, which reported
to be a much faster block-based DEFLATE (de)compression library.
libdeflate didn’t have Rust bindings, so I wrote some as a learning
exercise. The result is
exposes a safe Rust API to the library. Benchmarks indicate that the
library is around 2-3x faster than flate2, which is based on zlib
and minizip. That’s a pretty insane speedup for such a popular
PetaGene won best of show for their latest product, PetaSuite Protect
archive). I had a great time at the
event: people were super interested to learn what compression and
encrpytion can do for them. I am looking forward to helping develop
the PetaSuite Protect product :)
After many weekends and evenings of fixing little bugs, cleaning up
the codebase, and polishing the build, I’ve finally managed to publish
v1.0.0 of jobson.
I open-sourced jobson late November 2017. The version I demoed
here was already close
to release-grade in terms of implementation (the server had >200
tests, was used in prod, etc.). However, the deployment, installation, documentation, and maintenance needed work.
For the open-source release, I wanted to make sure that jobson was
OSS-grade before putting a 1.0.0 badge on it. The main changes over
the last year are:
Stabilized all user-facing APIs (CLI, configuration, HTTP). No known
breaking changes since Feb 2018.
Added more systemtests to ensure the above
Reimplemented UI in Typescript
Refactored and cleaned up server code
Fixed various bugs (race conditions, etc.)
Added various features into the specs (better output collection, templating, etc.)
Significantly improved the build, which now builds the full stack
into Debian packages, Docker images, etc.
I plan on patching 1.0.0 slightly with some little annoyances I
spotted (immediately after deploying, of course), followed by another
round of YouTube videos and other media. After that, it’s time to
start slowly chipping away at 1.1.0.
After several days of faffing around with Docker and build systems,
I’ve finally managed to launch a demo’s page here. I’ll
eventually integrate these into my about page, but
they’re at least a useful resource for showing some of the
technologies I’ve worked with.
One useful side-product of this work is that Jobson now has
a basic docker image,
which enables users to boot a pre-integrated Jobson UI + Jobson stack.
tl;dr: I used Rust to make
this CLI utility for
extracting Fallout 1+2 DAT files.
I love the occasional playthrough of Fallout 1 and 2. They were some
of the the first “serious” games I played. Sure, DOOM/Quake/Command &
Conquer were also “mature”—I played them around the age of ~10,
which marked me as doomed (heh) by the kind of adults that would also
think eating sweets is a surefire path to heroin addition or
something—but F1+2 included prostitutes, drugs, slavery, and
infanticide: irresistibly entertaining topics for a teenager.
You might think that, with
Bethesda buying the rights to Fallout over 10 years ago,
F1+2 would’ve had a commercial re-release by now, much like what
happened with Baldur’s Gate 1/2, or
that such a popular franchise with so many fans would create enough
momentum to get an open-source engine ported, but those things haven’t
really happened. Probably because of various historical hitches.
F1+2 were originally developed by Interplay, which had a meteoric rise
in the 90s followed by a precipitous fall in the early 00s
excavated Fallout from Interplay’s remains in 2007. However,
Interplay’s zombie, adopting a 90s zombie movie strategy of having a
character bitten right before the credits roll, decided to give away
F1+2+Tactics for free just before the IP passed over. As a
consequence, Bethesda probably sees a F1+2 reboot as
unprofitable. This assumes that the source code is even available to
Bethesda. Interplay may have only handed them the sales rights +
binaries, which would be a big shame.
I’ve always wanted to make a F1+2 source port, but it’s an incredibly
difficult task for several reasons: the literature for open-source RPG
game engines is thin, which means an engine will need to be built from
first-principles; F1+2 uses a bunch of
bespoke file formats,
which means deserializers will need to be coded for each of them; and
the game logic—what characters do, actions, etc.—is held as a binary
opcode language, which requires a small virtual machine to handle.
developers are the closest to surmounting the F1+2 Everest. They’ve
created something that is close to a complete engine, which is
amazing. I decided to chip away at the smaller problem of F1+2’s file
formats. The result was
fo2dat, a CLI utility for
unpacking F2 DAT files, which might help any developers wanting to
view the game’s assets in a file explorer.
The micro-project was also a chance to try Rust, the new hot thing in
systems programming languages. I enjoyed the experience quite a bit:
it feels like the Rust developers really understood the strengths of
languages like C (non-OO, simple procedures handling data arguments),
C++ (powerful compilers), Scala (pattern matching and abstractions),
and Lisp (code generation). They combined those ideas with an
excellent build system and package manager, which has resulted in a
very nice platform.
tl;dr: If you find you’re spending a lot of time integrating
various pieces of software across multiple computers and are
currently using a mixture of scripts, build systems, and manual
methods to do that, look into configuration managers. They’re easy to
pick up and automate the most common tasks. I’m using
ansible, because it’s standard,
simple, and written in python.
Research software typically requires integrating clusters,
high-performance numerical libraries, 30-year-old Fortran applications
by geniuses, and 30-minute-old python scripts written by PhD
A consistent thorn in my side is downloading, building, installing,
and deploying all of that stuff. For example, on a recent project, I
Checkout a Java (Maven) project from svn
Build it with a particular build profile
Unzip the built binaries
Install the binaries at a specific location on the client machine
Install the binaries at specific location on a cluster
Reconfigure Luigi to run the application with the correct arguments
Copy some other binaries onto the cluster’s HDFS
(Sometimes) rebuild all the binaries from source, if the source was
monkey-patched due to a runtime bug
(Sometimes) Nuke all of the above and start fresh
Each step is simple enough, but designing a clean architecture around
doing slightly different permutations of those steps is a struggle
between doing something the easy way (e.g. a directory containing
scripts, hard-coded arguments in the Luigi task) and doing something
the correct way.
The correct way (or so I thought) to handle these kinds of problems is
to use a build system. However, there is no agreed-upon “one way” to
download, build, and install software, which is why build systems are
either extremely powerful/flexible (e.g. make, where anything is
possible) and rigid/declarative (e.g. maven).
Because there’s so much choice out there, I concluded that researching
each would obviously (ahem) be a poor use of my valuable time. So,
over the years, I’ve been writing a set of scripts which have been
Initially they were bash scripts
Then they were ruby scripts that mostly doing the same as the bash
Then they were ruby scripts that integrated some build parts
(e.g. pulling version numbers out of pom.xml files), but were
mostly doing the same as the bash scripts
Then they were a mixture of structured YAML files containing some
of the build steps and ruby filling in the gaps
Then they were a mixture of YAML files containing metadata
(description strings, version numbers), YAML files containing build
steps, and Python filling in the gaps because Python’s easier to
integrate with the existing researcher/developer’s work
After many months of this, I decided “this sucks, I’ll develop a new,
better, way of doing this”. So I spent an entire evening going
through the weird,
standard build systems out
there, justifying why my solution would be better for this problem.
Well, it turns out this problem isn’t suitable for a build system,
despite it having similar requirements (check inputs, run something,
check outputs, transform files, etc.). Although my searches yielded a
menagerie of weird software, what I actually needed was a
manager. Ansible being a
particularly straightforward one.
This rollercoaster of “there probably isn’t a good solution already
available to this problem”, “I’ll hack my own solution!”, “My hacks
are a mess, I should build an actual system”, “oh, the system already
exists” must be common among software developers. Maybe it’s because
the problem isn’t actually about developing a solution: it’s about
understanding the problem well enough. If the problem’s truly
understood, it will be easier to identify which libraries/algorithms
to use to solve it, which will make developing the solution a lot
easier. Otherwise, you’ll end up like me: Keeper of the Mutant
So, I just spent an evening + morning refactoring the site into a, uh,
I only ocassionally work on this site these days—I now see it as the
sparse journal of a madman that also likes to distribute
mobile-friendly versions of his CV—but I thought it would be a nice
and easy blog post to reflect on how the site has changed over the
last 3 years.
The original version of adamkewley.com was launched in May 2015 and
was the fanciest design:
It makes me feel a bit ill. The first version was modelled off of the
kind of erudite landing pages you see 3-man startups use to try and
sell IoT soap bars or something. By May 2016, I clearly had gotten
sick enough of my own bullshit to remove a bunch of that cute stuff:
By May 2017, there were a few regressions:
And now, in March 2018, I’ve finally decided to just throw almost all
fancy tricks out the window and use the simplest HTML + CSS solution I
libraries and jekyll gems to build. There were also subtle bugs that
would pop up if you used it on a phone. Development/maintenance time
was dedicated to fixing that - I also couldn’t help but tweak with the
This new site is HTML + a ~50 line long CSS file. It felt strangely
liberating to make. Maybe because after working on ReactJS
(e.g.) and angular sites I
came across this absolute
gem that satirically makes the point that barebones sites are: a)
fast, b) easy to make, and c) responsive. I couldn’t argue with the
logic and immediately wanted to just rip out all the complexity in my
site, so here we are.
I’m currently implementing job resubmission in
Jobson UI and found that
state machines greatly simplify the code needed to render a user
A large amount of Jobson UI’s codebase is dedicated to dynamically
generating input forms at runtime.
Generating the relevant <input>, <select>, <textarea>s,
etc. from a Jobson job spec is
fairly easy (see createUiInputhere)
but became increasingly complex after adding job copying because extra
checks needed to be made:
Is the job “fresh” or “based on an existing job”?
Was there a problem loading the existing job?
Did the existing job load OK but can’t be coerced into the live
version of the job spec?
Did the user, on being informed of the coercion issue, decide to
start a fresh spec or make a “best attempt” at coercion?
Each of these conditions are simple to check in isolation but, when
combined, result in delicate state checks:
These checks were cleaned up slightly by breaking things into smaller
components. However, that didn’t remove the top-level rendering
For example, the isLoadingSpecs and errorLoadingSpecs checks can
put into a standalone <SpecsSelector /> component that emits
selectedSpecs. However, the top level component
(e.g. <JobSubmissionComponent />) still needs to decide what to
render based on emissions from multiple child components (e.g. it
would need to decide whether to even render <SpecsSelector /> at
State Machines to the Rescue
What ultimately gets rendered in these kind of workflows depends on a
complex combination of flags because only state, rather than state
and transitions are being modelled. The example above compensates
for a lack of transition information by ordering the if statements:
isLoadingSpecs is checked before isLoadingExistingJob because one
“happens” before the other.
This problem—a lack of transition information—is quite
common. Whenever you see code that contains a big block of if..else
statements, or an ordered lookup table, or a switch on a step-like
enum, that’s usually a sign that the code might be trying to model a
set of transitions between states. Direct examples can be found in
many network data parsers (e.g. websocket frame and HTTP parsers)
because the entire payload (e.g. a frame) isn’t available in one
read() call, so the parser has to handle intermediate parsing
states (example from Java jetty).
State Machines (SMs) represent states and transitions. For example,
here’s the Jobson UI job submission workflow represented by an SM:
From a simplistic point of view, SMs follow simple rules:
The system can only be in one state at a given time
There are a limited number of ways to transition to another state
I initially played with the idea of using SMs in ReactJs UIs after
exploring SM implementations of network parsers. I later found the
idea isn’t new. A similar (ish) post by Jeb Beich has been posted on
and contains some good ideas, but his approach is purer (it’s
data-driven) and is implemented in ClojureScript (which I can’t use
for JobsonUI). By comparison, this approach I used focuses on using
callbacks to transition so that individual states can be implemented
as standard ReactJS components. In the approach:
A state is represented by a component. Components can, as per the
ReactJS approach, have their own internal state, events, etc. but
the top-level state (e.g “editing job”) is represented by that sole
component (e.g. EditingJobStateComponent)
A component transitions to another state by calling a callback with
the next “state” (e.g. transitionTo(SubmittingJobStateComponent)).
A top-level “state machine renderer” is responsible for rendering
the latest component emitted via the callback.
This slight implementation change means that each component only has
to focus on doing its specific job (e.g. loading job specs) and
transitioning to the next immediate step. There is no “top-level”
component containing a big block of if..else statements.
A straightforward implementation involves a top-level renderer with no
decision logic. Its only job is to render the latest component emitted
via a callback:
A state is just a standard component that calls transitionTo when it
wants to transition. Sometimes, that transition might occur
Otherwise, it could be after a set of steps:
Either way, this simple implementation seems to work fine for quite
complex workflows, and means that each components only contains a
limited amount of “transition” logic, resulting in a cleaner codebase.
This pattern could be useful to webdevs that find themselves tangled
in state- and sequence-related complexity. I’ve found SMs can
sometimes greatly reduce overall complexity (big blocks of
if..else, many state flags) at the cost of a little local
complexity (components need to handle transitions).
However, I don’t reccomend using this pattern everywhere: it’s
usually easier to use the standard approaches up to the point of
standard approaches being too complex. If your UI involves a large,
spiralling, interconnected set of steps that pretty much require a
mess of comparison logic though, give this approach a try.
I’ve been running several webservers behind custom domains for a while
textadventurer.tk, and this site)
and it never ceases to amaze me how cheeky bots are getting.
For example, certbot recently started complaining that
textadventurer.tk’s TLS certificate
is about to expire. That shouldn’t happen because there’s a nightly
cronjob for certbot renew.
On SSHing into the server I found an immediate problem: the disk was
full. Why? Because some bot, listed as from
contabotserver.net decided to spam the server
with >10 million requests one afternoon and fill the HTTP
logs. Great. Looks like I’m finally going to implement some log
Then there’s the almost hourly attempts to find a PHPMyAdmin panel on
my sites. That one always surprised me: surely only a small percentage
of PHP sites are misconfigured that badly? Lets look at the stats:
Even if 1 % of them are misconfigured, we’re doomed.
I recently made screencasts that explain Jobson in more detail. The
first explains what Jobson is and how to install it. Overall, Jobson
seems well-recieved. The first video seems to be leveling off at
around 2700 views and Jobson’s
github repo has seen a spike
Will other teams start adopting it or not? Only time will tell.
This is a post about Jobson,
an application I developed and recently got permission to open-source
along with its UI.
Do any of these problems sound familiar to you?:
I’d like my application to have a UI.
I want to trace my application’s usage.
I want to share my application.
They’re very common problems to have, and are almost always solved by
building a UI in a framework (e.g. Qt, WPF), sprinkling on usage
metrics, and packaging everything into a fat executable.
That development approach is becoming challenging to do well nowadays
because clients are more likely to use a mixture of OSs and 3rd-party
resources. Therefore, new projects tend to need to choose between
Develop a CLI application. The easiest thing to develop in many
situations. However, they can be challenging to use and have the
usual client-side distribution problems (installers, etc.).
Develop a UI application. Easier to use use. However, they’re
usually less flexible for developers to use and are more
challenging to develop.
Develop a web API. Has similar flexibility to client-side CLI
applications but doesn’t have the distribution issues. However, web
APIs are challenging to develop because you need to deal with
networking, sysadmin, setting up a domain, authentication, etc.
Develop a full (API + UI) webapp. Easy to use and flexible
(because of the API). However, webapps have all the complexities of
the above and more: you effectively need to develop a full web API
and a working UI.
When scoped in this (biased) way it’s clear that webapps are ideal
vehicles for delivering the full “product” but CLI applications are
ideal for ease-of development and flexibility.
It’s clear that developing a method for turning CLI applications into
webapps would be valuable. It would enable developers to rapidly
develop and roll out applications. That’s what I explored with
Jobson: a web server that
turns CLI applications into webapps.
Jobson has now been working in a production environment for a few
months now and has proven to be an effective vehicle for delivering
new platforms. However it cannot turn any application into a
webapp. That would be tough: the potential “inputs” (applications) and
“outputs” (webapps) are far too varied.
Jobson’s primary limitation is that the only applications it handles
are batch applications that: a) start, b) take typical inputs,
c) write typical outputs, and d) end. Most applications
follow this model (e.g.echo, gcc, nmap, and ls).
With that limitation in place, Jobson could then be designed with
simple principles in mind:
Input applications are explained to Jobson via “specs” (at the
moment, YAML files). The applications themselves should not need to
be changed in order to suit Jobson.
Jobson uses the specs to host a standard HTTP REST API which can be
used by all general programming languages and utilities (e.g. curl
The API is regular and predictable: job specs should always create
the API you’d expect.
Jobson’s authentication uses standard protocols (e.g. HTTP Basic)
that can be integrated into a larger systems easily.
(Default) persistence uses plaintext files on the filesystem, which
allows for transparent debugging and administration.
Persistence can be swapped for other implementations (e.g. a MongoDB
Job execution itself uses the underlying operating system’s
fork(2), and exec(2) syscalls, which allows Jobson to run any
application the host OS can run.
By default, Jobson has no external dependencies beyond Java. It
should be bootable from a barebones linux server with minimal
Overall, this design means that Jobson can webify almost any
application very quickly. I could webify a major CLI tool in less
time than it took to write this post, resulting in a web API and UI
for that tool.
Why It’s Useful
Here are some typical development problems Jobson could help out with:
Problem: You’ve got a cool application idea you want to share
Develop the idea
Develop all the crap necessary to share that idea (download links,
websites, databases, etc.)
Develop the idea
Write a Jobson spec
Problem: You’ve got a toolchain (e.g. a pentesting toolchain) that
has hard-to-remember commands. You want to streamline that toolchain
such that you can run and queue multiple commands.
Develop a UI around the tools, dealing with the usual development
Use the tool
Write a few small scripts around the tools (if necessary)
Problem: You want to automate data requests at work, but there’s
a lot of flexibility in the requests (FYI: this is why Jobson was
Write an application to handle typical data requests. The
application will have to deal with request queues, request
Find that end-users suddenly want to make different requests
Write short scripts for handling a data request
Host it with Jobson, which automatically deals with all the other
Find that end-users suddenly want to make different requests
Add a new script + spec into Jobson
Overall, I believe this approach to developing a job system is
extremely flexible and much easier to work with. Jobson abstracts
all the typical job system faff (auth, APIs, queues, etc.) away from
what’s actually important (an application that does something),
resulting in a much cleaner system.
The next steps with Jobson are to streamline installation (by
implementing installers), add a landing+documentation page, and start
making tutorial videos for it. Watch this space =)