A small project I developed over xmas 2018 to produce this Arduino-based device. [Source Code (GitHub)]
A small project I developed over xmas 2018 to produce this Arduino-based device. [Source Code (GitHub)]
My current employment victim, PetaGene, has just scored $2.1 M in funding. Great news for an amazing team!
After many weekends and evenings of fixing little bugs, cleaning up
the codebase, and polishing the build, I’ve finally managed to publish
jobson late November 2017. The version I demoed
here was already close
to release-grade in terms of implementation (the server had >200
tests, was used in prod, etc.). However, the deployment, installation, documentation, and maintenance needed work.
For the open-source release, I wanted to make sure that
OSS-grade before putting a
1.0.0 badge on it. The main changes over
the last year are:
I plan on patching
1.0.0 slightly with some little annoyances I
spotted (immediately after deploying, of course), followed by another
round of YouTube videos and other media. After that, it’s time to
start slowly chipping away at
After several days of faffing around with Docker and build systems, I’ve finally managed to launch a demo’s page here. I’ll eventually integrate these into my about page, but they’re at least a useful resource for showing some of the technologies I’ve worked with.
One useful side-product of this work is that Jobson now has a basic docker image, which enables users to boot a pre-integrated Jobson UI + Jobson stack.
tl;dr: I used Rust to make this CLI utility for extracting Fallout 1+2 DAT files.
I love the occasional playthrough of Fallout 1 and 2. They were some of the the first “serious” games I played. Sure, DOOM/Quake/Command & Conquer were also “mature”—I played them around the age of ~10, which marked me as doomed (heh) by the kind of adults that would also think eating sweets is a surefire path to heroin addition or something—but F1+2 included prostitutes, drugs, slavery, and infanticide: irresistibly entertaining topics for a teenager.
You might think that, with Bethesda buying the rights to Fallout over 10 years ago, F1+2 would’ve had a commercial re-release by now, much like what happened with Baldur’s Gate 1/2, or that such a popular franchise with so many fans would create enough momentum to get an open-source engine ported, but those things haven’t really happened. Probably because of various historical hitches.
F1+2 were originally developed by Interplay, which had a meteoric rise in the 90s followed by a precipitous fall in the early 00s (details). Bethesda excavated Fallout from Interplay’s remains in 2007. However, Interplay’s zombie, adopting a 90s zombie movie strategy of having a character bitten right before the credits roll, decided to give away F1+2+Tactics for free just before the IP passed over. As a consequence, Bethesda probably sees a F1+2 reboot as unprofitable. This assumes that the source code is even available to Bethesda. Interplay may have only handed them the sales rights + binaries, which would be a big shame.
I’ve always wanted to make a F1+2 source port, but it’s an incredibly difficult task for several reasons: the literature for open-source RPG game engines is thin, which means an engine will need to be built from first-principles; F1+2 uses a bunch of bespoke file formats, which means deserializers will need to be coded for each of them; and the game logic—what characters do, actions, etc.—is held as a binary opcode language, which requires a small virtual machine to handle.
The Falltergeist developers are the closest to surmounting the F1+2 Everest. They’ve created something that is close to a complete engine, which is amazing. I decided to chip away at the smaller problem of F1+2’s file formats. The result was fo2dat, a CLI utility for unpacking F2 DAT files, which might help any developers wanting to view the game’s assets in a file explorer.
The micro-project was also a chance to try Rust, the new hot thing in systems programming languages. I enjoyed the experience quite a bit: it feels like the Rust developers really understood the strengths of languages like C (non-OO, simple procedures handling data arguments), C++ (powerful compilers), Scala (pattern matching and abstractions), and Lisp (code generation). They combined those ideas with an excellent build system and package manager, which has resulted in a very nice platform.
tl;dr: If you find you’re spending a lot of time integrating various pieces of software across multiple computers and are currently using a mixture of scripts, build systems, and manual methods to do that, look into configuration managers. They’re easy to pick up and automate the most common tasks. I’m using ansible, because it’s standard, simple, and written in python.
Research software typically requires integrating clusters, high-performance numerical libraries, 30-year-old Fortran applications by geniuses, and 30-minute-old python scripts written by PhD students.
A consistent thorn in my side is downloading, building, installing, and deploying all of that stuff. For example, on a recent project, I needed to:
Each step is simple enough, but designing a clean architecture around doing slightly different permutations of those steps is a struggle between doing something the easy way (e.g. a directory containing scripts, hard-coded arguments in the Luigi task) and doing something the correct way.
The correct way (or so I thought) to handle these kinds of problems is
to use a build system. However, there is no agreed-upon “one way” to
download, build, and install software, which is why build systems are
either extremely powerful/flexible (e.g.
make, where anything is
possible) and rigid/declarative (e.g.
Because there’s so much choice out there, I concluded that researching each would obviously (ahem) be a poor use of my valuable time. So, over the years, I’ve been writing a set of scripts which have been gradually mutating:
pom.xmlfiles), but were mostly doing the same as the bash scripts
After many months of this, I decided “this sucks, I’ll develop a new, better, way of doing this”. So I spent an entire evening going through the weird, wonderful, and standard build systems out there, justifying why my solution would be better for this problem.
Well, it turns out this problem isn’t suitable for a build system, despite it having similar requirements (check inputs, run something, check outputs, transform files, etc.). Although my searches yielded a menagerie of weird software, what I actually needed was a configuration manager. Ansible being a particularly straightforward one.
This rollercoaster of “there probably isn’t a good solution already available to this problem”, “I’ll hack my own solution!”, “My hacks are a mess, I should build an actual system”, “oh, the system already exists” must be common among software developers. Maybe it’s because the problem isn’t actually about developing a solution: it’s about understanding the problem well enough. If the problem’s truly understood, it will be easier to identify which libraries/algorithms to use to solve it, which will make developing the solution a lot easier. Otherwise, you’ll end up like me: Keeper of the Mutant Scripts.
So, I just spent an evening + morning refactoring the site into a, uh, “new” design.
I only ocassionally work on this site these days—I now see it as the sparse journal of a madman that also likes to distribute mobile-friendly versions of his CV—but I thought it would be a nice and easy blog post to reflect on how the site has changed over the last 3 years.
The original version of adamkewley.com was launched in May 2015 and was the fanciest design:
It makes me feel a bit ill. The first version was modelled off of the kind of erudite landing pages you see 3-man startups use to try and sell IoT soap bars or something. By May 2016, I clearly had gotten sick enough of my own bullshit to remove a bunch of that cute stuff:
By May 2017, there were a few regressions:
And now, in March 2018, I’ve finally decided to just throw almost all fancy tricks out the window and use the simplest HTML + CSS solution I could create:
This new site is HTML + a ~50 line long CSS file. It felt strangely liberating to make. Maybe because after working on ReactJS (e.g.) and angular sites I came across this absolute gem that satirically makes the point that barebones sites are: a) fast, b) easy to make, and c) responsive. I couldn’t argue with the logic and immediately wanted to just rip out all the complexity in my site, so here we are.
I wonder how quickly regressions will set in ;)
I’m currently implementing job resubmission in Jobson UI and found that state machines greatly simplify the code needed to render a user workflow.
A large amount of Jobson UI’s codebase is dedicated to dynamically generating input forms at runtime.
Generating the relevant
etc. from a Jobson job spec is
fairly easy (see
but became increasingly complex after adding job copying because extra
checks needed to be made:
Each of these conditions are simple to check in isolation but, when combined, result in delicate state checks:
These checks were cleaned up slightly by breaking things into smaller components. However, that didn’t remove the top-level rendering decisions altogether.
For example, the
errorLoadingSpecs checks can
put into a standalone
<SpecsSelector /> component that emits
selectedSpecs. However, the top level component
<JobSubmissionComponent />) still needs to decide what to
render based on emissions from multiple child components (e.g. it
would need to decide whether to even render
<SpecsSelector /> at
What ultimately gets rendered in these kind of workflows depends on a
complex combination of flags because only state, rather than state
and transitions are being modelled. The example above compensates
for a lack of transition information by ordering the
isLoadingSpecs is checked before
isLoadingExistingJob because one
“happens” before the other.
This problem—a lack of transition information—is quite
common. Whenever you see code that contains a big block of
statements, or an ordered lookup table, or a
switch on a step-like
enum, that’s usually a sign that the code might be trying to model a
set of transitions between states. Direct examples can be found in
many network data parsers (e.g. websocket frame and HTTP parsers)
because the entire payload (e.g. a frame) isn’t available in one
read() call, so the parser has to handle intermediate parsing
states (example from Java jetty).
State Machines (SMs) represent states and transitions. For example, here’s the Jobson UI job submission workflow represented by an SM:
From a simplistic point of view, SMs follow simple rules:
I initially played with the idea of using SMs in ReactJs UIs after exploring SM implementations of network parsers. I later found the idea isn’t new. A similar (ish) post by Jeb Beich has been posted on cogninet here and contains some good ideas, but his approach is purer (it’s data-driven) and is implemented in ClojureScript (which I can’t use for JobsonUI). By comparison, this approach I used focuses on using callbacks to transition so that individual states can be implemented as standard ReactJS components. In the approach:
A state is represented by a component. Components can, as per the
ReactJS approach, have their own internal state, events, etc. but
the top-level state (e.g “editing job”) is represented by that sole
A component transitions to another state by calling a callback with
the next “state” (e.g.
A top-level “state machine renderer” is responsible for rendering the latest component emitted via the callback.
This slight implementation change means that each component only has
to focus on doing its specific job (e.g. loading job specs) and
transitioning to the next immediate step. There is no “top-level”
component containing a big block of
A straightforward implementation involves a top-level renderer with no decision logic. Its only job is to render the latest component emitted via a callback:
A state is just a standard component that calls
transitionTo when it
wants to transition. Sometimes, that transition might occur
Otherwise, it could be after a set of steps:
Either way, this simple implementation seems to work fine for quite complex workflows, and means that each components only contains a limited amount of “transition” logic, resulting in a cleaner codebase.
This pattern could be useful to webdevs that find themselves tangled
in state- and sequence-related complexity. I’ve found SMs can
sometimes greatly reduce overall complexity (big blocks of
if..else, many state flags) at the cost of a little local
complexity (components need to handle transitions).
However, I don’t reccomend using this pattern everywhere: it’s usually easier to use the standard approaches up to the point of standard approaches being too complex. If your UI involves a large, spiralling, interconnected set of steps that pretty much require a mess of comparison logic though, give this approach a try.
For example, certbot recently started complaining that
textadventurer.tk’s TLS certificate
is about to expire. That shouldn’t happen because there’s a nightly
On SSHing into the server I found an immediate problem: the disk was full. Why? Because some bot, listed as from contabotserver.net decided to spam the server with >10 million requests one afternoon and fill the HTTP logs. Great. Looks like I’m finally going to implement some log compression+rotation.
Then there’s the almost hourly attempts to find a PHPMyAdmin panel on my sites. That one always surprised me: surely only a small percentage of PHP sites are misconfigured that badly? Lets look at the stats:
Even if 1 % of them are misconfigured, we’re doomed.
I recently made screencasts that explain Jobson in more detail. The first explains what Jobson is and how to install it. Overall, Jobson seems well-recieved. The first video seems to be leveling off at around 2700 views and Jobson’s github repo has seen a spike in attention.
Will other teams start adopting it or not? Only time will tell.
Do any of these problems sound familiar to you?:
I’d like my application to have a UI.
I want to trace my application’s usage.
I want to share my application.
They’re very common problems to have, and are almost always solved by building a UI in a framework (e.g. Qt, WPF), sprinkling on usage metrics, and packaging everything into a fat executable.
That development approach is becoming challenging to do well nowadays because clients are more likely to use a mixture of OSs and 3rd-party resources. Therefore, new projects tend to need to choose between several options:
Develop a CLI application. The easiest thing to develop in many situations. However, they can be challenging to use and have the usual client-side distribution problems (installers, etc.).
Develop a UI application. Easier to use use. However, they’re usually less flexible for developers to use and are more challenging to develop.
Develop a web API. Has similar flexibility to client-side CLI applications but doesn’t have the distribution issues. However, web APIs are challenging to develop because you need to deal with networking, sysadmin, setting up a domain, authentication, etc.
Develop a full (API + UI) webapp. Easy to use and flexible (because of the API). However, webapps have all the complexities of the above and more: you effectively need to develop a full web API and a working UI.
When scoped in this (biased) way it’s clear that webapps are ideal vehicles for delivering the full “product” but CLI applications are ideal for ease-of development and flexibility.
It’s clear that developing a method for turning CLI applications into webapps would be valuable. It would enable developers to rapidly develop and roll out applications. That’s what I explored with Jobson: a web server that turns CLI applications into webapps.
Jobson has now been working in a production environment for a few months now and has proven to be an effective vehicle for delivering new platforms. However it cannot turn any application into a webapp. That would be tough: the potential “inputs” (applications) and “outputs” (webapps) are far too varied.
Jobson’s primary limitation is that the only applications it handles
are batch applications that: a) start, b) take typical inputs,
c) write typical outputs, and d) end. Most applications
follow this model (e.g.
With that limitation in place, Jobson could then be designed with simple principles in mind:
Input applications are explained to Jobson via “specs” (at the moment, YAML files). The applications themselves should not need to be changed in order to suit Jobson.
Jobson uses the specs to host a standard HTTP REST API which can be
used by all general programming languages and utilities (e.g.
The API is regular and predictable: job specs should always create the API you’d expect.
Jobson’s authentication uses standard protocols (e.g. HTTP Basic) that can be integrated into a larger systems easily.
(Default) persistence uses plaintext files on the filesystem, which allows for transparent debugging and administration.
Persistence can be swapped for other implementations (e.g. a MongoDB implementation)
Job execution itself uses the underlying operating system’s
exec(2) syscalls, which allows Jobson to run any
application the host OS can run.
By default, Jobson has no external dependencies beyond Java. It should be bootable from a barebones linux server with minimal configuration.
Overall, this design means that Jobson can webify almost any application very quickly. I could webify a major CLI tool in less time than it took to write this post, resulting in a web API and UI for that tool.
Here are some typical development problems Jobson could help out with:
Problem: You’ve got a cool application idea you want to share
Problem: You’ve got a toolchain (e.g. a pentesting toolchain) that has hard-to-remember commands. You want to streamline that toolchain such that you can run and queue multiple commands.
Problem: You want to automate data requests at work, but there’s a lot of flexibility in the requests (FYI: this is why Jobson was made).
Overall, I believe this approach to developing a job system is extremely flexible and much easier to work with. Jobson abstracts all the typical job system faff (auth, APIs, queues, etc.) away from what’s actually important (an application that does something), resulting in a much cleaner system.
The next steps with Jobson are to streamline installation (by implementing installers), add a landing+documentation page, and start making tutorial videos for it. Watch this space =)
I’ve been distracted by other things going on, but I finally managed to spend an evening or two adding actual games into textadventuer.
The following games were added:
One thing the game search did for me was find similar platforms. I should’ve looked before hacking away at textadventuer because some of those platforms (e.g. http://textadventures.co.uk/) are very well made. Some even use clever tricks like emulating DosBox in the browser. However, I’m yet to come across a platform that can run any application via a browser (textadventuer’s “USP”, if you will), so hopefully other developers will find the server and UI source helpful if they have an idea along those lines.
One of the first things people learn when they start programming is how to write a text prompt. It’s a decades-old exercise that teaches new programmers input-output.
In order to inject a little excitement, learners are normally
encouraged to write interactive games using standard IO. This helps
them learn programming by interactively - they will need to learn
conditional logic to handle the “
Will you stab the monster with your
sword or run away?” prompt.
An ideal system would allow text adventures (console applications) to be written in any language but also be distributed on the web. This is what my latest project, textadventurer, tries to achieve (gallery).
textadventurer keeps interaction in the browser while moving
execution onto the server. Communication between those two layers is
achieved with websockets. This makes it easier for people to play the
game. The frontend focuses on presenting games (CLI applications) to
the players and provides a basic UI for sending input (STDIN) to the
server and recieving output (STDOUT) from the server.
Using process forking server-side affords a lot of flexibility: the
server is completely agnostic to the language or framework that the
game is written in. This means I can use
distribute any standard interactive CLI application. With that in
mind, I plan on deploying historic text adventure games to
textadventurer when I get the chance
so that people can enjoy those games once more without having to faff
around with installers, legacy interpreters, etc.
PP is missing many features I would of liked to have. However, a few weeks ago I took a long look in the mirror and decided to just tidy it then throw it up on the net. This let me start sharing it on sites such as reddit to gauge if anyone’s interested. After about a week of PP being up, I concluded that no, no one is interested and yes, I spent way too much time coming to that conclusion.
This is my attempt to articulate general some general rules of thumb I have picked up over the last few side-projects. It was written just after PP, one of my bigger projects, didn’t live up to my expectations. It is not a definitive guide nor something you should take seriously without reading other developers’ thoughts.
I used to approach software side projects as an opportunity to write full-blown applications that I might someday commercialize or turn into a big hit. This, I’ve learnt, is a silly approach. Side projects should be approached as side projects, not as full-blown projects.
If a side project requires a light scripting engine, command controller, custom UI elements, a fair amount of interactivity, servers, etc. then it’s doomed to failure. This isn’t because these features can’t be implemented—I believe any developer can implement any feature if given enough time—but because big projects don’t suit only getting “and hour here” and “an evening there”.
The problem with PP is that I specced my minimum viable product way too ambitiously. The spec included several headline features:
In isolation, each of these features are easy to implement. However, implementing all of them into one system requires more care and, crucially, architecture.
If you write software full-time as a job then you can afford to spend a month or two making prototypes, designing hierarchies, and, ultimately, implementing a system that’s quite large. However, side-projects won’t get more than, say, ~20 % of the time an at-work project will get. Because of that, those few weeks of prototyping and planning will stretch out across months.
In principle, many months isn’t a problem for a side-project with no time limits. However, spending many months without making anything requires a lot of discipline: no one finds planning and prototyping as fun as diving into the heart of an idea. The same goes for boring old architecture documents, automated testing, and design specs.
Those things mostly exist to make working on bigger projects more predictable and to make the maintenance of a working system easier for other developers. A small side project is only going to be worked on by one, maybe two, developers over a relatively short period of time; therefore, it isn’t worth worrying about the theoretical overhead of maintaining and sharing it. If a small side project becomes an overnight sensation then—because it’s small—there is scope for dropping the initial codebase and starting again with all the knowledge and expertise the first runthrough gave you.
The main take home is that side projects get developed much more slowly than real on-the-job projects because of time constraints. That makes working on bigger side projects especially painful because the boring planning, speccing, and testing steps—which tend to come in handy for bigger projects—last too long to keep the momentum of a side project going. By contrast, smaller projects work well with a “code and fix” development model which gives immediate feedback and a semi-working system very quickly. This helps maintain momentum and interest; however, “code and fix” gets treacherous once projects start growing, so don’t let them grow.
Keep small. Keep agile. Try things out. Get the audience’s opinion. Then grow.
I haven’t posted in a while because I have been busy with a few random studies and projects.
Low level programming has always, to a high-level programmer like myself, felt like a mysterious playground where all the real hackers write graphics engines, password crackers, and operating systems. I have always had a fascination with people who can write something hardcore like an emulator. Sure, I can open a window, draw some graphics, and update the UI to reflect state changes. However, an application that does all that and some low-level wizardry is in an entirely different league of complexity.
Many prominent programmers recommend learning something like C because it is a good way of learning how computers really work. Apart from the occasional toe-dip into C, I have almost exclusively coded in higher level languages.
I want to get better at programming. I’m getting bored of the high-level: there’s only so many ways classes, functions, currying, type systems, can be repackaged before you’ve seen it all. I like the idea of mastering the low level. However, deciding on where to start was challenging.
Last weekend, however, I decided to set myself what I thought would be a simple exercise:
Write a websocket server that capitalizes and echoes any text it recieves. You cannot use a websocket library.
I eventually hacked something together that works (github). I took a shortcut and used the nodejs http-parser and a base64 encoder written by Brad Conte. However, the websocket implementation was derived directly from RFC6455.
Even though it’s pathetically basic, is blocking (I don’t
fork or do
any fancy event queueing on the connections), and uses compile-time
allocated buffers for most of the messages (so it can’t stream big
websocket frames), this was still a very eye-opening first step into
some low-level programming. I somewhat regret writing the process
text adventurer in
clojure (with the help of
websocketd) after seeing
how straightforward a lower-level implementation would be.
It’s tempting to believe that having a flexible file format for a system’s configuration is enough to ensure forward-compatibility and flexibility as the system grows. The thought process is that with a suitable data format—for example, XML or JSON—extra information can be added without breaking existing systems. For adding new fields into the data, this is true. However, if you don’t engineer additional flexibility into the fields themselves then you may end up needing to make breaking changes anyway or, worse, end up with a much more complicated data file.
Imagine you are implementing an ecommerce server that exposes
products. The spec for the server stipulates that there will only be a
very small number of products. It also stipulates that rebooting the
server to refresh the system with new products is fine. With that in
mind, you add a
products field to the server’s configuration:
One day—and this day always comes—a change to the specification
comes: some products should be hidden on the site. To implement that
feature, you add a
hidden: field to the product. It is assumed that
hidden: field is not defined for a product then it should
false. This is backwards-compatible and easy to implement
given the extensible YAML format being used:
Later on, it emerges that product-hiding should not be binary. It
should depend on the group that the user belongs to. To implement
that, you add a
hidden_to: field. However, to allow for backwards
compatibility, it is enforced that
hidden_tos entries take priority
hidden state. Side-rules are beginning to creep in:
As the system gains popularity, it becomes increasingly necessary to
be able to add products to a running system. It is decided that
products should be held in a database rather being loaded from the
configuration file. The configuration file, therefore, should support
providing database details as an alternative to
backwards compatibility should still be maintained to prevent legacy
deployments and tests from breaking:
A heuristic is adopted that if
use_database has a value of
then the server should load products via the database defined in
database; otherwise, it should load them via
The sysadmin then mentions that it would be useful to allow the server to listen on multiple ports, some of them being HTTP and some of them being HTTPS. Marketing also wants to integrate email updates into the server. However, one week into developing email updates, they also mention that some systems may need to send tweets instead of emails. Your implementation strategy follows the same logic as previous development efforts:
This process of introducing additional data that is handled by special-case heuristics is a common pattern for extending configurations without breaking existing fields. However, it can become quite difficult to keep track of all the different rules that arise out of this gradual growth in complexity. Not only that, the resulting configuration file can be difficult to work with.
What would make this process a lot easier would be to extend the
actual fields themselves. However
products, for example, is defined
to contain an array of products. The only actions that can be carried
products are to add or remove products.
The solution to this is to to ensure that there are extension points at major parts of the configuration hierarchy when it’s initially designed. Relegate primitive values (which can’t be extended later) deeper into the configuration hierarchy and have clear extension points.
An example of adding extension points would be:
In this design
product (within product_data), and
could all be extended later with more type switches or other
information to allow them to be loaded/handled differently. Applying
this nesting concept to the configuration at each step of the above
scenario would result in something like this:
There would still be some heuristics involved with handling missing
keys, defaults, type switches, etc. However, the hierarchy between
features is maintained and there are now no sibling heuristics. As a
result, the config-handling code can be modularized to then read each
key in a configuration in isolation. The
products in this improved
example could be handed to
.handleProductsConfiguration(ProductsConfiguration conf), a function
that now doesn’t also need to receive sibling
A strict tree hierarchy maps cleanly onto polymorphic systems and is
surprisingly easy to implement using libraries such as
The above is compile-time verified, type-safe, and involves no
conditional logic. Completely new product sources can be configured by
defining a configuration that implements
updating the relevant visitor. Clearly, this implementation requires
more up-front engineering than the original design. However, for
projects with changing specification (i.e. most projects) allowing for
flexibility pays for itself.
In programming, “mature” is a funny word. When I first started programming, it had negative connotations. A “mature” programming language was boring, had complicated tools, over-complicated design patterns, and would likely die off soon. By contrast, non-mature languages were exciting, free-form, and carving out a new frontier in software.
My opinion has evolved since working as a programmer rather than studying as one. Mature now implies that a language will have have good debuggers and profilers. It will have an established build and testing systems for the language. Its core libraries won’t change midway through a critical project. Most importantly, though, mature languages (tend to) have moved past the same boring old problems that every language ecosystem encounters in its early days.
write a few HTML tags, a
<script> tag, and voila I could see stuff
happening in my browser that my code put there. Simple.
Unfortunately, it’s unlikely that my code would work in any other
fragmented. Internet explorer’s implementation of browser
functionality was non-standard. Even the most trivial DOM manipulation
or standard library call would behave differently between IE and
Firefox. It was not unusual to have a lot of
if(browser === IE) tests
Because of this stability, web developers could build bigger, more complex, sites. Animations, realtime updates, and dashboards became more popular. As a consequence, tools for debugging this complexity became more valuable. In an effort to secure their market share, Chrome and Firefox began to include developer tools which enabled developers to set breakpoints and profile their websites. Having these tools made web development a lot nicer; however, the workflow was now more complex.
The next issue developers encountered was managing libraries and
dependencies. Traditionally, libraries would be manually downloaded
and included in
<script src="..."> tags in the page. However, it was
tedious to do this once projects became split over many files. It was
also inefficient because the HTTP/1.1 protocol doesn’t suit
downloading many small files.
Backend frameworks, notably Rails, began to offer asset pipelines for combining, minifying, and including scripts in websites. This enabled developers to build bigger codebases fragmented over many files; however, they would also need to learn how their build system links files and use appropriate strategies to debug minified files.
continued to up the ante, creating more complex web
at the edges under this new complexity. Managing and installing the
ever-increasing glut of project dependencies became tedious.
npm raced to fill the gap, allowing developers to automatically
acquire and install dependencies. It became much easier to install
Now that developers could install many libraries easily and
painlessly, they installed many libraries easily and painlessly. The
module-by-module basis. Consequently, dependency hell
ensued. Libraries such as
requirejs came along to fix the problem,
allowing developers to declaratively state a module’s
dependencies. Developers no longer had to reorder
<script> tags to
make the application work; however, they did need to learn requirejs.
for doing stuff that the language ought to support. ES6 (/ ES2015+)
was specced out to deal with some of those issues. It offered
standardized higher-level language abstractions for building bigger
systems (classes, getters, setters, proxies), language-level
protections (const, Symbol), and declarative library loading
import). However, it also made the language a little more complex.
debugging and profiling tools, batteries-included libraries for
standard operations, a popular dependency manager (
module-level dependency management to prevent namespace pollution
import). It finally has all the features every other old-fogie
However, if I was starting out today, I wouldn’t use
akewley-lang. It’s amazing. You
can open a text editor, write
plot my data as a bar chart please and
see stuff happening right away. It’s so simple. Now, all I need is to
generate that bar chart and have a whizzbang animation when it
loads! Luckily, there’s a library for that on the
community forum I can download.
akewley-lang is so much simpler
of those complicated languages (and associated ecosystems) didn’t know
what they were doing.
Software development can be a bottomless well of languages, build environments, platforms, coding standards, and cat analysis algorithms. For someone who likes to study and learn new techniques, this is great. A bottomless well of stuff to better at. However, after almost ten years of doing just that, I’m willing to make a bold statement: curiosity can be maddening and knowledge can be hindrance.
Curiosity is maddening because the more I learn about assembler, linux, garbage collection, compiler theory, engineering techniques, software principles, functional programming, networking, etc. The more I’m convinced that I know almost nothing about computers.
Every time I turn over a figurative “small pebble” in the software world I end up discovering that there’s entire industries dedicated to what’s under that pebble. Even a statement as innocuous as “I’d love to know how my CPU actually sends input requests to this USB hard drive” - can yield a very long (but interesting) explanation.
Complexity isn’t unique to software development. However, combined with curiosity, software is one of the few fields where there is practically no inertia to ping-ponging. If I like the look of writing client side 3D games in C++, I can be doing it in under 20 minutes for free. 30 minutes later I can be writing a plugin for firefox that makes it purr whenever I visit a website with “cat” in the URL.
You’d think that healthy curiosity is a good thing: over time, it grows knowledge. However, knowledge can become a hindrance if you leverage it a little too much. A simple “Hello World!” application can scream out with potential pitfalls - do we have standard IO? What about cross-platform? What about internationalization? Is there a more flexible design for delivering “Hello” and “World” to the console? What character encoding is being used?
This, I think, is why there’s a significant amount of very young, inexperienced, software developers making interesting—sometimes very valuable—products in the field. They don’t know that their servers will implode once they have a few thousand concurrent connections. They don’t know that it’s going to be very painful to refactor that UI they hacked together with four bleeding-edge UI frameworks. They don’t know that there’s a big competitor in the market with a lot of money. They shoot first and ask questions later, which makes them one of the few demographs doing the shooting.
We’re all aware of this phenomenon. What I’m saying isn’t news. However, here I am, writing heavily designed, SOLID certified, high unit-test coverage software that few people use. Meanwhile, there’s teenagers out there hacking together VR experiences in which I get to fly around space as Thomas the tank engine.
What’s missing? Isn’t the reason we get curious, study, and train so that we can make even cooler stuff? Yes. However, as presented, curiosity is maddening and knowledge can be a hindrance. The trick, I’m slowly convincing myself, is to have a cool and interesting problem or project to apply that curiosity and knowledge to and not get hung up on the details. Otherwise, you may end up spending hours analyzing the random noise of a bingo game.
So, with that in mind, over the next year, I’m going to be making a conscious effort to just “make stuff” and care alot less about how philosophically pure its software engineering is or what amazing 0.5 % speed bonus I’ll get if I change to some slightly newer framework half way though.
Now, all I need to do is write a design document, get people to approve on it, do a technology review, select a programming language, ah yes, mutability is evil! It’s the reason I haven’t created a decent app yet, I should look into Haskell! Ok, now possible test harnesses, oh I wonder what’s the difference between this test harness and this other very similar test harness, that could really have a big impact on my project and I don’t want to make an error before I even start…