I occasionally have to write HTTP clients to handle third-party APIs. One thing that really bugs me is when useful HTTP APIs have additional custom authentication and encryption. Custom encryption is especially annoying when SSL could’ve been used instead.

Normal APIs

Here is an example of a great api to develop against:

  • Make a HTTP GET request to https://server.com/api/users/1?format=json to get a user’s profile details in a json format.

You can perform this request with any generic HTTP tool (e.g. curl). However, the API doesn’t mention authentication. This is usually where developers split hairs. The standard way of authenticating a connection is to use cookies to store login state:

  • Make a HTTP POST request to https://server.com/api/sessions/create with a content body containing your username and password. If successful, the server will respond with a status code of 200. A successful request will receive a response containing a Cookie-Set header. Subsequent requests to the server can then be authenticated by attaching this cookie data to those requests.

Some developers might be annoyed at how tediously stateful this is, so they might instead opt for using login tokens:

  • Make a POST request to https://server.com/api/sessions/create with a content body containing your username and password. If successful, the server will respond with a status code of 200. A successful request will be responded to with a content body containing a unique login token. Subsequent requests to the server can then be authenticated by attaching this login token to those requests.

Perfectly stateless and REST compliant. APIs like this are also fine and are usually required if you need to, for example, circumvent browser’s cross-domain protection.

An Example Custom API

Below are (abstract) instructions for the latest set of API I’m currently coding against. This API will likely be consumed by a wide variety of programming languages and frameworks. Spot why designing clients for this API might be time consuming:

  • Email the API’s developers to receive a public api key (UTF-8) and secret key (base64).
  • All requests must be SSL encrypted. However, the institute does not use a certificate that is verified by a popular certificate authority, so SSL signature certification must be disabled for all requests to the API unless you’re willing to manually install the necessary certificates.
  • Apart from requests to https://server.com/api/public-key, all requests require these preparation steps:
    • You must attach your public api-key to each request as an api-key HTTP header
  • The request type (GET, PUT, POST), current time timestamp (custom format, see instructions below), request path, and request parameters must be concatenated with unix newlines (\n) and be UTF-8 encoded. This concatenated string is then encrypted using the HMAC256 algorithm and secret key. The resulting ciphertext bytestream is base64 encoded and attached to requests as an api-signature HTTP header
  • Make a HTTP GET request to https://server.com/api/public-key to receive a base64-encoded exponent and modulus
  • Use the exponent and modulus to encrypt your password using the RSA encryption algorithm. Pre-encryption, the password must be UTF-8 encoded. Post-encryption, the output must be base64 encoded.
  • Make a HTTP POST request to https://server.com/api/sessions/create supplying a fully (windows domain) qualified username, encypted password, and api key in the content. Remember to sign your request as described above (HMAC256, etc.).
  • Successful requests will receive a response containing a api-usertoken and expiry in the content. The expiry timestamp is in a special non-standard format (see instructions below). Subsequent requests to the api authenticated by attaching the api-usertoken to the header (in addition to the signing steps, as described above).
  • Expired api-usertokens, invalid HMAC signed api-signatures, and invalid login credentials will be met with a blanket 401 (Unauthorized) status code and no additional feedback
  • Timestamps for the api-usetoken’s expiry and HMAC signing have the format YYYY-MM-DD hh:mm:ss.fffZ. Note: timestamps within the server’s response use a standard (UTC) timestamp though

Sounds simple enough, but it’s actually quite difficult to do all steps perfectly when the server is a black box—you will only receive a “fail” until your client’s implementation is perfect.

Client development to one side, most of these steps exist because the API doesn’t use a standard SSL certificate and, as a result, SSL’s in-built protection against man-in-the-middle (MITM) exploits is nullified.

If this API had a certificate, most of these design features to prevent MITMs would be eradicated. Unfortunately, because the API is missing a crucial component of SSL we have this API. The issue is, though, that it’s actually quite difficult to prevent MITM exploits if you don’t use a pre-shared key, and that’s where this particular API begins to unravel.

Hacking the API

Here is how you could MITM this API to get user’s domain passwords. Users’ domain accounts are used to access emails, HR details, calendar, payroll, etc. This hack assumes that certificate authentication has been disabled, which will likely be the case for implementations that can’t afford to require that users know how to install CAs (e.g. widely distributed line-of-business software):

  • Create a standard HTTP proxy that intercepts HTTP traffic between a client and the api server
  • Create a forged SSL key pair
  • SSL intercept traffic between clients and the server. In effect, give your forged public key to a client as-per the SSL handshake, decrypt any traffic they send to you with your forged private key (to intercept it), and re-encrypt the request using the API server’s public key to complete the proxy chain
  • For most traffic, proxy as normal. You can read any requests. Also, because the HMAC protection only prevents altering the path & parameters, you can alter POST/PUT request body’s at will. That is, you could can completely hijack legitimate POST/PUT requests.
  • For HTTP GET requests to https://server.com/api/public-key substitute the API’s public RSA exponent/modulus pair for your own forged pair. Store the server’s legitimate public key.
  • Intercept HTTP POST requests to https://server.com/api/sessions/create. These requests will contain an unencypted username and a password that was encrypted with your forged public exponent/modulus
  • Decrypt the password using your private key
  • You now have a user’s full domain username + password
  • Forward the HTTP POST request to the server, RSA encrypting the password using the server’s key. This will complete the proxy chain, which will make the API access seamless - it will be very annoying to figure out why people’s accounts are being hacked

The only thing you’ll have trouble hacking is anything covered by the HMAC encryption (the path + URL parameters). This is because the api-key/secret-key is essentially a pre-shared key (PSK) arrangement that can’t be easily MITMd. This begs the question why you also need user authentication in addition to this key pair because the secret-key could be leveraged for authentication.

This is just a simple example of why most web services use standard protocols. They’ve been developed over decades and are—for the most part—secure against most attacks.