Lambda Island

In this new series you’ll build an API with Ring from the ground up. The pace is slow and no steps are skipped, making it a great lesson for beginners.

Transcript

This is the start of a new series of episodes about backend development from the ground up, and the first point of call, the foundation of any Clojure web app, is Ring.

Ring is based on a simple but powerful idea, borrowed from Ruby’s Rack and Python’s WSGI. If we can agree on a standard way to represent HTTP requests and HTTP responses, then web servers, web apps, middleware, and other tooling that follows this standard will just effortlessly click together, like legos. This standard is embodied in the Ring specification.

HTTP is a simple text based protocol, the client, the so-called “user agent” opens a connection to the server and sends a request that looks like this. I’m using an Emacs package called HTTP Twiddle, so I can send this request and see the server’s response.

The first line is the “Request Line”, it has three parts: the HTTP “method”, the resource path, and the HTTP version. In this case we want to GET the resource at the server root (“/”).

Following the request line are one or more headers. I’ve included a few common ones.

GET / HTTP/1.1
Host: localhost:3000
User-Agent: Emacs (HTTP Twiddle)
Accept: */*

If we send this request to the server this is the response we get back. The first line is now a “status line”, containing the HTTP version, a numeric status code, and a “reason phrase”. “200 OK” means the request was succesful.

After the status line there are again a number of headers, followed by two newlines, and then the response body. In this case we’re getting an EDN document back.

HTTP/1.1 200 OK
Date: Mon, 01 Aug 2016 16:25:49 GMT
Content-Type: application/edn; charset=UTF-8
Content-Length: 110
Server: Jetty(9.2.10.v20150310)

({:id "e5a92946-4a35-48e4-b4c4-1f75357b1c8c", :full-name "Trish Turtle", :skills ["LISP" "Lambda Calculus"]})

So that’s what goes over the wire, but we don’t want to deal with that directly. Instead what we need is a nice Clojure-like representation, which is what ring gives us. When using Ring, that request would be represented as a Clojure map that looks like this. You can see that all the same information is there, the request method, the URI, the headers, but instead of plain text it’s now a Clojure map so it’s easy to work with.

{:request-method :get
 :uri "/"
 :protocol "HTTP/1.0"
 :headers {"host" "localhost:3000"
           "user-agent" "Emacs (HTTP Twiddle)"
           "accept" "*/*"}
 :scheme :http
 :server-name "localhost"
 :remote-addr "127.0.0.1"
 :server-port 3000}

Similarly the HTTP response can be represented as a map as well, with the status, headers, and body all there.

{:status 200,
 :headers {"Content-Type" "application/edn; charset=UTF-8"},
 :body "({:name \"Brasseur\", :given-name \"Arne\"})\n"}

To handle the plumbing of the HTTP protocol we need a HTTP server. We’ll use Jetty in this episode, a popular web server written in Java.

Jetty doesn’t use Clojure datastructures, since it’s written in Java. Instead it uses certain types of Java objects to represent HTTP requests and responses. Normally a web app that uses Jetty would have to know how to consume and create these objects.

In the Clojure world we don’t need no stinking Java objects though, we’re perfectly happy with representing requests and responses as Clojure maps.

We need something to bridge the gap. The Ring specification calls this type of thing an “adapter”. The adapter knows how to speak Java when it’s talking to Jetty, but when it talks to our app it follows the Ring specification, representing requests and responses as maps.

When the adapter gets a request from Jetty, it turns it into a Ring request map, passes that request map to our app, then takes the response map the app returned, turns it into a HttpResponse object, and passes that on to Jetty.

<!-- slides! -->

Enough with the theory, let’s see how that looks in code. We’re going to build an address book API, and because we like it 70’s vintage we’re going to call it “rolodex”.

Let’s set up the project. We need Clojure of course, and the Ring adapter for Jetty. Adding this dependency will also pull in Jetty itself, so we don’t need to add that explicitly.

(defproject rolodex "0.1.0"
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [ring/ring-jetty-adapter "1.5.0"]])

At the heart of our app is a function which turns Ring requests into Ring responses. The Ring SPEC calls this kind of function a “handler”.

Our handler doesn’t contain any logic yet, no matter what the request is that comes in, it will always answer with the same canned response. A status of 200 “OK”, a content-type header, and a body.

(ns rolodex.core)

(defn handler "request -> response"
  [req]
  {:status 200
   :headers {"Content-Type" "text/plain; charset=UTF-8"}
   :body "Babby's first ring app"})

Now we’re ready to spin up a Jetty instance using Ring’s Jetty adapter. The run-jetty function lives in the ring.adapter.jetty namespace, so require that. And then when calling run-jetty we pass it the handler and a list of options. We’ll run our app on port 3000.

(run-jetty handler {:port 3000})

Now I’m not going to evaluate this line yet, first there are a few things we should change about it or we’ll run into issues with our REPL based workflow.

Let me show you what happens when you run this in a REPL. The call blocks, it doesn’t return, and so we don’t get a new REPL prompt. We can interrupt the call with Ctrl-C, and Jetty will keep on going in the background, but it’s not exactly a great workflow.

The same thing happens when you do this in the CIDER REPL, you can see in the minibuffer there’s this pseudo-progressbar that shows CIDER is waiting for a response. To interrupt a call in CIDER you have to press Ctrl-C twice.

If we evaluate this in the buffer with CIDER then the same thing happens, but it’s not immediately clear what’s going on. All we notice is that none of the CIDER stuff works anymore. When you look at the REPL buffer though you can see here in the modeline that CIDER is waiting for a response.

So Jetty’s running now and there’s no way for us to stop it, short of stopping CIDER. So restart CIDER and let’s try something else.

To prevent this from happening, we pass this :join? false option, this means that the main thread should not join with Jetty’s server thread. Now we can start Jetty without CIDER becoming unresponsive.

I’m also assigning the return value from run-jetty to a var, this will come in handy if we want to stop or restart Jetty later.

(def server
  (run-jetty handler {:port 3000
                      :join? false}))

Ok so our server is running now, let’s see if it works. I’ll first show you how to test this with curl from the command line. The -v option gives us verbose output, so you see the full request and response as it’s sent over the wire, the -s option stands for “silent”, and stops curl from printing a progress meter.

$ curl -vs http://localhost:3000/
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 02 Aug 2016 13:55:32 GMT
< Content-Type: application/edn; charset=UTF-8
< Content-Length: 41
< Server: Jetty(9.2.10.v20150310)
<
({:name "Brasseur", :given-name "Arne"})
* Connection #0 to host localhost left intact

There are also some great Emacs packages you can use for this. You already saw me use HTTP Twiddle, simply put your request in a buffer, enable http-twiddle-mode, and press C-c C-c, and another buffer pops up with the response. Make sure your request ends on two newlines as it should.

An alternative package that’s a bit more full-featured is called RESTClient.

GET / HTTP/1.1
Host: localhost:3000
User-agent: Emacs (HTTP Twiddle)
Accept: */*

Back to the app, suppose we want to change our handler. We update the code, re-evaluate the defn, and make another HTTP request. Seems like Jetty is still using the old handler.

To make Jetty aware of the new handler we’ll have to stop and then start it afresh. That’s why we assigned the result of run-jetty to a var.

(.stop server)

(def server
  (run-jetty handler {:port 3000
                      :join? false}))

There’s a trick to make our lives easier though. Stop Jetty again, and now instead of passing it the handler directly, pass it the handler Var. Start Jetty again, and update the handler. Now if we try the HTTP request again we see it did use the new handler.

(def server
  (run-jetty #'handler {:port 3000
                        :join? false}))

So what’s going on? When you use def to assign a value to a name, or defn to assign a function, then clojure internally will create a “Var” object to hold that value. We don’t usually deal with the var itself, but we can get a hold of it with the var special form, or its #' shorthand. You can see that this is an object of type clojure.lang.Var.

A “Var” is a reference type, it’s a bit like an atom, a box that can contain a value. When you call def again with the same name, then Clojure will use the existing var, and just change its content.

(def answer 42)

answer                                             ;;=> 42

(var answer)                                       ;;=> #'user/answer
#'answer                                           ;;=> #'user/answer

(type #'answer)                                    ;;=> clojure.lang.Var

@(var answer)                                      ;;=> 42

The interesting thing is that a “var” can also behave like a function. It implement the IFn interface, so like keywords, maps, or sets, you can use it wherever you can use a function. If the Var contains a function, then calling the var has the same effect as calling the function.

(defn answer? [x]
  (= x 42))

(answer? 42)                             ;;=> true

(instance? clojure.lang.IFn #'answer)    ;;=> true

(#'answer? 42)                           ;;=> true

If we pass the handler var to Jetty, then it will always call whatever function is currently assigned to that var. If we redefine handler, then the handler var will contain a new function, but it will still be the same Var object, so Jetty will always be up to date.

That should stop us from having to stop and start Jetty, so let’s change this def to defonce. That way when we reload the namespace (in CIDER that’s C-c C-k) it won’t try to start another instance of Jetty. In other words we’re making sure our code is reloadable.

(defonce server
  (run-jetty #'handler {:port 3000
                        :join? false}))

Time to actually implement some functionality, here are the operations we eventually want to support.

GET /
  Retrieve a list of all contacts

POST /
  Create a new contact

GET /:id
  Retrieve a single contact

PUT /:id
  Update a contact

DELETE /:id
  Delete a contact

GET /?name=Smith
  Filter the list of contacts

As “database” for the contacts in our Rolodex we’ll just use an atom for now. I’ll already put one contact in there so we have something to show. As lookup key we’ll use the contact’s id.

Unlike Clojurescript, Clojure doesn’t have a built-in random-uuid function, but we can quickly polyfill it by calling the corresponding Java method.

(defn random-uuid []
  (java.util.UUID/randomUUID))

(def contacts (atom (let [id (random-uuid)]
                      {id {:id id
                           :full-name "Trish Turtle"
                           :skills ["LISP", "Lambda Calculus"]}})))

Let’s implement the first action, when doing a GET request to the root path, the handler will return a list of all contacts. So check the request-method and uri, and if they match we return an EDN document. EDN stands for “extensible data notation”, it’s basically a subset of the Clojure syntax that you can use to represent data, and we can generate EDN with prn-str.

We’ll set a Content-Type header. We’ll also explicitly set the character set to be the UTF-8 character encoding of unicode. EDN actually doesn’t have an IANA registered media type, this one here I just made up. It’s temporary anyway, EDN isn’t a great transport format, so we’ll switch to JSON or Transit later on.

Java strings are encoding-aware, they are internally represented as UTF-16 unicode strings. When you set a character set like we did here, Jetty will honor that and make sure the body text is correctly encoded over the wire.

It’s a good idea to always set the encoding explicitly. For certain content types Jetty will use Latin-1 as its default, also known as ISO-8859-1, and anything it can’t encode as Latin-1 will turn into question marks. I would put a sad emoji here but it wouldn’t even make it through :(.

(defn handler [{:keys [uri request-method]}]
  (when (and (= request-method :get) (= uri "/"))
    {:status 200
     :headers {"Content-Type" "application/edn; charset=UTF-8"}
     :body (prn-str (vals @contacts))}))

Show Notes

browse source code

This episode goes up to the ring-part-1 branch.

Code

project.clj

(defproject rolodex "0.1.0"
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [ring/ring-jetty-adapter "1.5.0"]])

src/rolodex/core.clj

(ns rolodex.core
  (:require [ring.adapter.jetty :refer [run-jetty]]))

(defn random-uuid []
  (java.util.UUID/randomUUID))

(def contacts (atom (let [id (random-uuid)]
                      {id {:id id
                           :full-name "Trish Turtle"
                           :skills ["LISP" "Lambda Calculus"]}})))

(defn handler "request -> response"
  [req]
  (if (and (= (:request-method req) :get) (= (:uri req) "/"))
    {:status 200
     :headers {"Content-Type" "application/edn; charset=UTF-8"}
     :body (prn-str (vals @contacts))}))

(defonce server
  (run-jetty #'handler {:port 3000
                        :join? false}))

Ring, part 1