Component and System

Stuart Sierra’s Component is a small library that can have a big impact on the structure and architecture of your application. In this episode you learn how you can use it to create components, and manage dependencies. You will create a Jetty and an Endpoint component, and use the PostgreSQL component from the System project.

Transcript

Component calls itself a tiny framework for managing the lifecycle and dependencies of software components which have runtime state. It’s a helpful tool for making runtime dependencies explicit, and it’s a great addition to a REPL based workflow.

Component is platform independent, the examples in this episode will focus on Clojure, but you can use it in ClojureScript just the same.

Component makes heavy use of records and protocols, two Clojure features that not everyone will be familiar with. If you need a refresher you can check out episode 23 and 24, which cover these in depth.

With that out of the way, let’s have a look at Component. Component is just a library, so to get started add it as a dependency to your project.clj or build.boot.

(defproject component-example "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [com.stuartsierra/component "0.3.2"]])

If you want to follow along you can also clone the lambdaisland/component_example project from Github and grab the ep26-start branch. This branch already contains all dependencies used in this episode, as well as a few utility functions that will come in handy later on.

git clone https://github.com/lambdaisland/component_example.git
cd component_example
git checkout ep26-start

The main thing that Component provides is a protocol called “Lifecycle”, containing two methods, start and stop. A “component”, then, is a record that implements this protocol.

The docstrings mention two important properties that you need to observe when implementing these methods. When calling start, the method should not return until the component is fully started and ready, and the result of calling start must be an updated version of the same component.

(ns com.stuartsierra.component
  ,,,)

(defprotocol Lifecycle
  (start [component]
    "Begins operation of this component. Synchronous, does not return
  until the component is started. Returns an updated version of this
  component.")
  (stop [component]
    "Ceases operation of this component. Synchronous, does not return
  until the component is stopped. Returns an updated version of this
  component."))

Let’s see how that looks in practice by implementing a component for the Jetty web server. Require the com.stuartsierra.component namespace so you can access the Lifecyle protocol. Then define a record, let’s call it JettyWebServer, which implements the protocol. For now start and stop are no-ops, they simply return the component that’s passed to them. This is also what the default implementation does.

(ns component-example.components.jetty
  (:require [com.stuartsierra.component :as component]))

(defrecord JettyWebServer []
  component/Lifecycle
  (start [component]
    component)
  (stop [component]
    component))

Next up think about what inputs this component needs. It needs a ring handler function to pass incoming requests to, and it needs to know what port to listen on. So add handler and port as fields to the record.

(ns component-example.components.jetty
  (:require [com.stuartsierra.component :as component]))

(defrecord JettyWebServer [handler port]
  component/Lifecycle
  (start [component]
    component)
  (stop [component]
    component))

Now inside start it’s time to actually start Jetty. Use the Jetty adapter provided by Ring, and pass it the handler and the port. :join? false means that it will not try to join the current thread. Without this run-jetty will cause the main thread to block, and start will never return, which is not what we want.

(ns component-example.components.jetty
  (:require [com.stuartsierra.component :as component]
            [ring.adapter.jetty :as ring-jetty]))

(defrecord JettyWebServer [handler port]
  component/Lifecycle
  (start [component]
    (ring-jetty/run-jetty handler {:port port :join? false})
    component)
  (stop [component]
    component))

To implement stop you need access to the jetty server instance, which you received by calling run-jetty. But start didn’t hang on to it, so let’s fix that. You can treat records like regular maps and assoc arbitrary values onto them, so return an updated version of the component that contains a reference to the Jetty server.

Now in stop you can call the .stop method on jetty, and finally remove the :jetty key from the record again. Here you see why it’s important that start and stop return an updated version of the component, it allows you to store this kind of state.

(ns component-example.components.jetty
  (:require [ring.adapter.jetty :as ring-jetty]
            [com.stuartsierra.component :as component]))

(defrecord JettyWebServer [handler port]
  component/Lifecycle
  (start [component]
    (assoc component
           :jetty (ring-jetty/run-jetty handler {:port port :join? false})))
  (stop [component]
    (.stop (:jetty component))
    (dissoc component :jetty)))

Let’s improve this component slightly. You might accidentally call start on an already started component, which will cause an exception. You really want to avoid components throwing exceptions, it’ll save you a lot of pain once we get into systems of components. So let’s make these methods “idempotent”, so that calling them multiple times has the same effect as calling them once.

If the component is already started then it will already have a :jetty key, so just return the component unchanged.

If it isn’t started and you try to stop it, then it will end up calling .stop on nil, so before calling .stop check if :jetty is there.

(ns component-example.components.jetty
  (:require [ring.adapter.jetty :as ring-jetty]
            [com.stuartsierra.component :as component]))

(defrecord JettyWebServer [handler port]
  component/Lifecycle
  (start [component]
    (if (:jetty component)
      component
      (assoc component
             :jetty (ring-jetty/run-jetty handler {:port port :join? false}))))
  (stop [component]
    (when-let [jetty (:jetty component)]
      (.stop jetty))
    (dissoc component :jetty)))

It’s a good idea to provide a constructor function for this component, let’s call it new-jetty-web-server. This gives you a place later on to put extra initialization code if needed.

(defn new-jetty-web-server [handler port]
  (map->JettyWebServer {:handler handler :port port}))

To try this out you will need a ring handler, let’s create one by defining some Compojure routes. I’m not using the defroutes macro but instead using the routes function directly, inside a kind of routes constructor function. Having this wrapper function will come in handy later on for passing dependencies to the routes. The route returns a body of “OK” to the client, so you can quickly see if it’s working or not.

Now instantiate a new-jetty-web-server, passing it the newly constructed routes plus a port number, and call component/start on it. Assign the result to a var.

(ns component-example.core
  (:require [com.stuartsierra.component :as component]
            [component-example.components.jetty
             :refer [new-jetty-web-server]]
            [compojure.core :refer [GET routes]]))

(defn new-routes []
  (routes
   (GET "/" [] {:status 200
                :body "OK"})))

(def jetty (-> (new-jetty-web-server (new-routes) 4567)
               component/start))

If all went well then Jetty is running now. You can try this out either in your browser or on the command line with curl. Ok, that seems to have worked.

$ curl http://localhost:4567
OK

You can stop this component again with component/stop, but you have to make sure to also update the var holding the component, since otherwise it’ll think it’s still in a started state, and trying to start it again will fail.

You could use an atom, and swap! it, but frankly that’s overkill. The key feature of atoms, their “raison d’être”, is to coordinate and serialize updates in the face of concurrency. If you have mutliple threads starting and stopping components at the same time then please take a step back and reconsider your ways.

The other thing with atoms is that any function you pass to swap! must be pure, because it might be retried several times. Starting and stopping components is very much not pure, so altogether atoms are just not a good fit.

Atoms aren’t the only reference type though. How about vars? Vars get created implicitly when using def, so our component is currently stored in the jetty var. If you use the symbol jetty in your code it will resolve to the contents of the var, but you can get a reference to the var itself with #', which is a shorthand for the (var ,,,) special form.

For vars the equivalent of swap! is alter-var-root. It’s less commonly used, but it’s the most common way to track component and system state.

(alter-var-root #'jetty component/stop)

So this is nice, it provides an elegant way to start and stop jetty, but really so far it’s mostly been busywork, you could have achieved the same thing without component, simply using a few plain functions.

component-example.core> (alter-var-root #'jetty component/stop)
#component_example.components.jetty.JettyWebServer{,,,}
component-example.core> (alter-var-root #'jetty component/start)
#component_example.components.jetty.JettyWebServer{:jetty ,,,}

Once you have your components though, you can combine them to form a system map, or simply “a system”. This way, you can start or stop the whole system, rather than each individual component. It also allows specifying dependencies between components. You’ll use this to give the Ring handler access to a database connection.

Let’s first wrap the Jetty component in a system, that way there’s a place to add more components later on.

I like to have a constructor function that takes some configuration as its input, and returns a new system-map. I’ll use a single map to hold the configuration, which currently only contains Jetty’s port number.

component/system-map takes an even number of arguments, the first of each pair is just an identifier for this component, which is followed by the component itself.

(def config {:port 4567})

(defn new-system [{:keys [port]}]
  (component/system-map
   :web (new-jetty-web-server (new-routes) port)))

Now like before, use a var to store the new system map, and then start it using alter-var-root to make sure the system var contains the latest system state.

(def system (new-system 4567))

(alter-var-root #'system component/start)
;;=> #<SystemMap>

If you get an error saying the address is already in use, then you probably still have another Jetty running in the background. Luckily we used a separate var, so you should still have the old Jetty around in order to stop it.

(alter-var-root #'jetty component/stop)

Calling start on a system map calls start on each component, so this started Jetty just like before, you can verify it with curl.

Trying to look at a system map doesn’t yield a lot of information, it just says “SystemMap”. A SystemMap is another record that implements the start/stop Lifecycle protocol, which in turns calls start or stop on the individual components, after setting up inter-component dependencies.

It’s printed representation is deliberately opaque, the author of Component tried to discourage you from poking around in there. But this Clojure detective has some tricks up his sleave. To see what’s inside you can use into to convert it to a regular map.

Ok, so there’s our JettyWebServer component, and it’s associated with the :web key, which is the name you gave it when setting up the system map.

(into {} system)
;;=> {:web #component_example.components.jetty.JettyWebServer{,,,}}

In the component-example.util namespace I’ve included an inspect-system function that can help you to get a bit more insight. It shows each key in the system map, and the list of keys available to each component. Before starting the system this is what it looks like. The Jetty component has access to the its two fields: handler and port.

After starting the system there’s one new key: the reference to the Jetty instance that was assoced onto the component.

(ns component-example.core
  (:require ,,,
            [component-example.util :refer [inspect-system]]))


(inspect-system system) ;;=> {:web (:handler :port)}

(alter-var-root #'system component/start)

(inspect-system system) ;;=> {:web (:handler :port :jetty)}

Let’s make things more interesting by adding some more components. Say you want to use a PostgreSQL database. You could implement another record that handles establishing a connection, but you’ll find that for popular open source projects like MySQL or PostgreSQL someone has already done the work for you.

The System project in particular contains a growing list of ready-made components. I could have used their Jetty component, but then I hadn’t had the pleasure to show you how to write your own.

If you’re following along at home you’ll have to set up PostgreSQL with a new database and user. I’ll add some pointers in the show notes.

We’ll add a postgres component, so require the system.components.postgres namespace as pg.

Next you need a configuration map that specifies how to connect to the database server. System contains a default map so you can just override the values that matter, like the database name, username and password.

Finally you can add a new :db component to the, create the component by calling the new-postgres-database constructor function with the database configuration.

(ns component-example.core
  (:require ,,,
            [system.components.postgres :as pg]))

(def config
  {:port 4567
   :pg-config (assoc pg/DEFAULT-DB-SPEC
                     :subname "component_example"
                     :user "component_example"
                     :password "component_example")})

(defn new-system [{:keys [port pg-config]}]
  (component/system-map
   :web (new-jetty-web-server (new-routes) port)
   :db (pg/new-postgres-database pg-config)))

If you’re following along then first stop the system before redefining the system var, otherwise you lose access to the old system which is still running, and there will be no way to stop the old Jetty instance.

(alter-var-root #'system component/stop)

(def system (new-system {:port 4567
                         :pg-config pg-config}))

(alter-var-root #'system component/start)

Now if you inspect the system it look like this. The web component hasn’t changed, but the :db component is new. :db-spec is the configuration we passed to it, and :connection is the actual database connection.

{:web (:handler :port)
 :db (:db-spec :connection :init-fn)}

So with that information you know enough to get from the system map to the database connection. The connection uses Java’s JDBC API for database access. I’ve added a few convenience functions to the project for querying the database based on the connection. They’re in the component_example.db namespace.

Let’s create a table called widgets, just to have something to play with. To run statements like this one that aren’t supposed to return anything, you can use db/execute!. Insert a few widgets into the table as well.

To actually run queries that return a result, you call db/query.

(ns component-example.core
  (:require ,,,
            [component-example.db :as db]))

(db/execute! (-> system :db :connection)
             "CREATE TABLE widgets (id BIGSERIAL PRIMARY KEY, name VARCHAR(100))")

(db/execute! (-> system :db :connection)
             "INSERT INTO widgets (name) VALUES ('button'), ('slider')")

(db/query (-> system :db :connection)
          "SELECT * FROM widgets")

Now what I would like is that the API endpoint returns a list of these widgets in the database. So somehow I’ll have to make sure that the routes function receives the database component, so it can query it. Let’s assume the constructor function receives something that has a :db key, and then work backwards from there.

(defn new-routes [{:keys [db]}]
  (routes
   (GET "/" [] {:status 200
                :body "OK"})))

To do this, first introduce a new component, called Endpoint. It’ll act as a wrapper for the routes constructor function. start calls the constructor and gets routes back, which it stores on the endpoint component with assoc. stop dissoces :routes again so it’s back in it’s initial state.

Also add a contructor function like you did before for Jetty, called new-endpoint.

The thing to notice is that, when calling the routes constructor, it passes in the endpoint component as argument. So somehow you’ll have to make sure that the endpoint contains the :db component, so the routes can access it.

(defrecord Endpoint [routes-fn]
  component/Lifecycle
  (start [component]
    (assoc component :routes (routes-fn component)))
  (stop [component]
    (dissoc component :routes)))

(defn new-endpoint [routes-fn]
  (map->Endpoint {:routes-fn routes-fn}))

Add the Endpoint to the system, passing it the routes constructor.

(defn new-system [{:keys [port pg-config]}]
  (component/system-map
   :endpoint (new-endpoint new-routes)
   :db ,,,
   :web ,,,
   ,,,))

The endpoint component passes itself to the routes constructor, so somehow you will have to get the database component into the endpoint component, so the routes can access it. You can achieve this with component/using. You pass it the component, and then a vector of the names of the components that it depends on. In this case :db.

This achieves two things: first it makes sure components are started in the right order. After all, if :endpoint needs access to the :db, then the :db component should be started before :endpoint gets started.

(defn new-system [{:keys [port pg-config]}]
  (component/system-map
   :endpoint (component/using (new-endpoint new-routes) [:db])
   :db ,,,
   :web ,,,
   ,,,))

(def system (new-system {,,,}))

(alter-var-root #'system component/start)

The second thing it does, is assoc any dependencies onto the components that need them. So in this case that means the :endpoint component will have a new key, :db, which points at the PostgreSQL component.

Now before starting the system this is what it looks like. The endpoint and the web components have access to the things we passed them, nothing more.

(inspect-system system)
;; =>
{:endpoint (:routes-fn)
 :web (:handler :port)
 :db (:db-spec :connection :init-fn)}

After starting the system, things look a little different. :endpoint now has :routes, and :web has :jetty, these are the things that got added in their respective start methods, but :endpoint now also has an extra key: :db. This is the :db component, the same one that’s in the system map.

Because we told component that the endpoint uses the database, it first started the database, and then added it to the endpoint, before starting the endpoint. So now in the endpoint’s start method it’s possible to access the database component.

(inspect-system system)
;; =>
{:endpoint (:routes-fn :routes :db)
 :web (:handler :port :jetty)
 :db (:db-spec :connection :init-fn)}

Here’s the endpoint again. It doesn’t directly access the database, but it passes itself on to the routes constructor function. In this way the endpoint component just becomes a vehicle for getting dependencies to the routes.

(defrecord Endpoint [routes-fn]
  component/Lifecycle
  (start [component]
    (assoc component :routes (routes-fn component)))
  (stop [component]
    (dissoc component :routes)))

So the new-routes function ends up receiving the endpoint component as argument, and can destructure it to get to any dependent components, in this case the database. This way you can inject any dependencies into the routes, simply by adding them to component/using.

(defn new-routes [{:keys [db] :as endpoint}]
  ,,,)

When declaring dependencies with component/using, the injected name doesn’t always match the name used in the system map. Therefore instead of using a vector to declare the dependencies, you can also pass a map.

You can read this a bit like a binding form, with the original name on the right, and the name to be assigned on the left. So if the postgres component is called :db in the system map, but the routes expect to find :database, then you can do something like this.

(defn new-routes [{:keys [database] :as endpoint}]
  ,,,)

(defn new-system [{:keys [port pg-config]}]
  (component/system-map
   :endpoint (component/using (new-endpoint new-routes) {:database :db})
   ,,,))

So that’s all very well, but this endpoint component isn’t doing anything yet. We need to change the Jetty component so it can work with the endpoint, instead of using a routes contstructor function directly.

So take out that first argument, now all you pass to Jetty is the port number, and instead use component/using to tell component that Jetty depends on the :endpoint.

(defn new-system [{:keys [port pg-config]}]
  (component/system-map
   :endpoint (component/using (new-endpoint new-routes) {:database :db})
   :web (component/using (new-jetty-web-server port) [:endpoint])
   ,,,))

Now it’s time to update Jetty, to make use of this endpoint that’s being injected.

Remove the handler field, it will no longer be needed, and instead starting from the Jetty component, find the endpoint component, and then pull out the :routes, which is the actual ring handler.

Also take care to update the constructor function.

(defrecord JettyWebServer [port]
  component/Lifecycle
  (start [component]
    (if (:jetty component)
      component
      (assoc component
             :jetty (ring-jetty/run-jetty (-> component :endpoint :routes) {:port port :join? false}))))
  (stop [component]
    (when (:jetty component)
      (.stop (:jetty component)))
    (dissoc component :jetty)))

Stop the old system, redefine new-system, now create a new system, and start it.

Now just like :db got added to the :endpoint, this time :endpoint gets added to the :web component. You can see that when you inspect the system again.

(inspect-system system)
;;=>
{:endpoint (routes-fn routes db)
 :web (:port :endpoint)
 :db ,,,}

So with that the last piece of the puzzle is in place, and the routes now have access to the database. Query it to get the list of widgets, and use pr-str to turn these into EDN.

(defn new-routes [{:keys [database] :as endpoint}]
  (routes
   (GET "/" [] {:status 200
                :body (pr-str
                       (db/query (:connection database)
                                 "SELECT name FROM widgets"))})))

After restarting the system, you can check with curl that things are actually working.

You now know the essential parts of component: the Lifecycle protocol with start and stop, the system map, and the using function to declare dependencies.

Component is a small library, but it’s a big topic to cover, and we’ve only scratched the surface of its motivations, and usage patterns.

In a follow up episode we’ll dig a bit deeper into its practical usage.

Show Notes

browse source code

The starting point for this episode is the ep26-start branch, the final result is in ep26-end. (diff)

Libraries

stuartsierra/component
danielsz/system Collection of pre-made components
weavejester/reloaded.repl
walmartlabs/system-viz Use graphviz to draw a visual representation of a system map
weavejester/ring-jetty-component A standalone Jetty component
weavejester/suspendable Add suspend/resume to component lifecycles

Discussions

Mailing list thread about using Component with Ring

PostgreSQL setup

Assuming you have the server set up locally

$ sudo -u postgres createdb component_example
$ sudo -u postgres psql component_example
component_example=# CREATE ROLE component_example PASSWORD 'component_example';
CREATE ROLE
component_example=# GRANT ALL PRIVILEGES ON DATABASE component_example to component_example;
GRANT

For info on how to install the server

Title screen attribution

Photo of peppermills by dinnerseries