Lambda Island

Integrant is a “micro-framework for data-driven architecture”. It manages the lifecycle of your application, and the dependencies between components, similar to Component or Mount, but with some very different design decisions.

Integrant forms the foundation for Duct, a Clojure web framework by James Reeves, aka Weavejester.

Transcript

Integrant describes itself as a “micro-framework for data-driven architecture”. It’s a library that lets you split your app into parts, and manage their interdependencies and start/stop life cycle.

Integrant was created by James Reeves, a prolific Clojure programmer known online as Weavejester. It occupies the same space as Stuart Sierra’s Component library, which we’ve covered in detail in episode 26 and 31, but makes some very different design decisions.

Integrant forms the basis for Duct, a highly modular Clojure Web Framework. If you’re interested in learning Duct then you should get a good grip on Integrant first.

To demonstrate Integrant we’ll take a very basic web app, and look at different ways to approach its implementation. The app implements a HTTP single endpoint, which responds with the current date and time.

This episode will be self-contained, if you just follow along you should have everything you need. I will link to a github repo with the resulting code in the show notes, in case you want to have a look at the final result.

$ curl localhost:3000
It's Thu Mar 08 16:41:59 CST 2018

I’m using the Clojure CLI tools that came out together with Clojure 1.9. This deps.edn file contains all dependencies used in this episode.

There’s Integrant of course, and also Integrant REPL, which we’ll be using later on. The Ring Jetty adapter will make it easy to bootstrap a web server.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.9.0"}
        integrant {:mvn/version "0.7.0-alpha1"}
        integrant/repl {:mvn/version "0.3.0"}
        ring/ring-jetty-adapter {:mvn/version "1.6.3"}}}

As a starting point, here’s the most basic, hard coded version of this web app I could come up with. There’s a handler which handles individual HTTP requests. It’s a plain, pure function which takes a request map and returns a response map.

The final line starts the actual web server, we invoke Ring’s Jetty adapter, passing it the handler we just defined, and some configuration data.

(ns li38.hard-coded
  "Hard coded web service"
  (:require [ring.adapter.jetty :as ring-jetty]))

(defn handler [{:keys [request-method uri] :as req}]
  {:status 200
   :body (str "It's " (java.util.Date.))
   :headers {"Content-Type" "text/plain"}})

(ring-jetty/run-jetty handler {:port 3000 :join? false})

Now as always I’m using a REPL connected to my editor, so I can easily evaluate these forms one by one. For instance I can change this handler, and re-evaluate the defn form to redefine it.

(defn handler [{:keys [request-method uri] :as req}]
  {:status 200
   :body (str "It's currently " (java.util.Date.) ". Coding time!")
   :headers {"Content-Type" "text/plain"}})

However when I test this out, I’m still seeing the old result. That’s because the jetty adapter is still using the old handler.

I can try evaluating the run-jetty function again, but that only gets me an error because the port is already in use.

The first problem has to do with Clojure being stateful, the second has to do with the application itself being stateful.

There are ways around this. You could save a reference to the Jetty server, so you can stop it, redefine the handler, and start a new one.

(def server
  (ring-jetty/run-jetty handler {:port 3000 :join? false}))

(.stop server)

In our ten line program this is still manageable, but for anything more complex your only choice will be to kill the Clojure process and start over, which isn’t great.

A lot has been said and done about solving this problem through what is known as the “reloaded workflow”. In this episode I’ll skip the discourse and focus on how Integrant does it, but I’ll leave several links in the show notes if you want to dig into this topic.

So without further ado let’s convert this mini app to Integrant.

Add integrant.core to the namespace, aliasing it to ig.

Integrant calls itself a mini framework for data-driven architecture, so it follows that we’ll have to start with data. In particular you start with a map which describes your application. Each key corresponds with one part of your application, and the corresponding value is the configuration for that part.

This config is most likely to come from an EDN file, but here I’ll just define it inline.

Let’s start with the handler. I’ll call it :handler/time, since it’s a handler that returns time information, and I’ll make the prefix and suffix configurable.

(ns li38.with-ig
  (:require [integrant.core :as ig]
            [ring.adapter.jetty :as ring-jetty]))

(def config
  {:handler/time {:prefix "It's ", :suffix "."}})

Now for each of these pieces of your application you need to tell integrant how to turn the configuration into a implementation, by implementing the integrant.core/init-key multimethod.

The multimethod’s “dispatch value” is the key you used in the configuration map, namely :handler/time. The method receives two arguments, the keyword, and the configuration, and it returns the actual handler.

(defmethod ig/init-key :handler/time [_ config]
  (fn [{:keys [request-method uri] :as req}]
    {:status 200
     :body (str (:prefix config) (java.util.Date.) (:suffix config))
     :headers {"Content-Type" "text/plain"}}))

You can try calling init-key yourself, and you’ll see you get a handler function back, which indeed when called returns a response map.

(ig/init-key :handler/time {:prefix "It's " :suffix "."})
;;=> #function[li38.v1/eval8089/fn--8090/fn--8092]

((ig/init-key :handler/time {:prefix "It's " :suffix "."}) {:request-method :get, :uri "/"})
;;=> {:status 200, :body "It's Fri Mar 09 11:58:07 CST 2018.", :headers {"Content-Type" "text/plain"}}

Normally you don’t call init-key directly however, but instead you use integrant’s init function, passing it a configuration map.

What you get back is again a map, with the same keys, but with each configuration value replaced by the implementation returned from init-key.

(ig/init config)
;;=> {:handler/time #function[li38.v1/eval8089/fn--8090/fn--8092]}

So far so good, now let’s add in the Jetty Ring adapter. Now it gets interesting, because besides the regular pieces of configuration like the HTTP port to listen on, the adapter also needs access to another piece of the application, the handler. You achieve this with an integrant reference.

This reference isn’t anything magical, it’s just a placeholder that integrant will recognize, and replace with the actual handler, before initializing :ring/jetty.

If you look at what config looks like now, you’ll see that there’s an instance there of integrant.core.Ref.

(def config
  {:handler/time {:prefix "It's "
                  :suffix "."}
   :ring/jetty   {:port 3002
                  :join? false
                  :handler (ig/ref :handler/time)}})

config
;;=>
{:handler/time {:prefix "It's "
                :suffix "."},
 :ring/jetty   {:port    3002
                :join?   false
                :handler #integrant.core.Ref{:key :handler/time}}}

init-key for :ring/jetty is straightfoward, just call run-jetty like before.

(defmethod ig/init-key :ring/jetty [_ {:keys [handler port join?]}]
  (ring-jetty/run-jetty handler {:port port :join? join?}))

This time when you call init, integrant will figure out that :ring/jetty needs to acecss the handler, so it will initialize the handler first, and then move on to initialize :ring/jetty. The end result is again a map with each value replaced with its implementation.

(def system
  (ig/init config))
;;=> {:handler/time #function[li38.v1/eval8065/fn--8066/fn--8068],
;;    :ring/jetty   #object[org.eclipse.jetty.server.Server@31d65f2b"]}

In a real world app you’re more likely to be loading your configuration from an EDN file, such as this one. Notice that this time you can’t use integrant’s ref function, but you can use the #ig/ref reader tag to mark something as being a reference.

;; config.edn
{:handler/time {:prefix "It's "
                :suffix "."}
 :ring/jetty   {:port 3002
                :join? false
                :handler #ig/ref :handler/time}}

Now use integrant’s read-string function to load this EDN. It’s similar to read-string in clojure.core, but it makes sure those references are read correctly.

(ig/read-string (slurp "config.edn"))

Now you know how to start the application, but to get back to a clean slate you also need to be able to stop it. This is where the halt! function and halt-key! multimethods come in.

The handler isn’t stateful, so there’s no need for a halt-key! implementation, but for the web server it does make sense to be able to stop it. halt-key! is similar to init-key, but instead of receiving the configuration for the key, it receives the implementation that init-key returned. The Jetty instance, in other words.

The return value of halt-key! is ignored, it only cares about side effects.

If you’re familiar with Component than you’ll notice that this is markedly different. In component every operation returns an updated component or system map, whereas in integrant any function or method that ends in a bang ignores its return value.

(defmethod ig/halt-key! :ring/jetty [_ server]
  (.stop server))

Now you can halt! the system, which will call halt-key! for each piece of the app. Pieces are stopped in the opposite order in which they were started, so in this contrived example Jetty will be halted first, and then the handler. This prevents the situation where one piece is already halted, while another one still depends on it.

(ig/halt! system)

Calling halt! always returns nil, in Integrant once you halt a system you simply throw it away. Sometimes however you do want to “pause” a system, and start it up again later on. Integrant calls these operations suspend and resume, respectively, and just like with init and halt! they each have a corresponding function and multimethod.

When you stop a Jetty instance it stops listening to the network, but you can start it again later on to resume its work, so let’s implement suspend and resume for the jetty adapter.

suspend-key! is similar to halt-key!, it receives the implementation returned from init-key, and stops it, whatever that means for this particular key. Notice that the name ends with a bang, so the return value of suspend-key! is ignored. It’s strictly there for side-effects, in other words. Our suspend-key! will simply stop the Jetty adapter, just like halt-key!.

(defmethod ig/suspend-key! :ring/jetty [_ server]
  (.stop server))

The final lifecycle method we’ll look at is resume-key. It’s the most intricate of all these operations, receiving not two but four arguments: the config key, the old configuration map, the new configuration and the implementation coming from init-key, the same one that was suspended by suspend-key!.

Before starting up the server again we should check if the configuration has changed, maybe we should actually be listening on a different port, for example. In that case we simply discard the old server instance and fall back to init-key instead.

I should mention that this approach to suspending and resuming Jetty is really a little too naive, but it illustrates the concept. The Integrant README has a more complete example of how to implement this.

(defmethod ig/resume-key :ring/jetty [key config old-config server]
  (if (= config old-config)
    (.start server)
    (ig/init-key key config)))

Now suspend! the system. Notice how this looks very similar to calling halt!, and in fact for any keys without a suspend-key! implementation integrant will simply fall back to calling halt-key! instead.

The main difference is that after calling halt! you’re expected to discard the system, if you want to start it again later on you initialize a new one from scratch, whereas with suspend! you hold on to the system, so you can resume it later on.

Notice that suspend is still strictly side-effectful, it doesn’t return anything, instead you hold on to the system you pass into suspend, and pass that same system later on to resume.

(ig/suspend! system)
;;=> nil

Now when you resume the system, you pass in the new configuration, and the old, suspended system. Any keys that have their own resume-key get access to both, and can figure out how much of the old system they want to reuse, and how much they want to initialize afresh.

Keys without a resume-key implementation fall back to init-key instead, and so are always created anew.

(def system
  (ig/resume config system))
;;=> {:handler/time ,,, :ring/jetty ,,,}

So, to recap, Integrant has four lifecycle functions you can call: init, halt!, suspend!, and resume.

(init config)
(halt! system)
(suspend! system)
(resume config system)

And four corresponding multimethods for you to implement.

(init-key key value)
(halt-key! key impl)
(suspend-key! key impl)
(resume-key key value old-value old-impl)

init and resume also have a version that lets you start only specific keys. This can be useful if you only want to start part of your application, or want to initialize different parts at different times.

(init config [:your/key])
(resume config system [:your/key])

There’s one more multimethod that you can use, called prep-key, which lets you do some preprossing on a key’s configuration. It’s a good place for instance to set some default values, or to look up symbolic references.

Let’s add a prep-key method for :ring/jetty, so that we can omit some common values in the configuration map. Convention over configuration, so to speak.

(def config
  {:handler/time {:prefix "It's " :suffix " ."}
   :ring/jetty   {:handler (ig/ref :handler/time)}})

(defmethod ig/prep-key :ring/jetty [_ config]
  (merge {:port 3000
          :join? false}
         config))

prep-key again has a corresponding function, prep, I hope you’re starting to see a pattern by now.

(def system (ig/init (ig/prep config)))

Now you might wonder what the benefit is of this prep operation, since you might as well do all of it inside init. There’s the simple benefit of splitting up responsibilities, making the code easier to reason about, but there’s also a potentially a real difference, but to understand that we first need to look at Clojure’s keyword hierarchies.

Keywords are generally just plain, isolated identifiers, but by using Clojure’s derive function you can specify that one keyword is a more specific instance of another keyword.

So when you say (derive :ring/jetty :http/server), you’re really saying that :ring/jetty is a specific instance of a :http/server. This lets you ask the question afterwards: is:ring/jetty a :http/server, and Clojure will tell you that yes, it is.

(derive :ring/jetty :http/server)

(isa? :ring/jetty :http/server)
;;=> true

This isa? function has some interesting properties. It’s what mathematicians call “transitive”, meaning that if “A is a B” and “B is a C”, then “A is also a C”.

(derive ::A ::B)
(derive ::B ::C)

(isa? ::A ::C)
;;=> true

Another thing to note is that when two things are equal according to Clojure’s = equality check, then isa? will also be true. To put it another way, isa? is really just =, except that it also checks for derived keywords.

(isa? ::A ::A)
;;=> true

(isa? 7 7)
;;=> true

The interesting thing is that when Clojure tries to find a multimethod implementation to dispatch to, it will use isa?, so if you don’t have a prep-key implementation for :ring/jetty, but you do have one for :http/server, then Clojure will use that one.

This allows you to implement a generic prep-key method for all HTTP servers. Only if a specific server has some special prepping needs would you add another defmethod that overrides the general one.

(derive :ring/jetty :http/server)

(defmethod ig/prep-key :http/server [_ config]
  (merge {:port 3000
          :join? false}
         config))

Making good use of derive also means you’re adding extra semantic information to your configuration, which you can query with the help of integrant.

Suppose you actually have multiple ring handlers in your configuration. By having them all derive from :ring/handler, you can find them all using integrant’s find-derived function.

This mechanism is used heavily in Duct to provide defaults, and automatically wire things up.

(derive :handler/time :ring/handler)
(derive :handler/lime :ring/handler)

(ig/find-derived {:handler/time "timey"
                  :handler/lime "tasty"}
                 :ring/handler)

;;=>([:handler/time "timey"] [:handler/lime "tasty"])

Notice that we have a pretty big pile of multimethods stacked together in a single namespace. In a real-world app you might be better off splitting these up into separate namespaces.

You can put them wherever you like, but Integrant does offer some conventions. Say you have a key like this: :my-app.ring/adapter, then the corresponding mulimethod defintions should go either in my-app.ring, or my-app.ring.adapter.

When you do this you face a new problem though: because these multimethod definitions are all in separate files, you need to make sure they’re all loaded before trying to initialize the system.

The good news is that if you’ve followed these conventions, than Integrant can help you out. Simply call load-namespaces, passing it your configuration map, and Integrant will try all these namespaces in turn to see if they can be loaded. This way you don’t have to add dummy :require statements just to load these files.

Just as with init and resume you can optionally pass a list of keys. Integrant will assume you’re only planning to start these keys, and will only load the namespaces that are needed.

(defmethod init-key :my-app.ring/adapter [_ _])
;;=> put this in (ns my-app.ring) or (ns my-app.ring.adapter)

(ig/load-namespaces config)
(ig/load-namespaces config keys)

At this point you’ve really seen all there is to know about integrant.core. There are a few more utility functions that might occasionally be helpful, if you really want to know all the details then I suggest taking ten minutes out of your day to read through the API documentation, or to take an hour and read through the source, Jame’s code is really quite instructive.

So that’s all good, but we still haven’t solved the problem we started with. If you change the handler function then you still need to manually halt the system, reload the handler code, and then restart the system.

Admittedly this has already gotten a little easier, you can now start and stop a complex application with a single function, and code reloading is really also a solved problem. The clojure.tools.namespace library implements intelligent unloading and reloading, to make sure Clojure’s view of the world is consistent with what’s on the file system.

What you need is something that ties the two together, so you can get a clean slate of both application slate and Clojure slate. This is where integrant.repl comes in.

A good place to load integrant.repl is in your user.clj, since that’s where code goes that should only be loaded during development. If you’re using Leiningen or the Clojure CLI tools than user.clj will be loaded automatically, so anything defined here can easily be called from the REPL.

Load integrant.core as well, you’ll be needing it soon enough.

(ns user
  (:require [integrant.core :as ig]
            [integrant.repl :as ig-repl]))

Now you need to tell integrant.repl how to get to your app’s configuration with set-prep!, passing it a zero-argument function which returns the Integrant configuration map.

(ig-repl/set-prep! (fn [] {:app/key {:the :config}}))

This time we’ll get the configuration from an EDN file. First define this load-config helper function that loads the configuration from an EDN file, using Integrant’s own version of read-string, to make sure references are read correctly.

(defn load-config []
  (ig/read-string (slurp "config.edn")))

Here’s config.edn, it’s pretty much the same as before, but I’ve given the keys an extra prefix, so that they make a bit more sense as namespace names. This way we can follow Integrant’s conventions of putting multimethod defintions in namespaces that match the names of the keys.

{:li38.handler/time {:prefix "It's "
                     :suffix "."}
 :li38.ring/jetty   {:port 3000
                     :join? false
                     :handler #ig/ref :li38.handler/time}}

This allows me to split the implementations of each key into their own file, and have them be loaded automatically. Here’s the implementation for the handler.

(ns li38.handler
  (:require [integrant.core :as ig]))

(defmethod ig/init-key :li38.handler/time [_ config]
  (fn [{:keys [request-method uri] :as req}]
    {:status 200
     :body (str (:prefix config) (java.util.Date.) (:suffix config))
     :headers {"Content-Type" "text/plain"}}))

And here’s the Jetty server. Notice that this is simply the code from before, split up into separate files.

(ns li38.ring
  (:require [integrant.core :as ig]
            [ring.adapter.jetty :as ring-jetty]))

(defmethod ig/init-key :li38.ring/jetty [_ {:keys [handler port join?]}]
  (ring-jetty/run-jetty handler {:port port :join? join?}))

(defmethod ig/halt-key! :li38.ring/jetty [_ server]
  (.stop server))

(defmethod ig/suspend-key! :li38.ring/jetty [_ server]
  (.stop server))

(defmethod ig/resume-key :li38.ring/jetty [key opts old-opts server]
  (if (= opts old-opts)
    (.start server)
    (ig/init-key key opts)))

Now in the set-prep! function load the configuration, make sure all necessary namespaces are loaded, and finally call prep, to make sure the configuration is all filled in and ready.

(ig-repl/set-prep! (fn []
                     (let [config (load-config)]
                       (ig/load-namespaces config)
                       (ig/prep config))))

With that done all that’s left to do is call (go), and you’re up and running.

(ig-repl/go)
;;=> :initiated

integrant.repl takes care of storing your system behind the scenes, so for instance to stop it again, you just call integrant.repl/halt! without any arguments.

(ig-repl/halt)
;;=> :halted

The same goes for suspend and resume. Note that this time none of the method names end in a bang. It’s a bit of an inconsitency, that’s why I’m pointing it out.

(ig-repl/suspend)
;;=> :suspended

(ig-repl/resume)
;;=> :resumed

Finally we get to the real workhorses: reset and reset-all. These combine the code reloading facilities from clojure.tools.namespace with Integrant’s lifecycle functionality.

Calling reset will suspend the system, reload and namespaces that have changed since the last time you called reset, and then resume. Note that this calls the “prep” function again, so if the configuration changed it will pick up on that.

user> (ig-repl/reset)
:reloading (li38.handler li38.ring user)
:resumed
user>

Consequent calls to reset will only reload files that changed in the meantime. If you want to make sure everything gets reloaded anyway, then you use reset-all.

user> (ig-repl/reset-all)
:reloading (li38.handler li38.ring user)
:resumed
user>

Integrant is one of those small libraries that can be used in really novel and creative ways, and I encourage you to play around it with it and to observe how it influences your designs. I feel it encodes a really useful pattern for dynamically linking parts together, and I expect to see it pop up in surprising places in the future.

Show Notes

browse source code

The master branch contains all code shown in the episode. If you want to follow along with the episode then you don’t need any code to get started, all you need are the Clojure CLI tools (tools.deps).

If you’re using an nREPL based environment that doesn’t yet have support for tools.deps, then check out the repo at tag ep39-start, and launch nREPL with clj -Anrepl.

Cheat sheet

integrant.core

;; lifecycle functions in integrant.core
(init config)
(halt! system)
(suspend! system)
(resume config system)
(prep config)

;; accompanying multimethods
(defn init-key :your/key [key value]
  ;; return implementation
  )

(defn halt-key! :your/key [key implementation]
  ;; stop component, return value is ignored
  )

(defn suspend-key! :your/key [key implementation]
  ;; pause component, return value is ignored
  )

(defn resume-key :your/key [key config old-config implementation]
  ;; resume implementation, must return implementation
  )

(defn prep-key :your/key [key config]
  ;; return updated config
  )

;; other useful functions in integrant.core
(read-string "...edn...")   ;; like clojure.core/read-string, but knows how to read #ig/ref
(load-namespaces config)    ;; Tries loading namespaces based on config map keys
(find-derived config key)   ;; Find any keys in config (map) that derive from the given key

;; not mentioned in the episode, but still useful
(defmethod pre-init-spec :your/key [key]
  ;; return a clojure.spec that can validate a key's configuration
  )

(fold system-map f v)       ;; similar to `clojure.core/reduce`, but folds over a config/system map in dependency order

integrant.repl

(set-prep! f)            ;; Sets the function f that returns an integrant config map

(prep)                   ;; Calls the prep function, and stores the resulting config
(init)                   ;; Uses the stored config to start the system
(go)                     ;; Same as (do (prep) (init))

(halt)                   ;; Call the corresponding integrant function on the current config/system
(suspend)
(resume)

(clear)                  ;; Stop the system and clear out the stored config and system map

(reset)                  ;; Suspend - reload modified - resume
(reset-all)              ;; Suspend - reload all - resume

Links

To learn more about Clojure’s hierarchy features, check out the official guide about multimethods.

For a better understanding of “the reloaded workflow”, as well as several gotchas to watch out for, have a look at this Lambda Island blog post: reloading woes.

Episode 26, Component and System and episode 31, Component in Practice talk a lot more about Component and the reloaded workflow. They’re by no means a requirement for working with Integrant, but might provide a better understanding of the context that Integrant was born out of.

If all the talk about Ring handlers and adapters confused you, then check out the video collection about Ring and Compojure, it’s made with beginners in mind, and walks you through these fundamental concepts.