Lambda Island

Learn how to incorporate Component in a real world project. This episode is a bit of grab bag. It shows how to set up a -main function and how to use it, how to use reloaded.repl and tools.namespace for a better REPL workflow. You get introduced to the Suspendable protocol that complements Component’s Lifecycle protocol. Finally it shows how to streamline your configuration with Aero.

Transcript

In episode 26 you learned the basics of Stuart Sierra’s Component. It’s a small and simple library, but you could fill a book about its practical implications.

That’s because Component provides a language for speaking about systems. We’ve gone over the vocabulary, now it’s time to learn how to speak it for real.

In this episode I’ll show you how to use component in practice, to speed up your development workflow, improve your tests, and manage your application’s lifecycle in production.

Setting up the main system

Let’s start by looking at a typical way to structure your app with Component. The Chestnut application template establishes some conventions which form a good starting point.

Generate a new application with lein new chestnut, followed by the name of your app, and flags for the features you want to include. The Chestnut README has a complete list. I’m tend to use Ring’s site middleware instead of the default API middleware, mainly because it provides session support. I’ll use the http-kit server instead of the default Jetty, and I’m adding the Contributor covenant Code of Conduct to the project.

$ lein new chestnut ep31 +site-middleware +http-kit +code-of-conduct
Retrieving chestnut/lein-template/0.15.0-SNAPSHOT/lein-template-0.15.0-20170517.134032-2.pom from clojars
Generating fresh Chestnut project.
README.md contains instructions to get you started.

cd into your new app and open application.clj under src/clj/ep31. Let’s take that from the top. The first required namespace is Component itself, followed by a handful of readymade components provided by System. Together endpoint, handler, and middleware form a web stack, powered by the http-kit web server. How these different components work together is described in detail in Episode 26.

(ns ep31.application
  (:gen-class)
  (:require [com.stuartsierra.component :as component]
            [system.components.endpoint :refer [new-endpoint]]
            [system.components.handler :refer [new-handler]]
            [system.components.middleware :refer [new-middleware]]
            [system.components.http-kit :refer [new-web-server]]
            [ep31.config :refer [config]]
            [ep31.routes :refer [home-routes]]))

The config namespace provides a constructor function which pulls together configuration from various sources, and returns them as a map. In this case some values are coming from environment variables, while others are hard-coded. How you set up your configuration is up to you, I’ll show some alternative approaches later in this episode.

The good thing is that by having this centralized from the start, it’s easy to use some other configuration system later on.

(ns ep31.config
  (:require [environ.core :refer [env]]
            [ring.middleware.defaults :refer [wrap-defaults site-defaults]]
            [ring.middleware.gzip :refer [wrap-gzip]]
            [ring.middleware.logger :refer [wrap-with-logger]]))

(defn config []
  {:http-port  (Integer. (or (env :port) 10555))
   :middleware [[wrap-defaults site-defaults]
                wrap-with-logger
                wrap-gzip]})

The final namespace included from application is routes, which provides a constructor function for Compojure routes. It takes an endpoint component as argument. This component is really just a vehicle for injecting dependencies into the routes, making any component that endpoint depends on accessible.

As your app grows you could add more of these routes constructor functions, each with their own corresponding endpoint component.

(ns ep31.routes
  (:require [clojure.java.io :as io]
            [compojure.core :refer [ANY GET PUT POST DELETE routes]]
            [compojure.route :refer [resources]]
            [ring.util.response :refer [response]]))

(defn home-routes [endpoint]
  (routes
   (GET "/" _
     (-> "public/index.html"
         io/resource
         io/input-stream
         response
         (assoc :headers {"Content-Type" "text/html; charset=utf-8"})))
   (resources "/")))

Note that this code uses routes inside a function, instead of a top-level defroutes form. The difference is that with defroutes, the routes are created immediately and assigned to a var, while a constructor function only creates the routes at the time when the function gets called.

The benefit of using a function is that the routes declarations “close over” the function arguments, so they can solely rely on local state, instead of having to reach out to globals.

The downside is that, because the routes now form a closure, any changes to the constructor function only get picked up when the routes are constructed anew, in other words, when restarting the system.

(defroutes home-routes ,,,)
;; same as
(def home-routes (routes ,,,))

Back to the application namespace. The app-system function is again a constructor, it creates a system map based on the configuration. Nothing spectacular here. It sets up the routes and middleware stack, then combines them into a handler, which is passed on to the web server.

(defn app-system [config]
  (component/system-map
   :routes     (new-endpoint home-routes)
   :middleware (new-middleware {:middleware (:middleware config)})
   :handler    (-> (new-handler)
                   (component/using [:routes :middleware]))
   :http       (-> (new-web-server (:http-port config))
                   (component/using [:handler]))))

Now when the app starts you need to pull everything together: get the configuration, use it to create a system, then start the system. Where do you put that code?

Ok, you already see that main function there, but let’s pretend that isn’t there yet.

A first idea could be to add a top-level statement that sets up and starts the system. Because it’s at the top level, not inside a function, it will happen as soon as the namespace loads.

I’ve done things like that, as I’m sure many have, maybe using a defonce so at least you can still reload the namespace. But think about it, this means that just loading the namespace has the side-effect of starting the system, web server and all. That’s a pretty big smell.

Do you want to start the system whenever you start a REPL, run your tests, or even when running a linter such as Eastwood over your code? Probably not.

(def system
  (-> (config)
      app-system
      component/start))

So instead you use a -main fuction. It builds up the configuration map, creates the system, and starts it. Finally it’s helpful to print a message so you know it’s all started up and ready to serve.

If you also have this (:gen-class) in the namespace declaration, then Clojure will compile this namespace to a Java class. This -main function turns into a static main method, so the JVM knows that this is a potential entry point for starting the app.

(ns ep31.application
  (:gen-class)
  ,,,)

(defn app-system [config]
  ,,,)

(defn -main [& _]
  (let [config (config)]
    (-> config
        app-system
        component/start)
    (println "Started ep31 on" (str "http://localhost:" (:http-port config)))))

Now you can start your application by passing this main namespace name to lein run with the -m flag.

(Pedantically, what you’re actually passing is a class name. This namespace compiles to a class of the same name as the namespace.)

$ lein run -m ep31.application
Started ep31 on http://localhost:10555

Having a -main function together with (:gen-class) also allows you to run the app from an uberjar. Make sure the jar is on Java’s classpath by passing it t java with the -cp flag. After that put the name of the namespace (or to be exact, the name of the class that this namespace has been compiled to).

$ lein uberjar
Compiling ep31.config
Compiling ep31.routes
Compiling ep31.application
Compiling ep31.common
Compiling ClojureScript...
Compiling "resources/public/js/compiled/ep31.js" from ["src/cljs" "src/cljc"]...
Successfully compiled "resources/public/js/compiled/ep31.js" in 25.474 seconds.
Created /home/arne/ep31/target/ep31-0.1.0-SNAPSHOT.jar
Created /home/arne/ep31/target/ep31.jar
$ java -cp target/ep31.jar ep31.application
Started ep31 on http://localhost:10555

You can also configure ep31.application to be the default entry point for the application by setting a :main key in project.clj. Again Chestnut already took care of this.

(defproject ep31 "0.1.0-SNAPSHOT"
  ,,,
  :main ep31.application)

With this a simple lein run is enough to start the application.

Setting a :main namespace in project.clj also influences the jars that Leiningen creates, turning them into a sort of stand-alone application that you can run with java -jar.

$ lein run
$ lein uberjar
$ java -jar target/ep31.jar

Jars contain a special file with metadata called MANIFEST. If it contains a Main-Class entry, then java -jar knows that’s the class it needs to execute.

$ unzip -p target/ep31.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Built-By: arne
Created-By: Leiningen 2.6.1
Build-Jdk: 1.8.0_91
Main-Class: ep31.application

Migrating to Component

Notice that the -main function isn’t holding on to the system map. The system gets created and started, but then gets thrown away. This is often fine for production environments, where if you want to restart the app you typically restart the whole process.

But because there’s no reference to the system, there’s also no way for code to access components willy nilly. The only way to get hold of a component is by using constructor functions, like home-routes, which close over the necessary dependencies, so you need to factor that in when designing your application, to make sure components are passed around to where they are needed.

This isn’t unreasonable or hard to do, but if you have an existing project that you’re adding component to, then chances are you have a lot of state being accessed through global vars. If you first need to refactor all that code to use dependency injection, then chances are you’ll put off migrating to Component forever.

Instead you can have a gradual transition by storing the system in a var, and accessing components through that var. This should make it relatively easy to update your code. You’ll have to change any references to application state so they access things through the system, but you can put off refactoring to dependency injection for a bit.

In fact you can stop there, and just always refer to the global system map. This is akin to what mount does, an alternative to Component.

(declare system)

(defn app-system [config]
  (component/system-map
   ,,,))

(defn -main [& _]
  (alter-var-root
   #'system
   (fn [_]
     (-> (config)
         app-system
         component/start))))

Development setup with reloaded.repl

So that’s that, you have your main system declared, and you have a main entry point for booting it up in production. Now let’s look at how to set things up for development.

When working from a REPL it’s sometimes easy to get your application in a weird state. You could restart the REPL to get a clean slate again, but let’s just say that instant startup time isn’t one of Clojure’s selling points. So instead what you want to do is reload and restart your app, without having to restart Clojure and the JVM.

If you have your state in a bunch of global vars then that’s quite hard to do, but with Component it’s a breeze. This is one of the reasons why Component was created: it makes it easy to clean up and restart to get a clean slate again.

For some added convenience you can use a library called reloaded.repl. Let’s try it out from the REPL. Require reloaded.repl, as well as the application and config namespaces. Now set up an init function, so reloaded.repl knows how to create the system map.

Now you’re ready to kick everything off with (rr/go). This will create the system, and then start it. If you open your browser at port 10555 you should see the familiar “Hello Chestnut!”.

Now you can use (rr/stop) and (rr/start) to switch the system on or off. Use (rr/go) again if you want to initialize the system again from scratch. This will first stop the old system if it’s still running.

(Note that you can copy/paste these repl forms. The #_ makes sure that Clojure ignores the prompt.)

#_user=> (require '[reloaded.repl :as rr]
    #_=>          '[ep31.application :as app]
    #_=>          '[ep31.config :as config])
#_user=> (rr/set-init! #(app/app-system (config/config)))
#function[user/eval46336/fn--46337]
#_user=> (rr/go)
:started
#_user=> (rr/stop)
:stopped
#_user=> (rr/start)
:started

So this way you can easily restart the system, but you also need a way to reload your code. Otherwise it’s quite easy to end up in a situation where the code that Clojure has in memory is quite different from its source code. Ideally what you want to do is unload all vars and namespaces, and then load them again from scratch. This you can do with tools.namespace.

In clojure.tools.namespace.repl you can find two functions, refresh and refresh-all. The first tries to intelligently reload all namespaces that have been changed since the last refresh, while the other one simply reloads all namespaces. This isn’t a trivial operation, since both the unloading and the loading needs to happen in a specific order based on dependencies, so it’s good that tools.namespace is there to handle that for you.

This is annoying though, instead of doing its work refresh is raising an error. I had seen this happen several times before, and for the longest time had no clue what was causing it, but I finally figured it out.

#_user=> (require '[clojure.tools.namespace.repl :as tn])
#_user=> (tn/refresh)
:reloading (ep31.routes ep31.config ep31.application user ep31.common)
:error-while-loading ep31.application
#error {
 :cause "No namespace: ep31.config"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "java.lang.Exception: No namespace: ep31.config, compiling:(ep31/application.clj:1:1)"
   :at [clojure.lang.Compiler load "Compiler.java" 7391]}
  {:type java.lang.Exception
   :message "No namespace: ep31.config"
   :at [clojure.core$refer invokeStatic "core.clj" 4104]}]}

It’s not exactly obvious either, but it turns out the problem is that lein uberjar that we did earlier. It compiles namespaces to Java classes in a process called AOT or ahead-of-time compilation. These class files end up inside your project directory under target/classes, and they confuse the h*ck out of tools.namespace. So: long story short, if you run into this then do a lein clean to clean up any leftovers, or just delete target/classes, and finally restart your REPL (yes, this time you’ll have to).

$ lein clean
# or
$ rm -rf target/classes

Now let’s try that again. Here’s a fresh REPL. Require clojure.tools.namespace.repl, aliased to tn, then call refresh. Cool, that seems to be working.

Now try refresh-all… oops. The first refresh ended up unloading and then reloading the user namespace, and now that tn alias is gone. No biggie, I’ll show you a better way to call tools.namespace instead.

#_user=> (require '[clojure.tools.namespace.repl :as tn])
#_user=> (tn/refresh)
:reloading (ep31.routes ep31.config ep31.application user ep31.common)
:ok
#_user=> (require '[clojure.tools.namespace.repl :as tn])
#_user=> (tn/refresh-all)
:reloading (ep31.routes ep31.config ep31.application user ep31.common)
:ok

So reloaded.repl lets you easily restart your system, and tools.namespace helps with reloading your code. Wouldn’t it be great if you could do both in a single step? Why yes, Arne, it sure would be, which is why reloaded.repl provides helpful reset and reset-all functions to do just that.

#_user=> (require '[reloaded.repl :as rr])
#_user=> (rr/reset)
#_user=> (rr/reset-all)

Now that you’re familiar with reloaded.repl and tools.namespace, take a peek at dev/user.clj. New REPL sessions open up inside this user namespace, so here you can define extra convenience functions for use in the REPL. This file won’t be loaded when you run your app in production, so if you have dev-time dependencies specified in a :dev profile in your project.clj, then it’s safe to load them from here.

Down here you find all the main reloaded.repl functions aliased into user. This way you have them available as soon as you open a REPL, and they’ll still be there even after doing a refresh. This is how Chestnut provides an easy to use workflow, where you simply start a REPL, type (go), and you’re off.

$ lein repl
#_user=> (go)

When you want to make sure you have a clean slate again, simply call reset.

#_user=> (reset)

A few more things to notice in user.clj, this set-refresh-dirs is telling tools.namespace which directories to scan for changes. Then there’s reloaded.repl/set-init!, which we used earlier, but here it’s using a different constructor than the one we used earlier.

(set-refresh-dirs "src" "dev")
(reloaded.repl/set-init! #(dev-system))

This dev-system function is interesting, because it takes the main system defined in ep31.application, but it then adds some extra components to it with assoc. You can do that, a system map is is a record, which means it behaves a lot like a map, so you can assoc, dissoc, or merge to your heart’s content.

Having a separate dev-system can also be handy for swapping out some configuraiton, for instance, if you’re accessing third party services, then from your dev environment you can point them at some sandbox implementation instead.

(defn dev-system []
  (assoc (ep31.application/app-system (config))
    :figwheel-system (fw-sys/figwheel-system (fw-config/fetch-config))
    :css-watcher (fw-sys/css-watcher {:watch-paths ["resources/public/css"]})))

There’s one more thing I haven’t told you yet about reloaded.repl. When you (reset) or (reset-all), it doesn’t just start and stop the components, but it “suspends” and “resumes” them. These are operations provided by the Suspendable protocol, which complements Components’ Lifecycle protocol.

suspend defaults to calling stop, and resume defaults to calling start, so you don’tw necessarily have to implement them, but you can if you want to implement a “lightweight restart”, by carrying some of the state over to the new component.

(defprotocol Suspendable
  (suspend [component]
    "Suspend a component and return the suspended component. Expected to be
    called before a resume.")
  (resume [component old-component]
    "Given a new component and a suspended old component, return a component
    that incorporates the suspended data into the new component, if possible."))

As an example, suppose you have some kind of server component that sets up a server socket listening on a certain port. I have a kind of pseudo-code implementation here. I didn’t go to the trouble of making it idempotent, but for the rest it could be a real component.

Now I’m making it suspendable. The suspend operation doesn’t do anything in this case. In particular it’s not cleaning up the the socket, but keeps it around, being optimistic that it can be reused.

Now when resuming, you get passed both a newly created Server component, as well as the previous one which has been resumed. If it turns out they’re both using the same port, then you can reuse the existing server socket, by carrying it over from the old to the new component.

If the port is different then you fall back to actually stopping the old component, which closes the old socket, and then starting the new one.

(defrecord Server [port]
  component/Lifecycle
  (start [this]
    (assoc this :socket (open-server-socket port)))
  (stop [this]
    (close-socket (:socket this))
    (dissoc this :socket))

  suspendable/Suspendable
  (suspend [this]
    this)
  (resume [this old-this]
    (if (= port (:port old-this))
      (assoc this :socket (:socket old-this))
      (do
        (component/stop old-this)
        (component/start this)))))

If you want to learn more about the rationale behind Component and reloaded.repl, I recommend you check out this post from 2013 by Stuart Sierra, the author of Component, titled My Clojure Workflow, Reloaded.

In it he describes some of the reasoning behind Component. The workflow he describes was later captured in the reloaded.repl project by James Reeves, who also created Ring, Compojure, and is now working on the Duct framework.

For Emacs users it’s worth noting that tools.namespace support is baked into CIDER. With the cider-refresh command, bound to C-c C-x, you can call (refresh) to reload modified namespaces, or give it a prefix argument (C-u C-c C-x) to do a (refresh-all).

To make cider-refresh behave more like reloaded.repl/reset, you can configure two special variables, cider-refresh-before-fn and cider-refresh-after-fn, so that your system gets suspended before code reloading, and resumed afterwards.

(setq cider-refresh-before-fn "reloaded.repl/suspend")
(setq cider-refresh-after-fn  "reloaded.repl/resume")

Once again Chestnut has you covered though. The .dir-locals.el file in the root of the project defines these variables for you, so they will be correctly set for any file you open inside this project.

;; .dir-locals.el
((nil . ((cider-refresh-before-fn . "reloaded.repl/suspend")
         (cider-refresh-after-fn  . "reloaded.repl/resume")
         (cider-cljs-lein-repl    . "(do (user/run) (user/browser-repl))"))))

Advanced configuration

Let’s revisit that config namespace. It builds up a configuration map, based on environment variables and hard coded values. This is not an uncommon thing to do, in fact some platforms like Heroku very much encourage the use of environment variables for configuration, and it’s at the heart of the so called “twelve-factor app” methodology.

But environment variables are not without problems. They were never designed for storing sensitive information, and so they easily get leaked in error logs, or passed on to untrusted sub-processes.

(ns ep31.config
  (:require [environ.core :refer [env]]
            [ring.middleware.defaults :refer [wrap-defaults site-defaults]]
            [ring.middleware.gzip :refer [wrap-gzip]]
            [ring.middleware.logger :refer [wrap-with-logger]]))

(defn config []
  {:http-port  (Integer. (or (env :port) 10555))
   :middleware [[wrap-defaults site-defaults]
                wrap-with-logger
                wrap-gzip]})

Instead you could store configuration in a plain EDN file. This emphasizes that configuration is just data.

Juxt, a Clojure consoltancy from the UK, has taken that idea and turned it into a great little library called Aero. Let’s see what it can do.

First add Aero as a project dependency

(defproject ep31 "0.1.0-SNAPSHOT"

  :dependencies [,,,
                 [aero "1.1.2"]]
  ,,,)

Create a file called config.edn at the root of the project directory.

{:http-port 10555}

Now you can require Aero, and use the read-config function to load your config file.

(require '[aero.core :as aero])
(aero/read-config "config.edn")
;; => {:server/port 10555}

So it reads EDN, you may think: big deal. The cool thing is that it adds a bunch of tagged literals that lets you specify some complex rules for your configuration.

When running on Heroku, the :http-port comes from the PORT environment variable. That’s easy to do, with #env:

{:http-port #env PORT}

But if the environment variable isn’t set, then it should fall back to 10555. No problem, just use #or

{:http-port #or [#env PORT 10555]}

There’s a lot more that Aero can do. It can include other files, which is useful for secrets, and it can branch based on the current profile, like dev or prod. Have a look at the README. It covers all the different tags, and explains a few different approaches for how you can incorporate Aero into your app.

Component in Practice