The Classpath is a Lie

A key concept when working with Clojure is “the classpath”, a concept which we inherit from Clojure’s host language Java. It’s a sequence of paths that Clojure (or Java) checks when looking for a Clojure source file (.clj), a Java Class file (.class), or other resources. So it’s a lookup path, conceptually similar to the PATH in your shell, or the “library path” in other dynamic languages.

The classpath gets set when starting the JVM by using the -cp (or -classpath) command line flag.

java -cp src:/home/arne/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar clojure.main

Entries on this “classpath” are either directories (like src), or JAR files (like clojure-1.10.3.jar), which are really just zip files in disguise.

unzip -l ~/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar

When you require a namespace, Clojure will look for a corresponding .clj, .cljc, or .class file “on the classpath”. You can do the same by using clojure.java.io/resource.

(require '[clojure.java.io :as io])
(io/resource "clojure/main.class")
;;=> #object[java.net.URL 0x3237dfe5 "jar:file:/home/arne/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar!/clojure/main.class"]

Intuitively we think of this performing something like the following pseudocode:

(some #(find-file-in-directory-or-jar % "clojure/main.class") the-classpath)

It’s a useful mental model. It is also wrong. No my sweet summer child, in the world of ClassLoaders and URLClassPaths nothing is ever that straightforward.

The trouble starts when you want to do anything more than find a named resource on the classpath. Perhaps you want to inspect the classpath, iterate over all the files on the classpath, add or remove entries to or from the classpath. What you find out is that in Java

  • everything happens somewhere else
  • all the good bits are hidden from you

Everything Happens Somewhere Else

This is supposedly a quote from Adele Goldberg, one of the pioneers working at Xerox PARC:

In Smalltalk, everything happens somewhere else.

(I can’t find a good source to support this though, I’d be grateful if anyone is able to trace this to a primary source.)

Java is no different. You don’t just do stuff. You ask an object to do it, which asks another object, which delegates to its parent implementation, and so forth. When you need to look up stuff on “the classpath”, you ask a java.lang.ClassLoader.

(.getResource ^ClassLoader loader "clojure/main.class")

ClassLoader is an abstract class with many descendants, including

jdk.internal.loader.ClassLoaders$AppClassLoader
jdk.internal.loader.ClassLoaders$PlatformClassLoader
jdk.internal.loader.BootClassLoader
jdk.internal.loader.BuiltinClassLoader
jdk.internal.loader.SecureClassLoader
java.net.URLClassLoader
clojure.lang.DynamicClassLoader

Each class retains a reference to the classloader it was loaded with:

(.getClassLoader (class (fn [])))
;; => clojure.lang.DynamicClassLoader@4413660d

(.getClassLoader clojure.main)
;; => jdk.internal.loader.ClassLoaders$AppClassLoader@443b7951

(.getClassLoader java.sql.Time)
;; => jdk.internal.loader.ClassLoaders$PlatformClassLoader@4d131e92

(.getClassLoader String) ; more on this special case below
;; => nil

So we’ve established we need a classloader before we can do anything classpath-y. Where do we get one? If you need access to a ClassLoader in Java for some reason you typically just get the one that the class of this was loaded with, and use that.

this.getClass().getClassLoader()

But we don’t have this in Clojure. Let’s maybe see which classloader Clojure itself uses when it needs to require a namespace:

package clojure.lang;

public class RT {

static public ClassLoader baseLoader(){
	if(Compiler.LOADER.isBound())
		return (ClassLoader) Compiler.LOADER.deref();
	else if(booleanCast(USE_CONTEXT_CLASSLOADER.deref()))
		return Thread.currentThread().getContextClassLoader();
	return Compiler.class.getClassLoader();
}

}

It first checks the clojure.lang.Compiler/LOADER dynamic var. From scouring the code it seems this is used to set the loader internally during a specific scope, but what this ultimately is used for I have no idea. It does give you a way to override the classloader that Clojure uses, by giving that var a root binding. This is something we do in Kaocha to allow us to add test directories to the classpath at runtime, although I’m not sure this is recommended, and I may reconsider how we do that after having leveled up considerably recently when it comes to classpath shenanigans.

Next it uses the “context class loader”, if USE_CONTEXT_CLASSLOADER is true, which by default it is. This one is interesting, it’s a thread-local, mutable ClassLoader field, so you can setContextClassLoader as well as getContextClassLoader.

What’s this for? According to this StackOverflow post which has lots of juicy details “[it] exists only because whoever designed the ObjectInputStream API forgot to accept the ClassLoader as a parameter, and this mistake has haunted the Java community to this day”. Perhaps a tad dramatic. Fact is that this is the ClassLoader Clojure looks at (under typical circumstances). And since it’s mutable that gives us some options for doing… interesting stuff.

The Classloader Chain

ClassLoaders don’t come alone, they bring all their ancestors with them. Each classloader has a reference to a parent.

(defn classloader-chain [cl]
   (take-while identity (iterate #(.getParent %) cl)))

When evaluating this in Clojure you should see at least three entries, one provided by Clojure, and two by Java. There is actually one more ClassLoader, the BootLoader, but it’s built-in to the virtual machine, you don’t get to see it. It’s represented by nil.

(classloader-chain (clojure.lang.RT/baseLoader))
;;=>
[clojure.lang.DynamicClassLoader@107842e8
 jdk.internal.loader.ClassLoaders$AppClassLoader@443b7951
 jdk.internal.loader.ClassLoaders$PlatformClassLoader@4d131e92]

(To keep the spacial metaphors straight, I’m going to refer to this list interchangeably as the “stack” or “chain” of classloaders. The ones higher in the list will be “up” the stack, the ones lower in this last are “down” the stack.)

If you evaluate this using nREPL you may instead have gotten a much longer list, with a whole bunch of clojure.lang.DynamicClassLoader instances at the top, followed by the same AppClassLoader and PlatformClassLoader. This is something I had noticed before, and never had gotten a satisfying answer about, until Daniel Szmulewicz recently blogged about it. I highly recommend reading his post, it makes a good complement for this one, and will help deepen your understanding.

So what happens when you call getResource or getResources on Clojure’s class loader? When you dig in you’ll see that neither DynamicClassLoader nor its parent URLClassLoader implement getResource, so they inherit the base implementation in java.lang.ClassLoader.

package java.lang;

public abstract class ClassLoader {
    public URL getResource(String name) {
        URL url;
        if (parent != null) {
            url = parent.getResource(name);
        } else {
            url = BootLoader.findResource(name);
        }
        if (url == null) {
            url = findResource(name);
        }
        return url;
    }
}

This checks in order:

  • the parent classloader
  • the BootLoader
  • findResource on this classloader

findResource is where classes like URLClassLoader implement their own logic for finding resources, getResource wraps findResource, but with added logic for traversing down the chain of parent classloaders.

What’s important to notice here is that the parent is checked first. This is a key insight, it means that if any of the classloaders down the chain return a resource, then the current classloader is simply bypassed.

And what’s that BootLoader stuff in the middle? It allows adding classpath entries with special -Xbootclasspath flags. These are always checked first, so this lets you replace Java’s own classes with patched versions.

Looking again at this classloader-chain, we now know that when looking for a resource, these are checked bottom-to-top.

[clojure.lang.DynamicClassLoader@107842e8
 jdk.internal.loader.ClassLoaders$AppClassLoader@443b7951
 jdk.internal.loader.ClassLoaders$PlatformClassLoader@4d131e92
 nil]

First the built-in “boot classloader” is checked, providing some core classes that are baked into the JVM, like String, and fundemenatal things like java.base or java.logging.

Then the PlatformClassLoader is checked. Here you find classes provided by Java SE. This class loader was introduced with the module system introduced in Java 9, and doesn’t use a classpath at all, but it knows how to load classes from specific core modules like java.sql or java.xml.dom.

Then we get the AppClassLoader, also called the “system class loader”. This is where “the classpath” (the one we passed to java with -classpath) gets checked.

If none of those find the resource, then Clojure’s DynamicClassLoader gets a turn. In this case it doesn’t do anything of its own, it simply inherits the implementation from URLClassLoader, and by default it does not have any URLs in the list that it checks.

So what’s it for? It implements methods like findClass and loadClass, to return classes which are only defined in memory. This allows the compiler to generate bytecode on the fly when you evaluate forms, without having to create .class files on disk. So far we’ve mostly talked about locating resources via classloaders, but as the name suggests their first use is to find “classes”, i.e. to load byte code from disk and turn it into a class definition that the JVM can work with.

(defn xxx [])
  
(.loadClass (clojure.lang.RT/baseLoader) "user$xxx")
;; user$xxx

(.loadClass (ClassLoader/getPlatformClassLoader) "user$xxx")
;; => java.lang.ClassNotFoundException

Here you see that when you define a function, it really creates a class (and an instance thereof). Clojure’s classloader knows about this class, it can find it in memory. Java’s classloader has no idea.

So what do you do with this?

Ok, that was already a lot of theory and nitty gritty details… why am I doing this to myself? (and to you, dear reader)

One thing I like to be able to do is add dependencies to a project without having to restart the REPL process every time. For a brief period in time this worked wonderfully for me using Pomegranate, but since Java 9 this stopped working, and my pleas for help largely fell on deaf ears.

Since then I found the Compiler/LOADER workaround which we use in Kaocha, and there’s an experimental tools.deps add-lib3 branch that adds this functionality directly to tools.deps.alpha. Cool beans!

Now let’s add a little twist. We have over a dozen Lambda Island open source libraries at this point, and many depend on each other. It happens regularly that you are working on one library, and halfway through you figure out you need some related changes or additions in another library.

The typical thing to do is to change deps.edn to use a :local/root reference, restart your REPL, and continue from there. When your REPL is quick to restart and you don’t have much state to build up again then maybe that’s not a big deal, but it can get pretty annoying.

We started running into the same problem with Nextjournal. We’re in the process of extracting and releasing some of the modules that go into making Nextjournal, the way we’ve already open-sourced clojure-mode. (Keep an eye out for this, it’s good stuff!)

But that means going from monorepo bliss to a situation where there’s a lot more overhead in maintaining and coordinating these things. On top of that for a big application like Nextjournal restarting your REPL can take a little while, it’s something we really like to avoid.

So we tried adding it with add-lib.

(require '[clojure.tools.deps.alpha.repl :as deps-repl]
         '[clojure.java.classpath :as cp]
         '[clojure.java.io :as io])

(deps-repl/add-lib {nextjournal.clojure-mode {:local/root "../clojure-mode"}})

(filter #(re-find #"clojure-mode" (str %)) (cp/classpath))
;; => ("/home/arne/Nextjournal/clojure-mode")

(io/resource "nextjournal/clojure_mode.cljs")
;;=> "/home/arne/.gitlibs/libs/nextjournal/clojure-mode/a83c87cd2bd2049b70613f360336a096d15c5518/src/nextjournal/clojure_mode.cljs"

Ok, what’s going on here? We’ve tried adding the :local/root version of clojure-mode to the classpath. We can see it’s on there, and yet when we look for something specific we see it’s still getting looked up in the gitlib version. Frustrating!

Let’s first get some better insight into what’s happening, by looking at the stack of classloaders again, and for each checking which locations it checks.

(defn classpath-chain
  "Return a list of classloader names, and the URLs they have on their classpath"
  []
  (for [cl (classloader-chain)]
    [(symbol
      (or (.getName cl)
          (str cl)))
     (map str (cond
                (instance? URLClassLoader cl)
                (.getURLs cl)
                (= "app" (.getName cl))
                (map #(File. ^String %)
                 (.split (System/getProperty "java.class.path")
                         (System/getProperty "path.separator")))))]))

Here we loop over the (classloader-chain) we had earlier. Some of them will be instances of URLClassLoader, and for these we can simply ask them what URLs they check.

For the app/system classloader the situation is a little different. Up to Java 8 this was also a URLClassLoader, but since Java 9 that’s no longer the case, and there’s no good way to inspect its actual classpath. It contains a URLClassPath instance, but it’s private. Remember, all the good bits are always out of reach.

But we can look at how it’s being initialized:

package jdk.internal.loaders

public class ClassLoaders {
    private static final BootClassLoader BOOT_LOADER;
    private static final PlatformClassLoader PLATFORM_LOADER;
    private static final AppClassLoader APP_LOADER;
    
    static {
        // ...
        String cp = System.getProperty("java.class.path");
        URLClassPath ucp = new URLClassPath(cp, false);
        APP_LOADER = new AppClassLoader(PLATFORM_LOADER, ucp);
    }
}

It takes the java.class.path system property as its path, so assuming no one has changed the property since then, this gives us a way to find out what classpath it’s looking at.

(classpath-chain)
;;=>
([clojure.lang.DynamicClassLoader@711fe6bb ("/home/arne/Nextjournal/clojure-mode")]
 [app
  ("dev"
   "test"
   "src"
   "resources"
   "/home/arne/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar"
   "/home/arne/.m2/repository/org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.jar"
   "/home/arne/.m2/repository/org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.jar"
   "/home/arne/.gitlibs/libs/nextjournal/clojure-mode/a83c87cd2bd2049b70613f360336a096d15c5518/src/nextjournal/clojure_mode.cljs")]
 [platform ()])

As you can see add-lib added the new directory to Clojure’s DynamicClassLoader, but the gitlib version is still part of the application class loader. And since we know that loaders lower down the stack are checked first, files in the gitlib will shadow files in the :local/root.

Why wasn’t (cp/classpath) telling us this? Turns out the current implementation is flawed, as soon as the DynamicClassLoader contains a URL (as is the case after calling add-lib, it completely ignores the system classpath, even though in reality it is still checked (and even gets priority!).

Sadly we can’t change the system classloader. Its classpath may as well be set in stone. But we can define our own classloader, one which plays by our own rules.

The plan is as follows:

  • Define our own subclass of URLClassLoader
  • Install it directly above the bottom-most DynamicClassLoader (one DynamicClassLoader is enough, the fact that there may be many is an unfortunate side-effect of how Clojure and nREPL interact).
  • Have it first check its own paths, then those of its parent, and only then delegate further to Java’s classloaders

Note that the old version (the gitlib) will still be there, but we only look for files there if they don’t exist in the :local/root version. This works fine if you are only changing or adding files, but if you delete a file you may start seeing it pick up the old version.

(defn priority-classloader
  [cl urls]
  (let [cp-files (map io/as-file urls)
        find-resources (fn [^String name]
                         (mapcat (fn [^File cp-entry]
                                   (cond
                                     (and (cp/jar-file? cp-entry)
                                          (some #{name} (cp/filenames-in-jar (JarFile. cp-entry))))
                                     [(URL. (str "jar:file:" cp-entry "!/" name))]
                                     (.exists (io/file cp-entry name))
                                     [(URL. (str "file:" (io/file cp-entry name)))]))
                                 cp-files))]
    (proxy [URLClassLoader] [(str `priority-classloader) (into-array URL urls) cl]
      (getResource [name]
        (or (first (find-resources name))
            (.findResource (.getParent this) name)
            (.getResource (.getParent this) name)))
      (getResources [name]
        (java.util.Collections/enumeration
         (distinct
          (concat
           (find-resources name)
           (mapcat
            enumeration-seq
            [(.findResources (.getParent this) name)
             (.getResources (.getParent (.getParent this)) name)]))))))))

Ok this is getting a little hairy. I was hoping I could just call (.findResource this ...) to have it search the list of paths, but that’s not working… I followed the trail from URLClassLoader to URLClassPath to Loader (remember how in Java everything always happens somewhere else?), and eventually gave up and implemented my own find-resources logic.

The important bits are here:

(or (first (find-resources name))          ; Search our own paths
    (.findResource (.getParent this) name) ; Search DynamicClassLoader
    (.getResource (.getParent this) name)) ; Search App/Platform/Boot

Now let’s try installing it (root-loader is helper to find the DynamicClassLoader that sits immediately above the application classloader, like I said, one is enough):

(.setContextClassLoader 
 (Thread/currentThread) 
 (priority-loader (root-loader) ["/home/arne/Nextjournal/clojure-mode"]))

That should do the trick, but it doesn’t, at least not when you evaluate this from nREPL. The problem there is that nREPL captures the current context classloader at the beginning of each eval operation, and restores it afterwards. So we need to somehow set it “outside” of the current eval.

(let [thread (Thread/currentThread)]
  (future
    (Thread/sleep 100)
    (.setContextClassLoader 
     thread
     (priority-loader (root-loader (context-classloader thread) ["/home/arne/Nextjournal/clojure-mode"])))))

We use a future which closes over the current thread, and then update the thread’s context classloader from the future, which runs on a different thread.

And… it works! Except that when you try to navigate with something like cider-find-var, presumably because that uses a separate nREPL session, which runs on a different thread, which has its own classloader. Here too there are solutions. The first thing I tried was simply forcing Orchard (the library backing CIDER) to use a specific classloader, and that works. Now I’m leaning towards looping over all threads, and installing this priority-loader on every thread that has a DynamicClassLoader.

Conclusion

So what did we learn?

  • There’s really no such thing as “the classpath”, but there’s a hierarchy of class loaders, and you can kindly ask them to give you stuff.
  • Clojure has two places where it looks for a classloader, Compiler/LOADER and the thread’s context classloader. Setting the first is easy, setting the second requires some more care.
  • Adding something to “the classpath” is easy once you have a DynamicClassLoader/URLClassLoader, but removing something that is part of the classpath the application started with is impossible, and replacing something only kind of works by using some heavy trickery to make the new entry take precedence over the old.
  • Use the source! Clojure and Java are both open source, which is fantastic. I would not have been able to get this far without having these sources available.

This is not the last word on these topics, these experiments are ongoing, we’ve set up a repo under lambdaisland/classpath where you can find the current state of things. In particular it contains this cool helper:

(update-classpath!
 '{:aliases [:dev :test :licp]
   :extra {:deps {com.lambdaisland/webstuff {:local/root "/home/arne/github/lambdaisland/webstuff"}}}})

This will read deps.edn, use it to construct a “basis” with the given options, and then add any new entries to the classpath. It’s as close to a deps.edn reload as you’ll get, and you can add local overrides, as I’ve shown here.

We’re also working on other quality-of-life helpers in there, like git-pull-lib to update the :git/sha of a library to the latest commit in a branch.

If you want to discuss this post just head on over to ClojureVerse!

More blog posts

A Tale of Three Clojures

By Alys Brooks

Recently, I was helping a coworker debug an issue with loading a Clojure dependency from a Git repository. (If you don’t know you can do this; it’s very handy. Here’s a guide.) I realized that there were really two Clojures at play: the Clojure that clojure was running to generate the classpath and the Clojure that was used by the actual Clojure program.

Taking a step back, there are really three things we might mean when we say “Clojure”:

Launching the Lambda Island Redesign

It’s finally live! A gorgeous, in-depth redesign of the Lambda Island website. After months of hard work we soft-launched earlier this week. Today we want to tell you a little more about the project, the whys and the hows, and to invite you to check it out. And when you’re done do come tell us what you think on our Discord.

Redesigned front page

We already told you in a previous post how Lambda Island and Gaiwan are changing. In a short amount of time we went from a one man endeavor to a team of six, drastically changing what we are able to take on and pull off.

Lambda Island Open Source Update July 2021

By Alys Brooks

It’s been a while since our last update! Community contributors and Lambda Island team members have been busy working on new projects and improving existing ones. We’re trying to strike a balance between venturing into new territory while ensuring existing projects continue to improve.

If you want to join in, check out our blog post calling for contributions, or jump right in to our first issues list.

Lambda Island is Changing

Last month marked the five year anniversary of Lambda Island. Five years since I quit my job, built a video platform, and figured out how to put together tutorial videos. It’s been quite a ride. All this time I’ve been fortunate to be part of the Clojure community, to watch it grow and evolve, and to help individuals and companies to navigate these waters.

I learned some hard lessons along the way. Like how hard it is to bootstrap an educational content business catering to a niche audience, or how lonely and stressful it can be in business to go it alone.

And with these lessons came changes. I started doing more consulting work again. To some extent it was a necessity, but it also provided me with an opportunity to get involved with many Clojure projects out there in the wild. To work with talented individuals, to learn what amazing things people were doing, and to help shape these companies, products, and the greater narrative around Clojure as a technology and community.

Why are Clojure beginners just like vegans searching for good cheese?

By Ariel Alexi

Have you ever wondered what to do if you love cheese but you want to be a vegan and how this affects you when you learn Clojure?

This was my question too. I love cheese, but I wanted to find a good vegan cheese replacement for my breakfast sandwich. This wasn’t a fun journey, all the vegan cheese substitutes weren’t like the real deal! After a talk with one of my vegan friends, he told me that my point of view over cheese replacement in the sandwich was wrong. Instead of searching for a good and tasty vegan cheese replacement, just replace it with an avocado. Avocado is a good replacement for your morning sandwich without the compromising that you would do with tasteless vegan cheese substitutes.

The beginner's way

By Ariel Alexi

An OOP developer finding her way in the functional world, what could go wrong?

So why Clojure?

Call for Contributions to Lambda Island Open Source

By Alys Brooks

We’re always excited when someone new contributes a fix, test, documentation update, or even a major new feature to our open source projects. If you’re new to programming, open source projects are a great way to get experience working on a larger project. If you’re not so new, they can be an opportunity to try a new technology or work on a kind of software you usually don’t get a chance to. Either way, it’s rewarding to help out your fellow users or developers—these issues are for Kaocha, one of the most popular and advanced Clojure test runners.

But open source can also be intimidating. If you haven’t been sure where to start, then this post is for you! We’ve outlined the process step by step, and we’re here to support you every step of the way.

Logging in Practice with Glögi and Pedestal.log

In an earlier post I tried to map out the Clojure/Java logging landscape a bit, so people can make an informed choice about how to set up logging for their application. There are several solutions out there with different trade-offs, and recently both Timbre and mulog have had some new releases. However until further notice the default logging setup in the “Lambda Island Stack” remains pedestal.log on the backend, and Glögi at the front.

This blog post is a bit of practical advice on how to make good use of logging, with the focus on these two libraries, which both provide the same API, just for a different platform.

This API consists of a number of macros corresponding with the different log levels, which all take key-value pairs as arguments.

Well Behaved Command Line Tools

Yesterday Vlaaad wrote a pretty cool blog post titled Alternative to tools.cli in 10 lines of code. It’s a great read and a very cool hack. It uses read-string to parse command line arguments, resolves symbols, and invokes the first symbol/function, passing in the rest of the parsed arguments. In other words: it’s the most straightforward way to translate a command line invocation into a Clojure function call.

The benefits are illustrated well in the post. It removes all possible friction. Want to add a new subcommand? Just add a function. Adding CLI arguments and options equals adding arguments and options to said function.

It’s actually not too different from what people do in shell scripts sometimes.

Logging in Clojure: Making Sense of the Mess

You may also like Dates in Clojure: Making Sense of the Mess

Logging seems like a simple enough concept. It’s basically println with a couple of extra smarts. And yet it can be oh so confounding. Even when you literally do not care a sigle bit about logging you may still be sent down a rabbit hole because of warnings of things that are or aren’t on the classpath. What even is a classpath?

Lambda Island Open Source Update May 2020

Felipe and I have been busy again, read all about what we’ve been up to in this month’s Lambda Island Open Source update.

We currently have two major projects we are working to get out: Regal and Kaocha-cljs2. They are both still a work in progress, but we’ve again made some good progress this month. Regal is not officially out yet but is at the stage where people can safely start incorporating it into their projects. Kaocha-cljs2 is a big undertaking, but we’re splitting this work into smaller self-contained pieces, and two of those saw their first release this month: Chui and Funnel.

Regal

Lambda Island Open Source Update April 2020

With people across the globe isolating themselves it feels sometimes like time is standing still, but looking back it’s been a busy month again for the folks here at Lambda Island. Besides all the maintenance releases we shipped some new features for lambdaisland.uri, our JS logging library Glögi, and a new micro-library, edn-lines. We also have some exciting work in progress, including a brand new browser-based test runner for ClojureScript, written with shadow-cljs in mind.

Funding

A big shout goes out to two of our clients, Nextjournal and Pitch. They have made a lot of this work possible either by funding projects directly, or by dogfooding our tools and libraries and giving us an opportunity to improve them. Thank you both for investing in this great ecosystem.

Coffee Grinders, part 2

Back in December I wrote about a coding pattern that I’ve been using more and more often in my work, which I dubbed “Coffee Grinders”. My thoughts around this were still gelling at the time, and so the result was a meandering semi-philosophical post that didn’t really get to the point, and that didn’t seem to resonate that much with people.

Some of the responses I got were “just use functions” or “sounds like a finite-state machine”, which makes it clear that I was not in fact making myself clear.

Having continued to encounter and apply this pattern I’d like to present a more concise, semi-formal definition of coffee grinders.

Advent 2019 part 24, The Last Post

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Day 24, I made it! I did skip a day because I was sick and decided it was more important to rest up, but I’m pretty happy with how far I got.

Let’s see how the others fared who took on the challenge. John Stevenson at Practicalli got four posts out spread out across the advent period, similar to lighting an extra candle every sunday of the advent. Good job!

Advent 2019 part 23, Full size SVG with Reagent

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

I’ve been a big fan of SVG since the early 2000’s. To me it’s one of the great victories of web standards. Mind you, browser support has taken a long time to catch up. Back then you had to embed your SVG file with an <object> tag, and support for the many cool features and modules was limited and inconsistent.

These days of course you can drop an <svg> tag straight into your HTML. What a joy! And since the SVG can now go straight into the DOM, you can draw your SVG with React/Reagent. Now there’s a killer combo.

Advent 2019 part 21, Project level Emacs config with .dir-locals.el

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

An extremely useful Emacs feature which I learned about much too late is the .dir-locals.el. It allows you to define variables which will then be set whenever you open a file in the directory where .dir-locals.el is located (or any subdirectory thereof).

Here’s an example of a .dir-locals.el file of a project I was poking at today.

Advent 2019 part 20, Life Hacks aka Emacs Ginger Tea

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Here’s a great life hack, to peal ginger don’t use a knife, use a spoon. I’m not kidding. Just scrape off the peal with the tip of the spoon, it’s almost too easy.

Here’s a Clojure + Emacs life hack:

Advent 2019 part 19, Advent of Random Hacks

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Something I’m pretty good at is coming up with random hacks. The thing where you’re like “hey how about we plug this thing into that thing” and everyone says “why would you do that that’s a terrible idea” and I’m like (mario voice) “let’s a go”.

And sometimes the result is not entirely useless. Like this little oneliner I came up with yesterday, using Babashka to “convert” a project.clj into a deps.edn.

Advent 2019 part 17, trace! and untrace!

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Here’s a little REPL helper that you may like.

(defn trace! [v]
  (let [m    (meta v)
        n    (symbol (str (ns-name (:ns m))) (str (:name m)))
        orig (:trace/orig m @v)]
    (alter-var-root v (constantly (fn [& args]
                                    (prn (cons n args))
                                    (apply orig args))))
    (alter-meta! v assoc :trace/orig orig)))

(defn untrace! [v]
  (when-let [orig (:trace/orig (meta v))]
    (alter-var-root v (constantly orig))
    (alter-meta! v dissoc :trace/orig)))

Advent 2019 part 16, Coffee Grinders

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Over the last year or so I’ve found myself using some variations on a certain pattern when modelling processes in Clojure. It’s kind of like a event loop, but adapted to the functional, immutable nature of Clojure. For lack of a better name I’m calling these coffee grinders. (The analogy doesn’t even really work but the kid needs to have a name.)

Since I saw Avdi Grimm’s OOPS Keynote at Keep Ruby Weird last year I’ve been thinking a lot about the transaction vs process dichotomy. Avdi talks about the “Transactional Fallacy” from around 15:25. From his slides:

Advent 2019 part 15, jcmd and jstack

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Two shell commands anyone using JVM languages should be familiar with are jcmd and jstack. They are probably already available on your system, as they come bundled with the JDK. Try it out, run jcmd in a terminal.

This is what the result might look like

Advent 2019 part 13, Datomic Test Factories

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

When I started consulting for Nextjournal I helped them out a lot with tooling and testing. Their data model is fairly complex, which made it hard to do setup in tests. I created a factory based approach for them, which has served the team well ever since.

First some preliminaries. At Nextjournal we’re big fans of Datomic, and so naturally we have a Datomic connection as part of the Integrant system map.

Advent 2019 part 12, Pairing in the Cloud with Tmux

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

I’m a strong believer in pair programming. It can be intense and exhausting, and its a skill you need to learn and get good at, but it’s extremely valuable. It improves knowledge sharing, prevents mistakes, and helps people to stay on track to make sure they are building the right thing, which is arguably one of the hardest aspects of our job.

But Gaiwan is a remote-first company. We are spread out across Germany, Brazil, Italy, and work with clients as far away as Singapore and Hong Kong, so we need good ways to pair remotely. For this we need a tool that is

Advent 2019 part 11, Integrant in Practice

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

I’ve been a fan of Integrant pretty much ever since it came out. For me there is still nothing that can rival it.

The recently released clip by the folks from Juxt does deserve an honorable mention. It has an interesting alternative approach which some may prefer, but it does not resonate with me. I prefer my system configuration to be just data, rather than code wrapped in data.

Advent 2019 part 10, Hillcharts with Firebase and Shadow-cljs

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Recently I led a workshop for a client to help them improve their development process, and we talked a lot about Shape Up, a book released by Basecamp earlier this year that talks about their process. You can read it for free on-line, and I can very much recommend doing so. It’s not a long read and there are a ton of good ideas in there.

One of these ideas has also become a feature in Basecamp, namely hill charts. These provide a great way to communicate what stage a piece of work is in. Are you still going uphill, figuring things out and discovering new work, or are you going downhill, where it’s mostly clear what things will look like, and you’re just executing what you discovered?

Advent 2019 part 9, Dynamic Vars in ClojureScript

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Clojure has this great feature called Dynamic Vars, it lets you create variables which can be dynamically bound, rather than lexically. Lexical (from Ancient Greek λέξις (léxis) word) in this case means “according to how it is written”. let bindings for instance are lexical.

(defn hello [x]
  (str "hello " x))

(defn greetings []
  (str "greetings" foo)) ;; *error*

(let [foo 123]
  (hello foo)
  (greetings))

Advent 2019 part 8, Everything is (not) a pipe

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

I’ve always been a big UNIX fan. I can hold my own in a shell script, and I really like the philosophy of simple tools working on a uniform IO abstraction. Uniform abstractions are a huge enabler in heterogenous systems. Just think of Uniform Resource Locators and Identifier (URLs/URIs), one of the cornerstones of the web as we know it.

Unfortunately since coming to Clojure I feel like I’ve lost of some of that power. I’m usually developing against a Clojure process running inside (or at least connected to) my trusty editor, and the terminal plays second fiddle. How do I pipe things into or out of that?

Advent 2019 part 7, Do that doto

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

doto is a bit of an oddball in the Clojure repertoire, because Clojure is a functional language that emphasizes pure functions and immutabilty, and doto only makes sense when dealing with side effects.

To recap, doto takes a value and a number of function or method call forms. It executes each form, passing the value in as the first argument. At the end of the ride it returns the original value.

Advent 2019 part 6, A small idiom

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

As an avid tea drinker I’ve been poring (pouring?) over this catalog of teas.

(def teas [{:name "Dongding"
            :type :oolong}
           {:name "Longjing"
            :type :green}
           {:name "Baozhong"
            :type :oolong}
           {:name "Taiwan no. 18"
            :type :black}
           {:name "Dayuling"
            :type :oolong}
           {:name "Biluochun"
            :type :green}])

Advent 2019 part 5, Clojure in the shell

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

I already showed you netcat, and how it combines perfectly with socket REPLs. But what if all you have is an nREPL connection? Then you use rep

$ rep '(clojure.tools.namespace.repl/refresh)'
:reloading ()
:ok

Advent 2019 part 4, A useful idiom

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Here’s a little Clojure idiom that never fails to bring me joy.

(into {} (map (juxt key val)) m)

Advent 2019 part 3, `every-pred` and `some-fn`

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Ah clojure.core, it’s like an all you can eat hot-pot. Just when you think you’ve scooped up all it has to offer, you discover another small but delicious delicacy floating in the spicy broth.

In exactly the same way I recently became aware of two functions that until now had only existed on the periphery of my awareness. I’ve since enjoyed using them on several occasions, and keep finding uses for them.

Advent 2019 part 2, Piping hot network sockets with Netcat

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

Part of what I want to do in this series is simply point at some of the useful tools and libraries I discovered in the past year. I’ve adopted a few tools for doing network stuff on the command line which I’ll show you in another post. First though we’ll look at a classic: netcat!

I’ve been using netcat for years, it’s such a great tool. It simply sets up a TCP connection and connects it to STDIN/STDOUT. Pretty straightforward. I’ve been using it more and more though because of Clojure’s socket REPL.

Advent 2019 part 1, Clojure Vocab: to Reify

This post is part of Advent of Parens 2019, my attempt to publish one blog post a day during the 24 days of the advent.

An interesting aspect of the Clojure community, for better or for worse, is that it forms a kind of linguistic bubble. We use certain words that aren’t particularly common in daily speech, like “accretion”, or use innocuous little words to refer to something very specific. Even a simple word like “simple” is no longer that simple.

We can thank Rich Hickey for this. He seems to care a great deal about language, and is very careful in picking the words he uses in his code, documentation, and in his talks.

Advent of Parens 2019

Ah, the advent, the four weeks leading up to Christmas. That period of glühwein and office year-end parties.

The last couple of years I’ve taken part in the Advent of Code, a series of programming puzzles posted daily. They’re generally fun to do and wrapped in a nice narrative. They also as the days progress start taking up way too much of my time, so this year I won’t be partaking in Advent of Code, instead I’m trying something new.

From the first to the 24th of December I challenge myself to write a single small blog post every day. If my friend Sarah Mirk can do a daily zine for a whole year, surely I can muster a few daily paragraphs for four weeks.

Lambda Island Streaming Live this Thursday and Friday

We are definitely back from holidays, and to demonstrate that we’re not just doing one but two live stream events!

Felipe and Arne pairing

Thursday 5 September, 13:00 to 15:00 UTC

Fork This Conference

Last weekend Heart of Clojure took place in Leuven, Belgium. As one of the core organizers it was extremely gratifying to see this event come to life. We started with a vision of a particular type of event we wanted to create, and I feel like we delivered on all fronts.

For an impression of what it was like you can check out Malwine’s comic summary, or Manuel’s blog post.

It seems people had a good time, and a lot of people are already asking about the next edition. However we don’t intend to make this a yearly recurring conference. We might be back in two years, maybe with another Heart of Clojure, maybe with something else. We need to think about that.

Advice to My Younger Self

When I was 16 I was visited by a man who said he had come from the future. He had traveled twenty years back to 1999 to sit down with me and have a chat.

We talked for an hour or so, and in the end he gave me a few pieces of advice. I have lived by these and they have served me well, and now I dispense this advice to you.

Become allergic to The Churn

ClojureScript logging with goog.log

This post explores goog.log, and builds an idiomatic ClojureScript wrapper, with support for cljs-devtools, cross-platform logging (by being API-compatible with Pedestal Log), and logging in production.

This deep dive into GCL’s logging functionality was inspired by work done with Nextjournal, whose support greatly helped in putting this library together.

Clojure’s standard library isn’t as “batteries included” as, say, Python. This is because Clojure and ClojureScript are hosted languages. They rely on a host platform to provide the lower level runtime functionality, which also allows them to tap into the host language’s standard library and ecosystem. That’s your batteries right there.

The Art of Tree Shaping with Clojure Zippers

This is a talk I did for the “Den of Clojure” meetup in Denver, Colorado. Enjoy!

Captions (subtitles) are available, and you can find the transcript below, as well as slides over here.

For comments and discussion please refer to this post on r/Clojure.

Test Wars: A New Hope

Yesterday was the first day for me on a new job, thanks to Clojurists Together I will be able to dedicate the coming three months to improving Kaocha, a next generation test runner for Clojure.

A number of projects applied for grants this quarter, some much more established than Kaocha. Clojurists Together has been asking people through their surveys if it would be cool to also fund “speculative” projects, and it seems people agreed.

I am extremely grateful for this opportunity. I hope to demonstrate in the coming months that Kaocha holds a lot of potential, and to deliver some of that potential in the form of a tool people love to use.

Two Years of Lambda Island, A Healthy Pace and Things to Come

It’s been just over two years since Lambda Island first launched, and just like last year I’d like to give you all an update about what’s been happening, where we are, and where things are going.

To recap: the first year was rough. I’d been self-employed for nearly a decade, but I’d always done stable contracting work, which provided a steady stream of income, and made it easy for me to unplug at the end of the day.

Lambda Island was, as the Dutch expression goes, “a different pair of sleeves”. I really underestimated what switching to a one-man product business in a niche market would mean, and within months I was struggling with symptoms of burnout, so most of year one was characterised by trying to keep things going and stay afloat financially, while looking after myself and trying to get back to a good place, physically and mentally.

D3 and ClojureScript

This is a guest post by Joanne Cheng (twitter), a freelance software engineer and visualization consultant based in Denver, Colorado. She has taught workshops and spoken at conferences about visualizing data with D3. Turns out ClojureScript and D3 are a great fit, in this post she’ll show you how to create your own visualization using the power of D3 and the elegance of ClojureScript.

I use D3.js for drawing custom data visualizations. I love using the library, but I wanted to try one of the several compile to JavaScript options, and I decided to look into ClojureScript. It ended up working out well for me, so I’m going to show you how I created a D3.js visualization using ClojureScript!

What we’re visualizing

Reloading Woes

Update: seems Stuart Sierra’s blog post has dropped off the internet. I’ve updated the link to refer to the Wayback Machine’s version instead.

Setting the Stage

When doing client work I put a lot of emphasis on tooling and workflow. By coaching people on their workflow, and by making sure the tooling is there to support it, a team can become many times more effective and productive.

The Bare Minimum, or Making Mayonnaise with Clojure

Making Mayonnaise

Imagine you have a grandfather who’s great at making mayonnaise. He’s been making mayonnaise since before the war, and the result is truly excellent. What’s more, he does this with a good old fashioned whisk. He’s kept his right arm in shape throughout decades just by beating those eggs and oil and vinegar.

Now he’s bought himself a handheld electric mixer after hearing his friend rave about hers, but after a few tries he gives up and goes back to his whisk. He says he just can’t get the same result. This seems slightly odd, so the next time you go over you ask him to show you how he uses the mixer.

Clojure Gotchas: "contains?" and Associative Collections

When learning a programming language we rarely read the reference documentation front to back. Instead someone might follow some tutorials, and look at sample code, until they’re confident enough to start a little project for practice.

From that point on the learning process is largely “just in time”, looking up exactly the things you need to perform the task at hand. As this goes on you might start to recognize some patterns, some internal logic that allows you to “intuit” how one part of the language works, based on experience with another part.

Developing this “intuition” — understanding this internal logic — is key to using a language effectively, but occasionally our intuition will be off. Some things are simply not obvious, unless someone has explained them to us. In this post I will look at something that frequently trips people up, and attempt to explain the underlying reasoning.

Dates in Clojure: Making Sense of the Mess

Update 2018-11-27: while most of this article is still relevant, I no longer recommend using JodaTime as the main date/time representation for new projects. Even existing projects that aren’t too invested in JodaTime/clj-time should consider migrating to java.time and clojure.java-time across the board.

Update 2 2019-05-29: Also check out the talk Cross Platform DateTime Awesomeness by Henry Widd, given at Clojure/north 2019

You can always count on human culture to make programming messy. To find out if a person is a programmer just have them say “encodings” or “timezones” and watch their face.

Clojure Gotchas: Surrogate Pairs

tl;dr: both Java and JavaScript have trouble dealing with unicode characters from Supplementary Planes, like emoji 😱💣.

Today I started working on the next feature of lambdaisland/uri, URI normalization. I worked test-first, you’ll get to see how that went in the next Lambda Island episode.

One of the design goals for this library is to have 100% parity between Clojure and ClojureScript. Learn once, use anywhere. The code is all written in .cljc files, so it can be treated as either Clojure or ClojureScript. Only where necessary am I using a small amount of reader conditionals.

Simple and Happy; is Clojure dying, and what has Ruby got to do with it?

The past week or so a lot of discussion and introspection has been happening in the Clojure community. Eric Normand responded to my one year Lambda Island post with some reflections on the size and growth of the community.

And then Zack Maril lamented on Twitter: “I’m calling it, clojure’s dying more than it is growing”. This sparked a mega-thread, which was still raging four days later. A parallel discussion thread formed on Reddit. Someone asked if their were any Clojure failure stories. They were pointed at this talk from RubyConf 2016, which seemed to hit a lot of people right in the feels, and sparked a subthread with a life of its own.

Meanwhile Ray, one of the hosts of the defn podcast reacted to the original tweet: “I’m calling it: Clojure is alive and well with excellent defaults for productive and sustainable software development.” This sparked another big thread.

Loading Clojure Libraries Directly From Github

Did you ever fix a bug in an open source library, and then had to wait until the maintainer released an updated version?

It’s happened to me many times, the latest one being Toucan. I had run into a limitation, and found out that there was already an open ticket. It wasn’t a big change so I decided to dive in and address it. Just a little yak shave so I could get on with my life.

Now this pull request needs to be reviewed, and merged, and eventually be released to Clojars, but ain’t nobody got time for that stuff. No sir-ee.

Lambda Island Turns One, The Story of a Rocky Ride

One year ago to date I launched Lambda Island, a service that offers high quality video tutorials on web development with Clojure and ClojureScript. It’s been quite a ride. In this post I want to look back at the past year, provide some insight into how this experience has been for me, and give you a glimpse of what the future has in store.

This story really starts in December 2015. After three years of doing contract work for Ticketsolve I decided it was time for a change. I have been self-employed for many years, but I knew that sooner or later I wanted to try my hand at selling a product, rather than selling my time.

In January and February I took some time for soul-searching, and recharging. I went to speak at RubyConf Australia, and got to hang out with some old friends around Australia and New Zealand. Once back in Berlin I got busy creating Lambda Island.

Writing Node.js scripts with ClojureScript

In the two most recent  Lambda Island episodes I covered in-depth how to create command line utilities based on Lumo, how to combine them with third party libraries, and how to deploy them to npmjs.com.

However there’s a different way to create tools with ClojureScript and distribute them through NPM, without relying on Lumo. In this blog post I want to quickly demostrate how to do just that.

To recap, Lumo is a ClojureScript environment based on Node.js, using bootstrapped (self-hosted) ClojureScript. This means the ClojureScript compiler, which is written in Clojure and runs on the JVM, is used to compile itself to JavaScript. This way the JVM is no longer needed, all you need is a JavaScript runtime to compile and run ClojureScript code, which in this case is provided by Node.js. On top of that Lumo uses nexe, so Lumo can be distributed as a single compact and fast executable binary.

Announcing lambdaisland/uri 1.0.0

I just released lambdaisland/uri, a pure Clojure/ClojureScript URI library. It is available on Github and Clojars.

This is a small piece of the code base that powers lambdaisland.com. It’s inspired by Ruby’s Addressable::URI, the most solid URI implementation I’ve seen to date, although it only offers a small part of the functionality that library offers.

It’s written in pure Clojure/ClojureScript, with only minimal use of .cljc reader conditionals to smooth over differences in regular expression syntax, and differences in core protocols. It does not rely on any URI functionality offered by the host, such as java.net.URL, so it’s usable across all current and future Clojure implementations (Clojure, ClojureScript, ClojureCLR).

re-frame Subscriptions Got Even Better

Up until recently, to use re-frame subscriptions in Reagent views, you had to use a form-2 component.

A form-2 component is a function that returns another function, which does the actual rendering of the component to hiccup. In contrast, a form-1 component renders the hiccup directly.

;; form-1
(defn todo-item [todo]
  [:div.view
   [todo-checkbox (:id todo) (:completed todo)]
   [:label {:unselectable "on"} title]
   [:button.destroy {:on-click #(dispatch [:todos/remove (:id todo)])}]])

;; form-2
(defn todo-item [todo]
  (fn [todo]
    [:div.view
     [todo-checkbox (:id todo) (:completed todo)]
     [:label {:unselectable "on"} title]
     [:button.destroy {:on-click #(dispatch [:todos/remove (:id todo)])}]]))

Game Development with Clojure/ClojureScript

This weekend it’s Ludum Dare again, the world’s longest running game jam. The idea is that, alone or with a team, you build a game in a weekend based on a certain theme.

We got a little team together here in Berlin, and so I’ve been reviewing what options there are for someone wanting to build a game in Clojure or Clojurescript.

The good news is there are plenty of options, as you’ll see from the list below. You can do desktop games, browser based games with canvas or webgl, and you can even create Unity 3D games, all from your comfortable Clojure parentheses.

Union Types with Clojure.Spec

Elm and other statically typed languages have a great feature called Union Types (also called Sum Types or Algebraic Data Types).

Here’s an example taken from Elm. Suppose your system used to represent users as integers, maybe just an auto-incrementing primary key, but then switched to UUIDs represented as strings.

To correctly model this situation, you need a way to create a type that can be either an integer or a string, that’s what union types give you.