Lambda Island

In this beginner friendly episode you’ll learn how to resolve a common type of Clojure error. It walks you through the process step by step, from analyzing the error, investigating the cause, coming up with a solution, up to finally reporting the issue on Github. In the process you’ll learn how to interpret stack traces, how to inspect the dependency tree with Leiningen, and how to influence Leiningen’s choice of version in case of a conflict.

Transcript

While working on some example code for an upcoming Lambda Island episode I ran into an issue, a problem with dependencies. It’s the kind of thing that easily happens, and I was quickly able to solve it and move on with what I was doing.

But then I thought: people run into this kind of things all the time, and unless you have some experience it’s not obvious at all what the problem is, or how you go about solving it. It can be a real energy drain.

So I decided to take a step, and documented exactly what I did to figure this out and solve it.

The project is called “Booklog”, and you can find the code on github. Grab the branch “dependency-error” to recreate the problem.

In this case the last thing I had done was adding Buddy to the project, a library that handles authentication and authorization. Buddy is split into several projects, here I’m using buddy-auth and buddy-hashers.

(defproject booklog "0.1.0-SNAPSHOT"
  :description "Keep track of the books you read, a sample project to demonstrate Buddy."
  :url "https://github.com/lambdaisland/booklog"

  :dependencies [[org.clojure/clojure "1.9.0-alpha14"]
                 [org.clojure/clojurescript "1.9.495" :scope "provided"]
                 [com.cognitect/transit-clj "0.8.297"]
                 [ring "1.5.1"]
                 [ring/ring-defaults "0.2.3"]
                 [bk/ring-gzip "0.2.1"]
                 [lambdaisland/ring.middleware.logger "0.5.1"]
                 [compojure "1.5.2"]
                 [environ "1.1.0"]
                 [com.stuartsierra/component "0.3.2"]
                 [org.danielsz/system "0.4.0"]
                 [org.clojure/tools.namespace "0.2.11"]
                 [reagent "0.6.1"]
                 [lambdaisland/garden-watcher "0.3.1"]
                 [hiccup "2.0.0-alpha1"]
                 [spicerack "0.1.2"]
                 [buddy/buddy-auth "1.4.1"]
                 [buddy/buddy-hashers "1.2.0"]]
  ,,,)

So far so good, I’ve started a new REPL, to make sure the newly added dependencies are downloaded and available on the classpath.

Now I try to require the buddy.auth.backends namespace, and boom, there’s the error.

user=> (require '[buddy.auth.backends :as backends])

CompilerException java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/FormatFeature, compiling:(cheshire/factory.clj:53:5)

In this case you only get a one line error message, in other circumstances you are more likely to see long stacktraces. There’s a lot of information in this one line, probably enough to point you in the right direction, but if you do want to see the full stacktrace then you can check the *e variable, which contains the last error that was raised.

user=> *e
#error {
 :cause "com.fasterxml.jackson.core.FormatFeature"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/FormatFeature, compiling:(cheshire/factory.clj:53:5)"
   :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 6926]}
  {:type java.lang.NoClassDefFoundError
   :message "com/fasterxml/jackson/core/FormatFeature"
   :at [java.lang.ClassLoader defineClass1 "ClassLoader.java" -2]}
  {:type java.lang.ClassNotFoundException
   :message "com.fasterxml.jackson.core.FormatFeature"
   :at [java.net.URLClassLoader findClass "URLClassLoader.java" 381]}]
 :trace
 [[java.net.URLClassLoader findClass "URLClassLoader.java" 381]
  [java.lang.ClassLoader loadClass "ClassLoader.java" 424]
  [sun.misc.Launcher$AppClassLoader loadClass "Launcher.java" 331]
  [java.lang.ClassLoader loadClass "ClassLoader.java" 357]
  [java.lang.ClassLoader defineClass1 "ClassLoader.java" -2]
  [java.lang.ClassLoader defineClass "ClassLoader.java" 763]
  ,,, ]}

In my case I was actually working from Emacs and CIDER. So I’m in this namespace, adding buddy.auth.backends to the list of required namespaces. And when reloading the file (in CIDER that’s C-c C-k), I’m greated with this *cider-error* buffer.

  Show: Clojure Java REPL Tooling Duplicates All  (97 frames hidden)

3. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling cheshire/factory.clj at (53:5)

2. Caused by java.lang.NoClassDefFoundError

1. Caused by java.lang.ClassNotFoundException
   com.fasterxml.jackson.core.FormatFeature

       URLClassLoader.java:  381  java.net.URLClassLoader/findClass
          ClassLoader.java:  424  java.lang.ClassLoader/loadClass
                  core.clj:    1  cheshire.core/eval63014

When something like this happens first take a step back and think about what you were doing. What changed? If you can go back to a previous version of your code that still works, then you can compare the two, this should give you some pretty good hints.

In my case I was trying to require Buddy. If the require isn’t there everything still works, but once I add it things blow up.

This isn’t always so obvious. Say you revisit a project after letting it rest for 6 months. You start by bumping the dependencies in your project.clj to their latest version, and then try to start a REPL. Boom!

In this case it’s not as obvious that requiring Buddy is causing problems, but on the other hand you do have a great data point: it has to be something with the dependencies.

 % lein repl
Exception in thread "main" java.lang.ExceptionInInitializerError
        at clojure.main.<clinit>(main.java:20)
Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/FormatFeature, compiling:(cheshire/factory.clj:53:5)
        at clojure.lang.Compiler.analyzeSeq(Compiler.java:6926)
        at clojure.lang.Compiler.analyze(Compiler.java:6701)
        at clojure.lang.Compiler.analyzeSeq(Compiler.java:6907)
        at clojure.lang.Compiler.analyze(Compiler.java:6701)
... (hundreds of lines of stack traces) ...

Half of bugfixing is just carefully reading error messages, so let’s have a better look at these stack traces. The two things to look for are: what went wrong, and where in the code did it happen.

Errors in Java, and hence also in Clojure, are called exceptions. They have types, like clojure.lang.Compiler$CompilerException or java.lang.NoClassDefFoundError.

Sometimes one exception triggers another one, which might trigger another one, and so on. You can see this in the stacktrace where it says “Caused by …”. In this case a clojure.lang.Compiler$CompilerException was caused by a java.lang.NoClassDefFoundError, which in turn was casued by a java.lang.ClassNotFoundException. In other words: the Clojure compiler had a problem, because it failed to find a definition for a given class, because trying to load the class failed.

So the root cause is that it didn’t find a class that was expected to be there, namely com.fasterxml.jackson.core.FormatFeature. That’s a pretty good indication that something is wrong with the dependencies.

The next question is: where did it happen. Now in a sense it happened in many places, every line in that stack trace is a call in the code that failed to complete. Notice that most of them are functions internal to Clojure. It’s not that likely that the bug is actually in Clojure, so you want to scan the stacktrace to find instances of library or application code.

3. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling cheshire/factory.clj at (53:5)

2. Caused by java.lang.NoClassDefFoundError
   com/fasterxml/jackson/core/FormatFeature

1. Caused by java.lang.ClassNotFoundException
   com.fasterxml.jackson.core.FormatFeature

       URLClassLoader.java:  381  java.net.URLClassLoader/findClass
          ClassLoader.java:  424  java.lang.ClassLoader/loadClass
                  core.clj:    1  cheshire.core/eval63014
                   jws.clj:   20  buddy.sign.jws/eval62241
                   jwt.clj:   15  buddy.sign.jwt/eval62233
                 token.clj:   15  buddy.auth.backends.token/eval62227
              backends.clj:   15  buddy.auth.backends/eval61761
                      REPL:    1  booklog.routes/eval61755

The interesting bits are usually the last place in library code that was called, and the last place in our own code that was called. Stacks grow upwards, the last thing added is at the top, and it’s no different with stack traces, so the first thing to look for is the top-most line that points at library code.

The first third party library in the stacktrace is Cheshire, a library for encoding JSON. It seems it was trying to invoke require.

                  core.clj: 5911  clojure.core/require
               RestFn.java:  457  clojure.lang.RestFn/invoke
                  core.clj:    1  cheshire.core/eval63014/loading--auto--

Further down you find buddy, which was also doing a require, so it seems buddy was trying to load cheshire.

                  core.clj: 5911  clojure.core/require
               RestFn.java:  551  clojure.lang.RestFn/invoke
                   jws.clj:   20  buddy.sign.jws/eval62241/loading--auto--

So all of this here is just Clojure requiring a single namespace. No wonder these stacktraces are so bloated. Hopefully tooling will get better at filtering these out, in the meanwhile we have to make the best of it.

                  core.clj:    1  cheshire.core/eval59202
                  core.clj:    1  cheshire.core/eval59202
             Compiler.java: 6978  clojure.lang.Compiler/eval
             Compiler.java: 6967  clojure.lang.Compiler/eval
             Compiler.java: 7430  clojure.lang.Compiler/load
                   RT.java:  374  clojure.lang.RT/loadResourceScript
                   RT.java:  365  clojure.lang.RT/loadResourceScript
                   RT.java:  455  clojure.lang.RT/load
                   RT.java:  421  clojure.lang.RT/load
                  core.clj: 6008  clojure.core/load/fn
                  core.clj: 6007  clojure.core/load
                  core.clj: 5991  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5812  clojure.core/load-one
                  core.clj: 5807  clojure.core/load-one
                  core.clj: 5852  clojure.core/load-lib/fn
                  core.clj: 5851  clojure.core/load-lib
                  core.clj: 5832  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  659  clojure.core/apply
                  core.clj: 5889  clojure.core/load-libs
                  core.clj: 5873  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  659  clojure.core/apply
                  core.clj: 5911  clojure.core/require
                  core.clj: 5911  clojure.core/require
               RestFn.java:  551  clojure.lang.RestFn/invoke

If you cut out all the noise, like I did here, then you can see that the problem ultimately originated in the first line of the booklog.application namespace, which is where the namespace declaration is that loads Buddy, which loads some internal namespaces, one of which depends on Cheshire. And apparently Cheshire needed this FormatFeature class but couldn’t find it.

                  core.clj:    1  cheshire.core/eval59202
                   jws.clj:   20  buddy.sign.jws/eval58429
                   jwt.clj:   15  buddy.sign.jwt/eval58421
                 token.clj:   15  buddy.auth.backends.token/eval58415
              backends.clj:   15  buddy.auth.backends/eval57949
           application.clj:    1  booklog.application/eval57943

It’s possible that Cheshire doesn’t probably declare its dependencies, and that that’s why the class isn’t there, but it’s unlikely. More likely is that the dependencies are in a bad state because of how library dependencies interact.

The best place to start looking for dependency problems is lein deps :tree. This shows you a detailed overview of all dependencies that are being loaded. Each library might pull in other libraries, which is visualized by this tree structure.

Note that a single library could be a dependency of many other libraries, but Leiningen will only show it once in the tree.

 [bk/ring-gzip "0.2.1"]
 [buddy/buddy-auth "1.4.1"]
   [buddy/buddy-sign "1.4.0"]
     [cheshire "5.7.0"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"]
       [tigris "0.1.1"]
   [funcool/cuerdas "2.0.2"]
 [com.cognitect/transit-clj "0.8.297"]
   [com.cognitect/transit-java "0.8.319"]
     [com.fasterxml.jackson.core/jackson-core "2.3.2"]
     [commons-codec "1.10"]
     [org.msgpack/msgpack "0.6.12"]
       [com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
       [org.javassist/javassist "3.18.1-GA"]

If you scroll back to the top you find a list of warnings about “possibly confusing dependencies”. Multiple libraries might have the same dependency, but they don’t necessarily depend on the same version. This means Leiningen needs to make a choice: does it pick the version required by library A, or the one asked for by library B?

There’s no perfect solution, Leiningen needs to guess. That’s why it’s warning you that maybe its choices aren’t ideal.

Possibly confusing dependencies found:
[org.clojure/tools.namespace "0.2.11"]
 overrides
[org.danielsz/system "0.4.0"] -> [org.clojure/tools.namespace "0.3.0-alpha3"]

Consider using these exclusions:
[org.danielsz/system "0.4.0" :exclusions [org.clojure/tools.namespace]]

[spicerack "0.1.2"] -> [com.google.guava/guava "19.0"]
 overrides
[org.clojure/clojurescript "1.9.495"] -> [com.google.javascript/closure-compiler-unshaded "v20170218"] -> [com.google.guava/guava "20.0"]

Consider using these exclusions:
[org.clojure/clojurescript "1.9.495" :exclusions [com.google.guava/guava]]

[com.cognitect/transit-clj "0.8.297"] -> [com.cognitect/transit-java "0.8.319"] -> [com.fasterxml.jackson.core/jackson-core "2.3.2"]
 overrides
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]
 and
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]
 and
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]

Consider using these exclusions:
[buddy/buddy-auth "1.4.1" :exclusions [com.fasterxml.jackson.core/jackson-core]]
[buddy/buddy-auth "1.4.1" :exclusions [com.fasterxml.jackson.core/jackson-core]]
[buddy/buddy-auth "1.4.1" :exclusions [com.fasterxml.jackson.core/jackson-core]]

The relevant section for us is this part. Both Transit and Buddy indirectly depend on com.fasterxml.jackson.core. Transit wants version 2.3.2, whereas Buddy (through Cheshire) asks for 2.8.6.

[com.cognitect/transit-clj "0.8.297"] -> [com.cognitect/transit-java "0.8.319"] -> [com.fasterxml.jackson.core/jackson-core "2.3.2"]
 overrides
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]
 and
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]
 and
[buddy/buddy-auth "1.4.1"] -> [buddy/buddy-sign "1.4.0"] -> [cheshire "5.7.0"] -> [com.fasterxml.jackson.core/jackson-core "2.8.6"]

Consider using these exclusions:
[buddy/buddy-auth "1.4.1" :exclusions [com.fasterxml.jackson.core/jackson-core]]

Now you can see how this plays out in the actual tree of dependencies. Underneath Transit you see jackson-core, being pulled in at version 2.3.2. Underneath Buddy jackson-core isn’t listed, even though it’s a dependency of Cheshire. So Transit is getting priority here, and overriding the version requested by Cheshire.

 [bk/ring-gzip "0.2.1"]
 [buddy/buddy-auth "1.4.1"]
   [buddy/buddy-sign "1.4.0"]
     [cheshire "5.7.0"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"]
       [tigris "0.1.1"]
   [funcool/cuerdas "2.0.2"]
 [com.cognitect/transit-clj "0.8.297"]
   [com.cognitect/transit-java "0.8.319"]
     [com.fasterxml.jackson.core/jackson-core "2.3.2"]
     [commons-codec "1.10"]
     [org.msgpack/msgpack "0.6.12"]
       [com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
       [org.javassist/javassist "3.18.1-GA"]

There are two ways to solve this. The easiest is to explicitly add the right version as an application dependency. Direct dependencies like these get preference, so this overrides both the Cheshire and the Transit dependency.

  :dependencies [,,,
                 [com.cognitect/transit-clj "0.8.297"]
                 [buddy/buddy-auth "1.4.1"]
                 [buddy/buddy-hashers "1.2.0"]
                 ,,,
                 [com.fasterxml.jackson.core/jackson-core "2.8.6"]]

You can verify this with lein deps :tree. The warnings for jackson.core are now gone, although there are still others that you could look into. Now instead of being shown as a dependency of Tranist, jackson.core is listed as a top level dependency, with the version we asked for, “2.8.6”.

 [buddy/buddy-auth "1.4.1"]
   [buddy/buddy-sign "1.4.0"]
     [cheshire "5.7.0"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"]
 ,,,
 [com.cognitect/transit-clj "0.8.297"]
   ,,,
 [com.fasterxml.jackson.core/jackson-core "2.8.6"]

You can think of this as the brute force approach, just hard code a version to make the problem go away, but it has its drawbacks.

Before the version of jackson-core was declared twice, once by Cheshire and once by Transit. now it’s declared three times. If you end up updating Transit or Buddy, or you add other libraries to your project, the chance of conflicts will only get bigger.

The better approach is to give Leiningen a gentle push in the right direction. Leiningen already showed you how to do that as part of the warnings. You can add an :exclusions vector to a dependency declaration. This tells Leiningen it should not bother with this transitive dependency. It’s as if Buddy or Cheshire never depended on this library at all.

This way the dependency through Transit will be favored, and it will use version 2.3.2, instead 2.8.6.

[buddy/buddy-auth "1.4.1" :exclusions [com.fasterxml.jackson.core/jackson-core]]

That’s not what you want though, version 2.3.2 is exactly the one that caused issues.

In this case Leiningen’s suggestion isn’t great, but with what you know now, you can also decide to do it the other way around, tell Leiningen not to bother with the Transit dependency, but to favor the one from Buddy/Cheshire instead.

Now 2.8.6 gets used. It’s the newer version of the two, but it still has the same major version number. This makes it highly likely that it’s backwards compatible with version 2.3.2. Many library authors promise to only introduce breaking changes in major releases, this is part of the versioning conventions known as “semantic versioning”.

[com.cognitect/transit-clj "0.8.297" :exclusions [com.fasterxml.jackson.core/jackson-core]]

You can verify once more with lein deps :tree that all looks good now, and indeed if you run the project this time the error is gone.

 [buddy/buddy-auth "1.4.1"]
   [buddy/buddy-sign "1.4.0"]
     [cheshire "5.7.0"]
       [com.fasterxml.jackson.core/jackson-core "2.8.6"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.8.6"]
       [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.8.6"]
       [tigris "0.1.1"]

You could leave it at that and move on. After all, your problem is solved, and you have work to do. It’s likely that someone else will run into this problem as well though, given that Transit and Cheshire are both popular libraries. It might even be you a few months from now.

The root cause here is that Transit hasn’t kept its dependencies up-to-date, so you can open an issue or pull request to have that fixed.

When engaging in open source like this there are three things to keep in mind: be friendly, be helpful, and be patient. More often then not the person on the other end is volunteering their free time because they enjoy doing open source. Don’t ruin that feeling!

Be helpful by providing as much relevant information as possible. In this case I can mention the old and new version, and the fact that it clashes with Cheshire, which shows that this is a real-world issue.

You could go one step further and run the project’s tests against the new version to see if anything breaks, after which you submit a pull request. This is assuming the project accepts pull requests, which is not the case for Cognitect projects like Transit.

Finally, be patient. In this case Alex Miller was very quick to update the version and do a new release, but sometimes this takes time. Resist the urge to “ping the maintainer”, or “bump the thread”. It’s unlikely they didn’t see it. The fact that the issue is documented is already helpful for others running into it.

Resolving Dependency Conflicts