Using JavaScript libraries in ClojureScript

One of the benefits of ClojureScript is that it lets you leverage the great JavaScript ecosystem. Using arbitrary JS libraries isn’t always trivial though, and a lot of head scratching can ensue. In this episode you’ll learn the exact steps from here to success, and you’ll gain insight into ClojureScript’s build process, so you can tackle any challenge that comes your way.

Transcript

ClojureScript is great, but it gets even better when you combine it with the vast amount of JavaScript libraries out there. There’s more than one way to do that, and because of the way ClojureScript compilation works there are a few snags to watch out for.

You’re going to see quite some JavaScript interop forms in this episode. If you’re still a bit fuzzy on those then I recommend first watching the episode on JavaScript interop.

I’ve set up a minimal ClojureScript project to demonstrate how to do things. It’s on Github under lambdaisland/thirdpartyjs. I’ll use a separate branch for each thing I’m demonstrating, so you can easily check them out.

The project is fairly typical. I’ve included ClojureScript and the lein-cljsbuild plugin, and configured two ClojureScript builds, one with and one without optimizations. I’ve added a main namespace, with a bit of code just to see that it’s working, and added an index.html that loads the compiled CloureScript. So far so good.

(defproject thirdpartyjs "0.1.0-SNAPSHOT"
  :license {:name "Mozilla Public License 2.0" :url "https://www.mozilla.org/en-US/MPL/2.0/"}

  :dependencies [[org.clojure/clojure "1.8.0"]
                 [org.clojure/clojurescript "1.9.229"]]

  :source-paths ["src/clj"]

  :plugins [[lein-cljsbuild "1.1.4"]]

  :clean-targets ^{:protect false} [:target-path :compile-path "resources/public/js/compiled"]

  :cljsbuild {:builds
              {:dev {:source-paths ["src/cljs"]
                     :compiler {:main thirdpartyjs.core
                                :optimizations :none
                                :output-to "resources/public/js/compiled/thirdpartyjs.js"
                                :output-dir "resources/public/js/compiled/dev"
                                :asset-path "js/compiled/dev"
                                :source-map-timestamp true}}

               :prod {:source-paths ["src/cljs"]
                      :jar true
                      :compiler {:main thirdpartyjs.core
                                 :optimizations :advanced
                                 :output-to "resources/public/js/compiled/thirdpartyjs.js"
                                 :source-map "resources/public/js/compiled/thirdpartyjs.js.map"
                                 :output-dir "resources/public/js/compiled/prod"
                                 :source-map-timestamp true
                                 :pretty-print false}}}})

The advice I want to give you first is to use CLJSJS whenever possible. CLJSJS is a community effort to package up JavaScript libraries so they’re easy to consume from ClojureScript. A few hundred libraries are already packaged up, and more are being added all the time.

This only takes three steps. First, find the library on cljsjs.github.io, and copy the dependency vector into your Leingen or Boot configuration.

:dependencies [,,,
               [cljsjs/d3 "4.2.2-0"]]

Now look at the usage instructions to find the provided name of the package, also known as its “synthetic namespace”. By requiring this in our namespace we make sure the package gets loaded.

(ns thirdpartyjs.core
  (:require [cljsjs.d3]))

Finally use whatever JavaScript globals the library creates. This depends on the project, in this case there’s a global d3 object that we can access through the js pseudo-namespace. Here’s some example code that draws the outline of an ice berg. Notice how I’m accessing d3 through js/d3, and not through the cljsjs.d3 namespace that we required. The cljsjs.d3 namespace is called a “synthetic namespace”. It doesn’t really exist. It’s only used to signal that this code uses D3, so it will be loaded.

(def values #js [0 3 2 5 4 7 8 6 2 6 5 1 0])

(def width 960)
(def height 500)

(let [svg (.. js/d3
              (select "#app")
              (append "svg")
              (attr "width" width)
              (attr "height" height))
      x-scale (.. js/d3
                  scaleLinear
                  (range #js [0 width])
                  (domain #js [0 12]))
      y-scale (.. js/d3
                  scaleLinear
                  (range #js [height 0])
                  (domain #js [0 10]))
      line (.. js/d3
               line
               (x #(x-scale %2))
               (y y-scale))]

  (.. svg
      (append "path")
      (datum (clj->js values))
      (attr "fill" "none")
      (attr "stroke" "#3355ee")
      (attr "stroke-width" "5px")
      (attr "d" line)))

Now do a dev build, that’s lein cljsbuild once dev. This goes relatively fast. Open index.html in your browser, and you should see this working.

lein cljsbuild once dev
firefox resources/public/index.html

You can do the same for the minimized build. This time run lein cljsbuild once prod, and reload your browser. Look at that!

lein cljsbuild once prod

Now suppose CLJSJS isn’t an option, what’s the simplest thing that could possibly work? Well let’s just load D3 as a separate script tag, by adding a script tag to index.html. This should make the d3 global available like before, so our code should work unchanged. The only thing that changes is the namespace declaration, that synthetic require can go, we’re taking care of loading D3 ourselves now.

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.2.3/d3.min.js" type="text/javascript"></script>

Now do a development build and reload your browser again. Seems this “simplest thing that could possibly work” actually worked, it almost seems too easy!

lein clean ; lein cljsbuild once dev

Let’s try a production build instead. Reload the browser, oops, looks like we broke something. Peek under the cover and you’ll see an error message like this one

TypeError: d3.select(...).append(...).Va is not a function

d3.select.append, that still looks good, but then it tries to call a Va method. That’s not in our code. Where does this gibberish come from?

Open the compiled file, thirdpartyjs.js, and scroll to the bottom. You should recognize bits of our code, but it’s all been jumbled and obfuscated.

var Pe=d3.select("#app").append("svg").Va("width",960).Va("height",500),Qe=d3.Vb().Tb([0,960]).domain([0,12]),Re=d3.Vb().Tb([500,0]).domain([0,10]),Se=d3.fc().x(function(a,b){return function(a,d){return b.b?b.b(d):b.call(null,d)}}(Pe,Qe,Re)).y(Re);Pe.append("path").dc(Oe([0,3,2,5,4,7,8,6,2,6,5,1,0])).Va("fill","none").Va("stroke","#3355ee").Va("stroke-width","5px").Va("d",Se);

So how did this happen? The ClojureScript compiler is built on top of the Google Closure Compiler. That’s Closure with an s, not with a j. One of the main goals of the Closure Compiler is to reduce loading time, by packing all the JavaScript together in a single file, and then making it as small as possible.

It has a few ways of doing this, including renaming functions to something shorter. In this case it has renamed every occurence of attr with Va. When it does that all over the code base then that’s not a problem, but because we loaded D3 separately it hasn’t been processed, and so we’re calling a function that doesn’t exist, leading the browser to divulge this universally acknowledged, inaliable truth: “Undefined is not a function”.

One way around this is to simply not use advanced optimizations. There are four optimization levels to choose from, and only one of them causes this problem. If squeezing the last byte out isn’t vital to what you are doing, then maybe :whitespace or :simple optimization is good enough.

Those advanced optimizations are nice though, can we have our cake and optimize it too?

Google Closure has its own module and dependency tracking system, a bit like CommonJS or AMD. If D3 was a library written for Google Closure then we could just add it to our build, and it would be optimized with the rest of the code. That’s not the case, but let’s have a look anyway what such a module would look like.

Here’s a “padding” module that implements a “leftPad” and a “rightPad” function. The goog.provide clause declares the namespace that this module defines. goog.provide will create a JavaScript object with that name, so we can assign properties onto it. Any function we want to make publicly available, like leftPad or rightPad, must be added to this object. makePadding is just an internal helper, so we don’t need to add it.

With goog.require we can include another library, in this case goog.string, since we’re using the goog.string.repeat function.

goog.provide('thirdpartyjs.padding');
goog.require('goog.string');

function makePadding(str, len) {
  var padLen = len - str.length;
  if (padLen <= 0) {
    return str;
  }
  return goog.string.repeat(' ', padLen);
}

thirdpartyjs.padding.leftPad = function (str, len) {
  return makePadding(str, len) + str;
}

thirdpartyjs.padding.rightPad = function (str, len) {
  return str + makePadding(str, len);
}

Notice the name I used in goog.provide, it looks just like a ClojureScript namespace. That’s because for our purposes it is. To save this file, use the same naming conventions as you would in ClojureScript, except that now instead “cljs” it has a “js” extension. So namespace segments become directories, and the last segment becomes the file name.

.
├── clj
│   └── thirdpartyjs
│       └── core.clj
└── cljs
    └── thirdpartyjs
        ├── core.cljs
        └── padding.js

Now from ClojureScript, require it just like a regular namespace. :refer won’t work in this case, but :as does, so we’ll give it a more convenient alias.

(ns thirdpartyjs.core
  (:require [thirdpartyjs.padding :as padding]))

(enable-console-print!)

(prn (padding/leftPad "Hello!" 10))

This time it works both with the development and with the production build. Let’s peek under the covers and see how they differ.

The HTML we send to the browser is pretty minimal, there’s just one JS file that we load.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link href="css/style.css" rel="stylesheet" type="text/css">
  </head>
  <body>
    <div id="app"></div>

    <script src="js/compiled/thirdpartyjs.js" type="text/javascript"></script>
  </body>
</html>

When looking at the inspector there’s a lot more going on though. We started with a single Script tag, but now there are at least a dozen. Where do all those extra script tags come from?

<body>
    <div id="app"></div>

    <script src="js/compiled/thirdpartyjs.js" type="text/javascript"> </script>

    <script src="js/compiled/dev/goog/base.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/deps.js"></script>
    <script src="js/compiled/dev/cljs_deps.js"></script>
    <script>if (typeof goog == "undefined") console.warn("ClojureScript could not load :main, did you forget to specify :asset-path?");</script>
    <script>goog.require("thirdpartyjs.core");</script>

    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/string/string.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/object/object.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/math/integer.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/string/stringbuffer.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/debug/error.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/dom/nodetype.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/asserts/asserts.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/array/array.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/reflect/reflect.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/math/long.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/../cljs/core.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/../thirdpartyjs/padding.js"></script>
    <script type="text/javascript" src="file:///home/arne/thirdpartyjs/resources/public/js/compiled/dev/goog/../thirdpartyjs/core.js"></script>
</body>

If you look at thirdpartyjs.js, you’ll see that it doesn’t contain any application code, all it does is inject some tags to do the actual loading. Three of these are important for us.

base.js contains supporting code for Google Closure’s module system. It’s here that goog.provide and goog.require are defined, so code can be loaded dynamically, on the fly.

cljs_deps.js is a generated file containing a list of calls to goog.addDependency. Together they form the dependency graph for our application, specifying the dependencies between namespaces, and the files they are defined in.

Once that is done a call to goog.require will load the thirdpartyjs.core namespace. All of the necessary dependencies will now be loaded first. core depends on padding, and also uses ClojureScript built-in functions, defined in cljs.core, which in turn depends on stuff from the Google Closure Library, so that’s what these are.

I hope you managed to follow that :) The reason I’m showing you all this is to drive home the point that, when optimizations are disabled, Google Closure forms a dynamic module system, where each namespace forms its own file, and namespaces can be loaded at runtime.

document.write('<script src="js/compiled/dev/goog/base.js"></script>');
document.write('<script src="js/compiled/dev/cljs_deps.js"></script>');
document.write('<script>goog.require("thirdpartyjs.core");</script>');

When we go to the other end of the optimization spectrum, skipping :whitespace or :simple optimizations, and moving straight to :advanced we get a very different picture.

I now did a production build, so with advanced optimizations. It’s still using the same HTML, containing a single <script> tag. If you look at the inspector, it’s no longer loading any extra files, so it must be that our whole application has now been packed into this one file. It’s highly optimized code, not meant for human consumption, but with some effort you can still find leftPad in there, except now it looks like this:

Several things are remarkable about this bit of generated code. The makePadding function is gone, it has been inlined into leftPad. Namespaces are gone, instead every individual function has gotten a unique short name, flattening the hierarchy.

And what happened to rightPad? Well, it’s simply gone. Google Closure figured out we weren’t using it, and simply ripped it out. The same happens for ClojureScript core or library functions: if you don’t use it, you don’t pay for it. That’s one of the superpowers of ClojureScript.

The takeaway from this is that what was essentially a system for loading dependencies at run-time, now becomes a tool for analyzing and optimizing code. And, given the magic it does, it’s also easy to see how things can go wrong.

function Oe(a,b){var c=b-a.length;return(0>=c?a:da(" ",c))+a}

Back to D3, we understand now what went wrong. We had D3 outside of our build, and so the compiler couldn’t take into account the functions it’s defining.

We need to tell the compiler that it should leave certain function names alone, because they’re defined elsewhere. This is done through an “externs” file, a JavaScript file that doesn’t contain any code, but instead contains declarations for all the identifiers that should be left alone. Here’s a small externs file with declarations for all the bits of D3 that we’re actually using.

var d3 = {};
d3.select = function() {};
d3.selection = function() {};
d3.selection.prototype.append = function() {};
d3.selection.prototype.attr = function() {};
d3.selection.prototype.datum = function() {};
d3.scaleLinear = function() {};
d3.scaleLinear.prototype.range = function() {};
d3.scaleLinear.prototype.domain = function() {};
d3.line = function() {};
d3.line.prototype.x = function() {};
d3.line.prototype.y = function() {};

Save that to d3-externs.js and add it to the ClojureScript compiler configuration under :externs

Now we can do an advanced compilation without breaking the app.

:externs ["d3-externs.js"]

In this case we’re only using a handful of functions, but for a real world application you really don’t want to be writing these externs manually. Luckily there are tools to generate them for you, like this online externs generator by Michael McLellan. Just put in a link to the D3 source file, choose which top level variable you’ll be referencing, in our case that’s d3, and click on “Extern!”. Copy the result to your externs file and you’re done.

Or at least, mostly done. Occasionally it might not detect all the necessary externs, and you need to add a few yourself. Doesn’t sound like fun? That’s why we have CLJSJS. CLJSJS packages already have the necessary externs included, so you don’t have to worry about them. If you have choice in the matter, than always use CLJSJS, but I already told you that.

So far we’re still loading D3 separately by adding an extra script tag. That means doing an extra network request, and if we get more dependencies that is going to get costly. It would be better if we could compile D3 together with the rest of our code into a single artifact.

Let’s first pull D3 down so we can use it locally.

curl https://cdnjs.cloudflare.com/ajax/libs/d3/4.2.3/d3.min.js > d3.js

To get a JavaScript file into the build that isn’t a Google Closure module, you use the :foreign-libs option in the compiler configuration. :foreign-libs takes a vector of maps, one map for each file you want to include. First point it at the right file, and then use :provides to give it a “synthetic namespace”. Provides takes a vector, but normally you just put a single string in there.

All this will accomplish is including the D3 source as-is in the final output. It just concatenates the files together. It won’t analyze it, or do smart optimizations, so we still need the externs file to make sure our own code stays compatible after optimizations.

:foreign-libs [{:file "d3.js"
                :provides ["js.d3"]}]
:externs ["d3-externs.js"]

Now just like with CLJSJS we need to require this namespace, or it won’t be included in the build. After that you can reference D3 like any JavaScript global.

This final approach is in most cases the recommended one. It doesn’t allow Google Closure to optimize the third party library, but for many libraries that will probably never be the case, given there are some very specific requirements in how the code is written.

(ns thirdpartyjs.core
  (:require [js.d3]))

What I haven’t told you yet, is that Google Closure Compiler does ship with the ability to load other module formats, like CommonJS, AMD, or ES6. It’s still a bit of an experimental feature, I definitely had some mixed results trying this out, but I’ll show you anyway, because it’s a pretty cool feature, and because it lets you imagine what might be possible in the future.

To show you how that works I’ve changed padding.js to use ES6 import and export statements.

Import is used to pull in the goog.string.repeat function. The goog: prefix is needed here to show that this is a Google Closure namespace we’re loading from, and not another ES6 file. You can also use it to import functions defined in ClojureScript namespaces, which I think is pretty wild.

Finally it exports leftPad and rightPad, the two public functions of this padding namespace.

import {repeat} from 'goog:goog.string';

function makePadding(str, len) {
  var padLen = len - str.length;
  if (padLen <= 0) {
    return str;
  }
  return repeat(' ', padLen);
}

function leftPad (str, len) {
  return makePadding(str, len) + str;
}

function rightPad (str, len) {
  return str + makePadding(str, len);
}

export { leftPad, rightPad };

To include an ES6 module also add it to :foreign-libs. We specify the filename and the provided namespace, like before, but now add an extra option :module-type.

This makes a pretty big difference, where before the foreign library would naively be concatenated with the rest, now it becomes a first class part of the build, taking part in dependency resolution, and getting the full load of optimizations.

:foreign-libs [{:file "padding.js"
                :provides ["thirdpartyjs.padding"]
                :module-type :es6}]

The application code is unchanged from before, we require the padding namespace, and call leftPad. Pretty cool!

(ns thirdpartyjs.core
  (:require [thirdpartyjs.padding :as padding]))

(enable-console-print!)

(prn (padding/leftPad "Hello!" 10))

That’s it, you now know all there is to know about using JavaScript libraries from ClojureScript. Most of the time CLJSJS will be all you need, but sometimes it’s not. With this stuff under your belt you should have all the tools available to figure whatever challenge comes your way.

Show Notes

browse source code

Each approach is demonstrated in a separate branch, the README in the repo contains an overview.

	Module format	Max optimization level	3rd party code
script	any	:simple	side-loaded
script + externs	any	:advanced	side-loaded
Closure module	Google Closure	:advanced	optimized + bundled
foreign-libs	any	:simple	bundled
foreign-libs + externs	any	:advanced	bundled
module-type	ES6, AMD, CommonJS, UMD*	:advanced	optimized + bundled

Compiler options

The ClojureScript compiler options relevant for this episode are

:optimizations :none|:whitespace|:simple|:advanced
:foreign-libs {:file "path/to/file.js"
               :file-min "path/to/file-minified.js"
               :provides ["synthetic.namespace"]
               :requires ["other.name" "spac.es"]
               :module-type :es6|:amd|:commonjs}
:externs ["path/to/externs.js"]

`:optimizations`

:none
Don’t invoke the Google Closure Compiler, load Google Closure modules with base.js
:whitespace
Concatenate sources into a single artifact
:simple
Do optimizations that are guaranteed to be safe, like local variable renaming
:advanced
Perform global analyzing and renaming, dead code elimination, inlining, etc.

`:file-min`

Not discussed in this episode, it is possible to point to both the regular and minified version of a library. In development mode (:optimizations :none) the regular version is used for easy debugging, with optimizations enabled the minified version is used. This is useful since when using :foreign-libs the 3rd party library is simply concatenated into the final build artifact, it is not analyzed or optimized.

Links and Further reading

JavaScript Externs Generator

ClojureScript Wiki:

Closure Compiler Wiki:

JS Modules