Lambda Island

Now that the “birch” command is working, it’s time to release it to the world. In this episode you’ll learn how to turn your Lumo script into an NPM package. You’ll also learn how to use packages from NPM (cli-colors) and Clojure library jars (tools.cli) to make your script even more awesome.

Transcript

In episode 29 I showed you how to write a Lumo script, a re-implementation of the Unix tree command, using several of Node’s built-in packages.

In this episode you’ll learn how to publish your script as an NPM package, and how to make use of third party libraries from NPM and Clojars.

So far this script is just a single file, and it has no dependencies apart from what ClojureScript and Node provide out of the box.

Let’s try to make it more interesting, by using an npm package to add some color to the output.

First create a package.json inside the birch directory. There’s a handy npm init command to help you with that. The defaults are fine for now, just press enter a bunch of times, or you can add --yes to accept all defaults.

$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (birch)
version: (1.0.0)
description:
entry point: (index.js)
test command:
git repository:
keywords:
author:
license: (ISC)
About to write to /tmp/birch/package.json:

{
  "name": "birch",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}


Is this ok? (yes)

Now you can install the cli-color npm package. Add the --save flag, this way cli-color gets added as a dependency in package.json. This also adds a node_modules directory which contains the downloaded package. If you’re planning to make a git repo for this project then make sure to exclude this directory from version control.

$ npm install cli-color --save

{
  "name": "birch",
  ...
  "dependencies": {
    "cli-color": "^1.2.0"
  }
}

Now you can require cli-color in the script. The module contains a bunch of functions for different colors, we’ll just use the blue one to make directory names pop out a bit.

(def cli-color (js/require "cli-color"))
(def blue-text (.-blue cli-color))

Before changing the render function you first need to update tree-entry, to make sure each entry’s map contains all the information needed to render it. So inside the map add a :directory? key which will contain true or false.

(defn tree-entry [parent name]
  (let [path (path-join parent name)]
    {:name name
     :directory? (directory? path)
     :children (if (directory? path) (child-entries path))}))

Finally in render-tree use that information to change the color of the entry name. If it’s a directory then make the name blue, otherwise leave it as-is.

(defn render-tree [{:keys [name children directory?]}]
  (cons
   (if directory?
     (blue-text name)
     name)
   ,,,))

When executing this script Node will find the node_modules directory, since it’s in a directory above the one where the script is, and from there it will know how to load cli-color.

Now suppose you want to publish this package to npmjs.com, so you can help/impress your friends and colleagues. There are two different ways you could approach this. Before publishing the package you can compile the script to JavaScript, and then package up the JavaScript version instead of the ClojureScript one. This makes the package more lightweight: consumers don’t need lumo, and the code can be optimized using the Google Closure Compiler.

The downside is that you need to start messing with ClojureScript compiler configuration, which isn’t always straightforward. You have to introduce either boot or Leiningen, or rely on Lumo’s build API, which is currently still a bit immature. The final downside is that by introducing an extra compile step you can really slow down your development and deployment workflows.

So instead just go Lumo all the way. Since Lumo is itself an npm package you can add it as a dependency to make sure it’s available. That way you can bundle the uncompiled ClojureScript, and run it on the fly.

This does complicate matters slightly, because the lumo executable will be somewhere under node_modules, and even /usr/bin/env won’t be able to find it, so you’ll need a small wrapper script to find and launch lumo.

First lets restructure the project a bit. Create a src which will contain the source files.

$ mkdir -p src/birch

This time follow the conventions for namespaces. The script as you have it now becomes the birch.core namespace which goes into src/birch/core.cljs.

$ mv bin/birch src/birch/core.cljs

Now get rid of the shebang line and the modeline comment, and instead give it a proper namespace form. You can get rid of the lumo.core requirement. The command line arguments will come from somewhere else now. Note that in an ns form you don’t need a quote in front of the vectors.

(ns birch.core
  (:require [clojure.string :as str]))

Rename the birch function to -main, and delete the last line that calls it. We’ll have lumo call this main function directly instead.

(defn -main [dir]
  (->> (tree-entry "" dir)
       render-tree
       (str/join "\n")
       print))

So far we’ve been calling lumo with the name of the script as a single argument, but you can also invoke it in a different way. With -c src you tell it that src is part of the “classpath”, in other words, that lumo should look for namespaces inside that directory.

With the -m flag you can give it a “main” namespace. Lumo will load this namespace, look for a -main function, and call it, passing any remaining command line arguments.

lumo -c src -m birch.core ..

It should still be possible to launch this as birch, so we’ll have to add a wrapper script. Create another bin/birch file, but this time as a regular old shell script, so give it a shebang pointing at /bin/bash. Put the lumo invocation in there, followed by "$@", in double quotes. This special variable will contain all arguments to the script, so this way you can pass them all on to the actual ClojureScript code.

You might have seen $*, or $@ without the quotes somewhere. All of these are subtly broken when it comes to dealing with arguments that contain spaces.

#!/bin/bash

lumo -c src -m birch.core "$@"

Make the new script executable, and now it should work like before.

$ chmod +x bin/birch

Next you need to tell npm about your script, by adding it to the scripts section in package.json. This way if someone install your package with the -g flag, npm will install your script in a location where the shell can find it.

{
  ...
  "scripts": {
    "birch": "bin/birch"
  },
  ...
}

Also add lumo as a dependency, so when people install your package, lumo will be installed with it. Careful of the commas when editing JSON! Clojure programmers are so spoiled in this regard.

{
  ...
  "dependencies": {
    "cli-color": "^1.2.0",
    "lumo-cljs": "1.4.1"
  }
}

You’re almost there, there’s just one more problem. You installed lumo globally, so on your computer the wrapper script will always be able to find it, but if someone else installs your package then lumo will be installed under node_modules, and the script won’t work. You’ll have to add some extra logic to always find the correct lumo executable.

You could ask npm for the location of packages that were installed globally with -g, and resolve the path from there.

Inside the birch package there’s a node_modules, which contains lumo-cljs, and in there under bin is the lumo executable.

$ npm root -g
/home/arne/.nvm/versions/node/v6.10.2/lib/node_modules
$ ls `npm root -g`/birch/node_modules/lumo-cljs
bin  scripts  package.json  README.md
$ ls `npm root -g`/birch/node_modules/lumo-cljs/bin
lumo
$ `npm root -g`/birch/node_modules/lumo-cljs/bin/lumo
Lumo 1.4.1
ClojureScript 1.9.521
Node.js v7.9.0
 Docs: (doc function-name-here)
 Exit: Control+D or :cljs/quit or exit

cljs.user=>

This works pretty well, but it bothers me that this assumes birch is installed globally. Maybe that’s not the case. Maybe you’re just running it from inside a cloned repository, or installing it as a dependency of another package. It would be better if birch could find the location of lumo relative to its own location.

For this the bash script needs to figure out its own location on the file system, which is a lot harder than it should be. There’s a StackOverflow question where people have discussed the merits of various solutions at length, I’ll link to it in the show notes. I’m going with this particular oneliner, which is the simplest solution that actually works.

It does not resolve symlinks, which in this case doesn’t matter, and it breaks when the final path ends on a newline, which since this script is in a bin directory should not be the case.

With that you get the location of the bin directory, and from there you can traverse into Birch’s own node_modules directory to find the lumo executable.

#!/bin/bash

export BIRCH_BIN="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

$BIRCH_BIN/../node_modules/lumo-cljs/bin/lumo -c src -m birch.core "$@"

Oh and one more thing, add this $BIRCH_BIN in quotes, or it won’t work if there are spaces in the path. Oh the joys of shell scripting.

"$BIRCH_BIN"/../node_modules/lumo-cljs/bin/lumo -c src -m birch.core "$@"

Now your package is ready to be deployed. You’ll have to give it a slightly different name though, since “birch”, just like most words in the dictionary, is already taken. I’ll call mine lambdaisland-birch.

{
  "name": "lambdaisland-birch",
  ...
}

If you haven’t published to npm before then first register an account with npm adduser. You need to pick a username and password, and enter your email address. After that you are logged in, and you can run npm publish in your package directory, and start sending links to your friends!

$ npm adduser
Username: lambdaisland
Password:
Email: (this IS public) arne@lambdaisland.com
Logged in as lambdaisland on https://registry.npmjs.org/.
$ npm publish
+ lambdaisland-birch@1.0.0

So far so good for JavaScript dependencies, but what if you want to use ClojureScript libraries that are hosted on Clojars? This is actually a bit more involved. First you need to make sure that all the libraries you’re using are bootstrap-friendly. Lumo is based on so called “bootstrapped” or “self-hosted” ClojureScript, and not all libraries can work with that.

Assuming the libraries you’re using are compatible, you then need to get them, and all of their dependencies, on Lumo’s classpath, one way or another.

The clojure.tools.cli library has no further dependencies, so I’ll use that one to demonstrate how to do this manually. If you have many more ClojureScript dependencies you should consider if you’re not better off compiling your script with boot or leiningen instead.

On the tools.cli Github page, follow the link to “all releases”, and download the jar for the latest version.

In your project make a lib directory, and put the downloaded jar file in there.

$ mkdir lib
$ mv ~/Downloads/tools.cli-0.3.5.jar lib

Now you need to make sure that jar file is on Lumo’s classpath, so it will look inside the jar for namespaces. Let’s refactor this a wee bit by storing $BIRCH_HOME instead of $BIRCH_BIN. Entries on the classpath are separated by colons, so after src add the location of the tools.cli jar.

#!/usr/bin/bash

export BIRCH_HOME="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/.."

"$BIRCH_HOME"/node_modules/lumo-cljs/bin/lumo \
  -c src:"$BIRCH_HOME"/lib/tools.cli-0.3.5.jar \
  -m birch.core "$@"

Finally you’re ready to use tools.cli in your code. Require the cljs.tools.cli namespace, and refer to the parse-opts function, for intelligent parsing of command line options.

(ns birch.core
  (:require [clojure.string :as str]
            [cljs.tools.cli :refer [parse-opts]]))

Define the options your program understands by defining a vector of vectors, containing the short and long command line flag, and a description. We’ll make colors in the output optional, since not all terminals support them.

(def cli-opts [["-c" "--color" "Colorize the output"]])

Now the -main function can receive any number of arguments, so change its signature, and call parse-opts in a let block, passing it the incoming arguments, and the vector of recognized options.

The result is a map with :options and :arguments. There should still be just one argument, the directory to turn into a tree.

You can look up :color now in the options map, this will be either nil or true. Pass it as an extra argument to render-tree.

(defn -main [& args]
  (let [{:keys [options arguments]} (parse-opts args cli-opts)]
    (->> (tree-entry "" (first arguments))
         (render-tree (:color options))
         (str/join "\n")
         print)))

Finally change render-tree to accept that extra argument, and only change the color if it is true. Also make sure to pass it on to the recursive calls to render-tree.

(defn render-tree [colorize? {:keys [name children directory?]}]
  (cons
   (if (and colorize? directory?)
     (blue-text name)
     name)
   (mapcat (fn [child index]
             (let [subtree (render-tree colorize? child)
                   ,,,]
               ,,,))
           children
           (range))))

That’s it! I hope this episode has inspired you to take Clojure to the command line. For all its flaws the venerable Unix shell has still persisted throughout the ages, for certains tasks it’s terse textual interface is hard to beat.

Show Notes

browse source code

You can browse the code the way it looks at the end of this episode in the ep30-end branch.

Links

Unravel and Mach are two Lumo CLI apps distributed as NPM packages. You can check out their source for inspiration.
Abio and cljs-node-io are both libraries that provide ClojureScript wrappers for built-in node API’s. If you’re planning to do non-trivial IO or host access you should check these out.
Calvin provides a build tool that uses Lumo to compile your ClojureScript project. Even though young and immature it’s definitely a tool to watch. Interestingly it’s not packaged as a Lumo script, but instead uses lein-cljsbuild to compile to JavaScript, after which it’s packaged as a regular Node package.
StackOverflow on how to get the location of a bash script.

Building Command Line Applications with Lumo, part 2