Lambda Island

JVM based Clojure isn’t very suitable for command line scripting. On the command line you want quick feedback, which is at odds with the time Clojure needs to boot up, and just to run a single source file you need to jump through a few hoops.

Instead you can use Lumo, a ClojureScript runtime environment based on Node.js. In this episode you’ll learn how to write a script in Lumo, and how to make use Node’s built-in libraries, by building your own version of the Unix “tree” command. It’ll also be a handy refresher on recursion.

Transcript

In this episode I’ll show you how you can use Lumo, a ClojureScript runtime based on Node.js, to write command line utilities. While it’s possible to write scripts with JVM based Clojure, it’s not something people do often, mostly because Clojure needs quite a bit of time to boot up.

Lumo’s startup time is measured in milliseconds, rather than seconds, and with the npm ecosystem at your disposal it’s a compelling platform to build upon.

Make sure you have Node.js installed. I’m using the latest long-term-support release, 6.10.2, but older versions should work fine. ClojureScript is compatible with version 0.12 and up. I’ll link to Node’s installation instructions in the show notes.

$ node --version
v6.10.2
$ npm --version
4.5.0

You install Lumo through Node’s package manager, npm. Use the -g flag to install it globally, so you can access it from anywhere.

$ npm install -g lumo-cljs

Now you get the lumo command line utility at your disposal. Invoke it without any arguments to start a ClojureScript REPL.

$ lumo
Lumo 1.0-alpha
ClojureScript 1.9.293
 Exit: Control+D or :cljs/quit or exit
cljs.user=>

You can also pass lumo the name of a Script to run. Here I have a cljs file that prints out “Hello from Lumo!”. When given the name of the script on the command line, instead of starting a REPL, lumo will compile the script to JavaScript and and run it using Node.js.

;; lumo_test.cljs
(println "Hello from Lumo!")

$ lumo lumo_test.cljs
Hello from Lumo!

To do something a little more interesting than hello world I’m going to show you how to implement your own version of the UNIX command line tool tree, which prints out a visual tree of a part of the filesystem. I was thinking of naming it after a type of tree would be nice, maybe “chestnut”? Mmmmm, that would be confusing. How about “elm”? No that doesn’t work either. How about “birch”? Yeah I like the sound of that. Let’s call it birch.

$ tree .
.
├── bundled
│   └── lumo
│       ├── analyzer.cljs
│       ├── build
│       │   └── api.cljs
│       ├── classpath.cljs
│       ├── cljs_deps.cljs
│       ├── closure.cljs
│       ├── compiler.cljs
│       ├── io.cljs
│       ├── json.cljs
│       ├── util.clj
│       └── util.cljs
└── snapshot
    └── lumo
        ├── common.cljs
        ├── core.cljs
        ├── js_deps.cljs
        ├── pprint
        │   └── data.cljs
        ├── repl.clj
        ├── repl.cljs
        └── repl_resources.cljs

6 directories, 17 files

Later on I’ll also show you how to package this script up and publish it, this will require a few extra files, so let’s start by creating a project directory.

Call the project directory birch, and in there create another directory named bin, it’s a common name for directories that contain executables. Small tip, you can create nested directories in one go if you pass the -p or --parents flag to mkdir.

$ mkdir -p birch/bin
$ cd birch

Now inside bin/ make a new file called birch, it doesn’t have an extension since this will be a script that you execute directly from the terminal.

The first line of the file will be the “shebang” line. The name comes from a contraction of “hash-bang”. It tells the operating system which program to use to run your script, so you want to point this at the Lumo executable. The problem though is that Lumo doesn’t have a fixed, standardized location. In my case it’s somewhere under /home/arne/.nvm, but for you the exact location will likely be different.

That’s why there is a tool called env. It’s location is standardized, it’s always under /usr/bin. env will search the shell $PATH for an executable named lumo, and then call that executable, passing it the name of your script.

#!/usr/bin/env lumo

Because this is just a script it doesn’t really need a namespace declaration, instead require the stuff you need directly with require.

birch will take a single command line argument, which you can get a hold of through the *command-line-args* var in lumo.core.

#!/usr/bin/env lumo
(require '[lumo.core :refer [*command-line-args*]])

(println "-->" (first *command-line-args*))

You need to make the file executable now by flipping its executable bit with chmod. After that you should be able to run it.

$ chmod +x bin/birch
$ bin/birch hello
--> hello

Now you might have noticed that my editor isn’t showing any syntax highlighting, and my trusty Paredit is also messing. What gives?

Because the file doesn’t have an extension Emacs doesn’t recognize it as a ClojureScript file, and so it won’t switch to the right mode. You can give it a little push by adding a special comment at the top.

Now save the file (C-x C-s) and re-load it (C-x C-v RET), so Emacs picks up the special comment. For vim there’s a similar “modeline” syntax, although you have to enable support for it first in your .vimrc. For other editors like Atom or Sublime Text there are packages to add support for these special comments.

#!/usr/bin/env lumo
;; -*- mode: clojurescript -*-
;; vim: ft=clojure:
(require '[lumo.core :refer [*command-line-args*]])

(println "-->" (first *command-line-args*))

" .vimrc
:set modeline

That lumo.core namespace might make you think there’s a rich standard library bundled with Lumo. That’s not the case, but it also doesn’t really need one. There is already the stuff that’s bundled with ClojureScript itself, like clojure.string, clojure.set, zippers, pprint, and spec. There’s the Google Closure library which is brimming with goodies; and lumo is built on Node.js, so you have access to all the APIs that Node has to offer, things like buffers, streams, process control, and network and file system access.

If that’s still not enough then you can use ClojureScript libraries from Clojars, assuming they’re bootstrap compatible, as well as some 450’000 packages on npm. That’s a lot of packages.

Our tree clone needs to scan directories for files, so let’s load some of Node’s built-in packages to do that.

You use the require function from Node, through the js pseudo-namespace. This loads the fs module, a JS object containing functions.

I’m setting up a few aliases so these look like regular ClojureScript functions. readdir lists the entries in a directory, and stat let’s you find out a couple things, like whether it’s a file or a directory.

The regular versions of these functions all take callbacks, whereas the Sync variant simply returns a value, which is a bit easier to work with.

From the path module you’ll need join, to build up nested directory paths. This is better than just joining them with a slash, because the separator differs between operating systems. Finally add a directory? predicate function, this will come in handy soon enough.

#!/usr/bin/env lumo
;; -*- mode: clojurescript -*-
;; vim: ft=clojure:
(require '[lumo.core :refer [*command-line-args*]])

(def node-fs (js/require "fs"))
(def node-path (js/require "path"))

(def read-dir (.-readdirSync node-fs))
(def stat (.-statSync node-fs))
(def path-join (.-join node-path))

(defn directory? [f]
  (.isDirectory (stat f)))

This is the kind of result we’re looking for. If you pry that apart you will find three different “building blocks”, used to “draw” the tree. I’ll call these a T-branch, an I-branch, and an L-branch.

.
├── A
│   ├── B
│   ├── C
│   └── D
└── bin
    └── birch

I’ll define a few constants already so we can just piece these together later on. There are the three types of branches, and then also a spacer for when a part of the tree needs to be pushed to the right, as is the case with birch.

(def I-branch "│   ")
(def T-branch "├── ")
(def L-branch "└── ")
(def SPACER   "    ")

Implementing tree isn’t too hard once you realize you can split it up in two steps: first gather all data in a handy datastructure so it’s easy to work with, then write a render function that transforms that data into a visual tree.

Let’s start by stubbing out these functions, and adding a main function so you can test things out as we go along.

tree-entry takes a path and a file or directory name, and returns a datastructure representing the file system at that point.

render-tree turns that into the tree ASCII art we’re looking for. For reasons we’ll get into in a bit it will return a seq of strings, one for each line, rather than a single string.

Now add a main function, call it birch, which combines the others. It looks for the given directory name starting from the current directory, that’s why the path argument is just an empty string. The result of render-tree then needs to be joined together, and printed out to the console. Also make sure to require clojure.string.

Now at the top level you can call this function, passing it the first command line argument.

(require '[lumo.core :refer [*command-line-args*]]
         '[clojure.string :as str])

(defn tree-entry [path name]
  ,,,)

(defn render-tree [entry]
  ,,,)

(defn birch [dir]
  (->> (tree-entry "" dir)
       render-tree
       (str/join "\n")
       print))

(birch (first *command-line-args*))

This is what the data structure will look like: a simple map containing a :name and a list of :children. These children in turn have a :name and :children, and so forth, all the way down. So if you have a directory A containing the files B and C, and D, then it looks like this.

{:name "A"
 :children '({:name "B" :children nil}
             {:name "C" :children nil}
             {:name "D" :children nil})}

So tree-entry returns a map with :name and :children. The name is right there in the arguments, for the :children part you can write a separate function, call it child-entries. Only directories have children, so add a check for that. Let’s first pull this joined path into a let block, expand it, and then you can use that path to check if it’s a directory. This way child-entries can safely assume it always receives a valid directory path.

If you make render tree write out its input as a string with prn-str then you can already try out tree-entry.

(defn child-entries [path]
  )

(defn tree-entry [path name]
  (let [path (path-join path name)]
    {:name name
     :children (if (directory? path) (child-entries path))}))

(defn render-tree [entry]
  [(prn-str entry)])

You can call this from the console to see tree-entry in action.

$ bin/birch .
{:name ".", :children nil}

child-entries needs to return a seq of tree entry maps, one for each file or directory under path. So use read-dir to get all the file names, and turn them into tree entries.

Now child-entries calls tree-entry, and tree-entry calls child-entries, this is known as mutual recursion.

Functions in ClojureScript need to be defined or declared before they can be used. Since these two functions call each other, you need to declare one of them first before you can define the other, so let’s add a define statement for tree-entry.

(declare tree-entry)

(defn child-entries [path]
  (map #(tree-entry path %1) (read-dir path)))

(defn tree-entry [parent name]
  (let [path (path-join parent name)]
    {:name name
     :children (if (directory? path) (child-entries path))}))

Now that you have a representation of the tree in data, it’s time to render it. This will be the job of render-tree.

(defn render-tree [{:keys [name children}]
  )

render-tree accepts a tree-entry map, which falls apart into :name and :children, and will return a seq of strings, one for each line of output. Now if you run the tree command on a single file, or on an empty directory, then it simply prints out the name of the file, so start there by returning a one-element vector containing the file name.

(defn render-tree [{:keys [name children]}]
  [name])

The next step is to render the children, you can map over them, for each you pull out the child’s name, and then add a tree branch in front. The name of the parent needs to come before all this, so prepend it to the result of map with cons. Now you can already render a tree of depth 1.

(defn render-tree [{:keys [name children]}]
  (cons
   name
   (map (fn [child]
          (str L-branch (:name child)))
        children)))

To make this more general you can replace the child name’s with a rendering of the tree starting from that child. The result of render-tree is a list of strings, one for each line in the subtree, so use map to process all of them.

Now the outer map will result in a list of lists, instead it should be a flat list. By using mapcat instead of map all inner lists get concatenated together, so the result is a single list again.

(defn render-tree [{:keys [name children]}]
  (cons
   name
   (mapcat (fn [child]
             (map #(str L-branch %) (render-tree child)))
           children)))

This is already starting to look tree-like, but it doesn’t look quite right yet. Especially if you have trees with more nesting this becomes apparent.

A
└── B
└── C
└── D
└── └── E
└── └── F
└── └── G

I’ve only used L-branches so far, but sometimes it will have to be a T-branch, an I-branch, or a spacer instead.

Compare this again with the intended output. Directly in front of a child’s name there should be a T-branch, except for the last child, which gets an L-branch.

If a subtree consists of multiple lines, as is the case with “A” or “D”, then only the first line needs to get a branch, the remaining lines just need to be pushed to the right so they align.

When shifting a subtree to the right, such as this tree below “A”, it should use I-branches, except if it’s the last child of the parent, as is the case with “D”, then it uses spaces.

So pull that render-tree into a let block, so you can destructure it into a first line and a seq of the remaining lines. Map over everything but the first line, adding spaces in front, and then finally add a branch to the first line, adding it to the result again with cons.

(defn render-tree [{:keys [name children]}]
  (cons
   name
   (mapcat (fn [child]
             (let [[first & rest] (render-tree child)]
               (cons (str L-branch first)
                     (map #(str SPACER %) rest))))
           children)))

This looks better, but we’re treating all entries now as if they are the last child of their parent. Let’s fix that.

A
└── B
    └── BA
    └── BB
    └── BC
└── C
└── D
    └── DA
    └── DB
    └── DC

Add an index argument to the mapcat function by mapping over (range) as well as children. The last child is the one where its index is one fewer than the number of children.

The first line and the remaining lines need different prefixes, so let’s give those name. The last child’s subtree starts with an L-branch on its left, followed by spaces, whereas other children get prefix with a T-branch, followed by straight lines.

There, look at that, a birch tree. I hope it doesn’t give me allergies.

In the first part of this two-part series on Lumo you learned how to create a Lumo script, how to use Node’s built-in packages, and you got a good refresher on recursive functions. In part 2 I’ll show you how to share your script as an NPM packages, and how to make use of third party libraries from Clojars or NPM, so see you soon!

(defn render-tree [{:keys [name children]}]
  (cons
   name
   (mapcat (fn [child index]
             (let [subtree (render-tree child)
                   last? (= index (dec (count children)))
                   prefix-first (if last? L-branch T-branch)
                   prefix-rest  (if last? SPACER I-branch)]
               (cons (str prefix-first (first subtree))
                     (map #(str prefix-rest %) (next subtree)))))
           children
           (range))))

Building Command Line Applications with Lumo, part 1