2 Clojure REPLs

Clojure's Built-in REPL

Starting the REPL

A natural starting point is the REPL that comes with Clojure itself. This is not the REPL you get with lein repl though! To see Clojure's built-in REPL all we need is the single JAR file which contains Clojure. If you've done any Clojure before you probably already have it on your system, you can find it with this command

find ~/.m2 -name clojure-1.*.jar

If that doesn't yield anything, then download Clojure from the official download page, extract the zip file, and you should find the JAR in there.

Now run this command in your terminal, and you should find yourself looking at a beautiful Clojure REPL.

java -jar /path/to/clojure-x.y.z.jar

For example

$ java -jar clojure-1.8.0.jar
Clojure 1.8.0
user=> (+ 1 1)
2
user=>

If you're curious you can find the implementation in clojure.main/repl.

REPL Special Features

When this REPL starts we are in the user namespace. Besides the usual stuff from clojure.core there are a few extra functions available to you in this namespace. These are all very handy to have around during development, so make sure you acquaint yourself with them.

From clojure.repl

source
Show the source code of a function. (source clojure.main/repl).
apropos
Given a string or regular expression, find all functions that match. (apropos "some")
doc
Show the documentation for a function, macro, or special form. (doc map-indexed)
dir
Shows a list of functions and other vars in a certain namespace. (dir clojure.string)
pst
Given an exception, print its message and stacktrace. (try (/ 1 0) (catch Exception e (pst e)))
find-doc
Searches all docstrings for a string or regex pattern. (find-doc "join")

From clojure.java.javadoc

javadoc
Opens the documentation for a Java class in your browser. (javadoc java.util.regex.Pattern)

From clojure.pprint

pprint
Show a "pretty" formatted representation of a value. (pprint (mapv #(vec (range % 10)) (range 10)))
pp
Pretty print the previous result, same as (pprint *1)

There are also a few "magic" variables available

*1, *2, *3
The result of the last, penultimate, and third to last evaluation
*e
The last uncaught exception

To exit the REPL, type Ctrl-D.

Line Editing and History Support

When you try to use the arrow keys in the REPL you might end up with gibberish ^[[A. Clojure's REPL does not come with editing or history support out of the box.

Most REPLs rely on the GNU Readline library for these features, and in the Java world there's JLine which emulates Readline.

There's a nifty little tool called rlwrap, which can add Readline support to programs that don't have it themselves. So if you want a Clojure REPL with full editing and history support, you can start it like this:

rlwrap java -jar clojure-1.8.0.jar

Socket REPL

When a REPL "reads" and "prints", where does it read from, or print to? The answer is: standard input, and standard output.

Every process gets a byte stream that it can read from called "standard input", also called STDIN, System.in in Java, or clojure.core/*in* in Clojure. If you start a program from a terminal, then this input stream receives your keyboard's keystrokes.

Each process also starts with two output streams, "standard output" and "standard error". Standard output (STDOUT, System.out, clojure.core/*out*) is where the REPL prints its results, and so they show up in your terminal.

So what if you hook something else up as input and output stream? For instance, a network socket. Would that still work?

Turns out it does, and this is what Clojure's Socket REPL can do for you. The name Socket REPL is actually misleading, because there is nothing REPL-specific about this feature. Instead it's a general facility that causes Clojure to start a TCP server, and wait for connections. Whenever a connection is received, Clojure will connect *in* and *out* to the network connection's input and output streams, and then pass control over to a chosen function.

If that function happens to be clojure.main/repl, well, then you've got yourself a network REPL!

You tell Clojure to start a socket REPL using the clojure.server.repl property, which you can configure like this:

java -cp clojure-1.8.0.jar -Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}" clojure.main

When you run that command you will see a REPL appear, but that's not the socket REPL we are talking about, it's just the regular REPL started by clojure.main. To see the socket REPL in action, open a second terminal, and connect to it with Netcat (alternatively you can use telnet)

nc localhost 5555

You can even add rlwrap to the mix to get a feature rich REPL experience. The great thing about this socket REPL is that

  1. it works over the network, so you can connect to an app running somewhere else
  2. it doesn't require any source code changes, you can have an app that's already fully compiled and deployed, and give it REPL support just by adding a command line flag.

Don't expose this socket REPL to the internet though! Make sure your firewall blocks outside traffic to this port, then set up an SSH tunnel between the server and your local machine to connect to it.

Creating Your Own "Socket REPLs"

We told Clojure to call the clojure.core.server/repl function each time it receives a new connection. This is just a small variant of clojure.main/repl with some extra initialization. We can also tell Clojure to use a different function.

Create a file called wrapple.clj in the current directory.

(ns wrapple)

(defn echo-time []
  (println (.toString (java.util.Date.))))

Now start a "socket REPL" using this function:

java -cp clojure-1.8.0.jar:. -Dclojure.server.repl="{:port 5555 :accept wrapple/echo-time}" clojure.main

And you've got yourself a network service that echos the time!

$ nc localhost 5555
Thu Jul 07 16:22:17 CEST 2016

We can even make our own REPL loop, like this one, which will turn anything you give it into uppercase.

(ns wrapple
  (:require [clojure.string :as str]))

(defn prompt-and-read []
  (print "~> ")
  (flush)
  (read-line))

(defn uprepl []
  (loop [input (prompt-and-read)]
    (-> input
        str/upper-case
        println)
    (recur (prompt-and-read))))
java -cp clojure-1.8.0.jar:. -Dclojure.server.repl="{:port 5555 :accept wrapple/uprepl}" clojure.main
$ nc localhost 5555
~> isn't this awesome?
ISN'T THIS AWESOME?
~>

nREPL

nREPL stands for "network REPL", and while this may sound pretty similar to a "socket REPL", they are completely different animals.

The REPLs we've seen so far are stream based, they read lines from an input stream, and write the result (and any output from side effects) to an output stream. This makes them conceptually simple, but tedious to communicate with programmatically.

nREPL seeks to fix this by being message-based, putting program-to-program first through a client-server architecture, where the client is your editor or IDE, and the server is the nREPL "REPL".

The client initiates the interaction by sending a message to the server, and as a response the server will send one or more messages back to the client.

The easiest way to start an nREPL server is either through Leiningen (lein repl) or Boot (boot repl). In either case the first line of output will contain a port number that an nREPL client can connect to.

Messages

A message might look something like this.

{:id "10"
 :op "eval"
 :code "(+ 1 1)\n"
 :ns "user"}

nREPL supports a number of "operations", the quintessential being "eval", which simply evaluates some code.

Conceptually a message is like a Clojure map, a set of key-value pairs. The precise keys used depend on the type of message. In the case of eval it needs to know the code to evaluate, as well as the namespace to use for context.

The response will look something like this.

{:id "10"
 :out "2\n"}

An editor or IDE could provide a Clojure REPL by implementing Read, Print, and Loop, and letting nREPL handle the Evaluate part, but it can do much more than just emulating a traditional REPL. nREPL can power "live" evaluation (sending forms from an editor buffer directly to a Clojure process), it can be used to look up documentation, inspect a running program, and much more.

nREPL provides a range of operations beyond "eval", together they can be used to offer rich Clojure editing support. Through "nREPL middleware" it is possible to add support for new operations.

nREPL supports these operations out of the box

eval
Evaluate some code, returns the output value
interrupt
Attempt to interrupt the current eval operation
describe
Returns a list of currently supported operations, as well version information
load-file
(re-)load a complete source file

Sessions

nREPL comes with a built-in "session" middleware. This way you can have multiple REPLs backed by a single nREPL server. The session middleware adds three operations

ls-session
list the current sessions
clone
create a new session
close
close a session

Each message will now carry a session-id. This becomes important when dealing with the standard in- and output streams, *in*, *out*, and *err*.

A single eval operation might cause many lines of output, possibly asynchronously, so they arrive after the eval has completed. The session middleware will intercept this output, and send it back to the client with the same session-id as the original eval message.

Here's an example interaction, evaluating the code

(dotimes [i 3]
  (Thread/sleep 1000)
  (println (str "==> " i)))

I'm using the message representation used by CIDER, with --> indicating a message to the server, and <-- being a message sent back to the client.

(-->
  ns  "user"
  op  "eval"
  session  "10299efe-8f84-4b97-814c-21eb6860223b"
  code  "(dotimes [i 3] (Thread/sleep 1000) (println (str \"==> \" i)))\n"
  file  "*cider-repl localhost*"
  line  53
  column  6
  id  "15"
)
(<--
  id  "15"
  out  "==> 0\n"
  session  "10299efe-8f84-4b97-814c-21eb6860223b"
)
(<--
  id  "15"
  out  "==> 1\n"
  session  "10299efe-8f84-4b97-814c-21eb6860223b"
)
(<--
  id  "15"
  out  "==> 2\n"
  session  "10299efe-8f84-4b97-814c-21eb6860223b"
)
(<--
  id  "15"
  ns  "user"
  session  "10299efe-8f84-4b97-814c-21eb6860223b"
  value  "nil"
)
(<--
  id  "15"
  session  "10299efe-8f84-4b97-814c-21eb6860223b"
  status  ("done")
)

Custom Middleware

Through "nREPL middleware" you can even implement your own operations. Some notable "middlewares" are piggieback, cider-nrepl, and nrepl-refactor. Here's an example of the "eldoc" operation provided by CIDER-nREPL.

(-->
  op  "eldoc"
  session  "ff822558-e885-49ed-8cf6-b5331bc8553b"
  ns  "user"
  symbol  "str"
  id  "10"
)
(<--
  docstring  "With no args, returns the empty string. With one arg x, returns\n  x.toString().  (str nil) returns the empty string. With more than\n  one arg, returns the concatenation of the str values of the args."
  eldoc  (nil
 ("x")
 ("x" "&" "ys"))
  id  "10"
  name  "str"
  ns  "clojure.core"
  session  "ff822558-e885-49ed-8cf6-b5331bc8553b"
  status  ("done")
  type  "function"
)

Writing your own middleware isn't hard. Let's make a middleware that adds a "classpath" operation, which returns the current Java classpath. It's trivial but perhaps not altogether useless.

(ns classpath-nrepl.middleware
  (:require [clojure.tools.nrepl
             [middleware :refer [set-descriptor!]]
             [transport :as transport]]))

(defn wrap-classpath [handler]
  (fn [{:keys [id op transport] :as request}]
    (if (= op "classpath")
      (transport/send transport {:id id
                                 :classpath (->> (java.lang.ClassLoader/getSystemClassLoader)
                                                  .getURLs
                                                  (map str))})
      (handler request))))

(set-descriptor! #'wrap-classpath
  {:requires #{}
   :expects #{}
   :handles {"classpath" {:doc "Return the Java classpath"}}})

At the heart of any nREPL server is the "handler", a function which handles a single incoming message. A middleware is a function which takes the existing handler function and "wraps" it, returning a new handler function.

Our new handler will look at each incoming request, and if the operation is anything but classpath, it simply calls the old handler. It's basically saying, "I don't care about this request, you handle it!"

When it receives a classpath message however, then it constructs a response, and sends it back to the client.

As part of the incoming request the handler received a "transport" object, which it needs to use to reply to the client. This is how nREPL achieves asynchrony. A handler can send several messages back to the client based on a single incoming message, possibly much later or from another thread. It only needs to make sure to use the transport and id of the original message.

To see this middleware in action, make sure you have tools.nrepl in your dependencies, and add some :repl-options to your Leiningen project.clj to insert the middleware into the "middleware stack".

(defproject middleware-test "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [org.clojure/tools.nrepl "0.2.12"]]

  :repl-options {:nrepl-middleware [classpath-nrepl.middleware/wrap-classpath]})

If you're using Boot then you can insert the middleware like this in your build.boot

(require 'boot.repl)

(swap! boot.repl/*default-middleware*
       conj 'classpath-nrepl.middleware/wrap-classpath)

If you're starting your own tools.nrepl server directly instead of using lein repl or boot repl, then you can pass the middleware as an argument to the "default handler".

(require '[clojure.tools.nrepl.server :as nrepl])
(require '[classpath-nrepl.middleware :refer [wrap-classpath]])

(nrepl/start-server :handler (server/default-handler #'wrap-classpath))

Now start a REPL with lein repl, and note the port number that Leiningen prints in the first line of output, for example nREPL server started on port 41065 on host 127.0.0.1. Now you can connect to it, either from the same process, or from a completely different REPL.

user> (require '[clojure.tools.nrepl :as repl])
nil
user> (def conn (repl/connect :port 41065))
#'user/conn
user> (repl/message (repl/client conn 1000) {:id 1 :op "classpath"})
({:classpath ["file:/home/arne/.m2/repository/org/clojure/clojure/1.8.0/clojure-1.8.0.jar" ,,,], :id 1})

Here's what the interaction looks like:

(-->
  op  "classpath"
  session  "ff822558-e885-49ed-8cf6-b5331bc8553b"
  id  "1"
)
(<--
  classpath  ("/home/arne/.m2/repository/org/clojure/clojure/1.8.0/clojure-1.8.0.jar" ...)
  id  "1"
  session  "ff822558-e885-49ed-8cf6-b5331bc8553b"
  status  ("done")
)
  1. Notable middleware

    The CIDER-nREPL README has a list of included middlewares and their supported operations.

    Many more operations are added by cider-nrepl.

    Piggieback allows using nREPL with ClojureScript, see the Piggieback section under ClojureScript REPLs.

Protocol

Messages between an nREPL client and server are sent over the wire using "BEncode" (pronounced B-encode). BEncode was originally developed for use in BitTorrent, it's a binary format that supports integers, strings, lists, and dictionaries.

While BEncode is a binary protocol, not intended for human consumption, it is kind of readable if you know what you're looking for.

Take this Clojure Map

{:id 1 :op "eval" :code "(+ 1 1)\n"}

Here's the BEncoding of it

"d4:code8:(+ 1 1)\n2:idi1e2:op4:evale"

The way to read this is

  • d dictionary
    • 4:code a 4 byte string: "code"
    • 8:(+ 1 1)\n an 8 byte string: "(+ 1 1)\n"
    • 2:id a 2 byte string: "id"
    • i1e an integer, 1
  • e end of dictionary

Clients

It should be clear by now that communicating with nREPL is best done through a dedicated client library. The tools.nrepl package contains both the nREPL server and a client implementation that you can use from Clojure.

The CIDER README lists editors and tools that support nREPL.

ClojureScript clients for nREPL

Advanced, you can safely skip this section.

This section is a stub, you can help by expanding it.

This section is about connecting to an nREPL server from a ClojureScript client. If instead you want your nREPL server to use a ClojureScript environment to do evaluation, see the section on Piggieback.

By default nREPL communicates over TCP sockets, using Bencode as its data format. Raw TCP sockets are not available from a browser, but they are available in Node.js. You can use the nrepl-client package on NPM to connect from Node.js to an nREPL server.

The nREPL transport is pluggable, so a more browser friendly alternative would be to use HTTP over JSON, which is what Drawbridge provides. It is implemented as a Ring handler, so you need to use it in conjunction with a Ring server adapter (Jetty, Http-kit, Aleph, …)

Drawbridge contains an nREPL client that uses the same transport, but that one is for Clojure, not ClojureScript. For a ClojureScript client you can use drawbridge-cljs.