Test Wars: A New Hope

Yesterday was the first day for me on a new job, thanks to Clojurists Together I will be able to dedicate the coming three months to improving Kaocha, a next generation test runner for Clojure.

A number of projects applied for grants this quarter, some much more established than Kaocha. Clojurists Together has been asking people through their surveys if it would be cool to also fund “speculative” projects, and it seems people agreed.

I am extremely grateful for this opportunity. I hope to demonstrate in the coming months that Kaocha holds a lot of potential, and to deliver some of that potential in the form of a tool people love to use.

In this post I’d like to set the stage by providing a bit of context, and explaining the problems Kaocha tries to solve, as well as providing a rough plan for the coming months.

Conventions and interfaces

When growing an ecosystem it’s important to have conventions and well defined interfaces in place. This allows parts of the ecosystem to develop separately, while still functioning as a whole.

Clojure has done a fairly good job of providing this shared foundation with regard to testing, and many tools and libraries have been built on top of it. At the same time some have chosen to go off and do their own thing, possibly because the affordances provided by Clojure’s testing conventions did not support their use cases.

When talking about a testing ecosystem we’re talking about a bunch of different things, which may or may not come in a single package.

  1. Test syntax: how to define tests in code
  2. Test semantics: how does the system define what is a test
  3. Assertion libraries: ways to write falsifiable test statements
  4. Mocking/stubbing libraries: utilities to help with test isolation
  5. Test fixtures and factories: utilities for building up test data and establishing application states
  6. Progress reporters: ways to get progress information while tests are running
  7. Result reporters: summarizing the results of a test run for human or machine consumption
  8. Test runners: comprehensively load, detect, and run test suites

Clojure 1.0 provided exactly one of these, number 2. When a var has :test metadata, then that is considered a test, and can be run with clojure.core/test. You could argue it also had a bit of 3 with clojure.core/assert, and a bit of 4 with with-redefs.

Clojure 1.1 added the clojure.test library which improves on number 3 (assertions) with an extensible macro: (is), as well as providing syntax (1), and reporters (6 and 7).

More importantly it provides conventions so you can bring your own version of these things. (is ...) is powered by the clojure.core/assert-expr multimethod, so for instance matcher-combinators provides a custom (is (match? ...)) assertion.

The reporter in clojure.test is also a multimethod. It receives “events” like :begin-test-var, :pass, or :fail, and can be extended to handle new types of event, like matcher-combinator’s :mismatch. It can also be swapped out with with-redefs, for instance when running tests through CIDER a custom reporter will be used to collect test results.

There is a chance of conflict here though, what if one library extends report, and the other replaces it? This illustrates a general problem with clojure.test, to build tooling on top of it you are forced to rely on details of the implementation, leading to brittle results that don’t compose well.

Clojure itself does not contain a test runner. There are some utilities in clojure.test to execute tests once they are defined, but no comprehensive way to say “load all test namespaces under this directory path, find all tests in them, and run them.” The role of a test runner has initially been performed by Leiningen, which includes lein test. It’s fairly basic but gets the job done. It will set an appropriate process exit code (important on CI), and provides some ways to filter or skip tests. It uses stock clojure.test output which is rather spartan.

Several libraries and Leiningen plugins came about to provide nicer output for failed assertions (humane-test-output, ultra), as well as things like a nice progress bar, output capturing and running tests in parallel (eftest).

When Boot came around it also needed a test runner, and since lein test is tightly linked to Leiningen the boot folks implemented boot-test, which uses humane-test-output under the hood.

The fine folks from Metosin later created bat-test, which is perhaps the most feature complete of the ones covered so far. It works with boot and leiningen, supports eftest and cloverage, has global pre/post hooks and more.

And then tools.deps / Clojure CLI came around and we were back to square one, waiting for someone to port their existing test runner, or implement a new one. Cognitect came around with cognitect-labs/test-runner, which is probably the least feature complete of the bunch.

Finally there’s circleci.test, which provides some really neat features like enforcing tests are pure and generating junit.xml output.

Imperative vs functional

If we have this many test runners already, why create another one? All of the mentioned projects are available for a subset of Clojure build tools (lein, boot, Clojure CLI), and implement a subset of all possible test runner features, but they don’t compose. I can’t take some features of one and a few of the other.

Clojure’s philosophy is to provide pure functions over common data structures, so it’s easy to piece things together. The problem is that running tests is inherintly imperative. You load code, execute tests in order, and observe events and exceptions that occur. These get aggregated in some kind of mutuble structure, while progress is reported to the user.

The problem is that this kind of code doesn’t compose well, and so we end up with all these different projects with very little opportunity for synergy or reuse.

Kaocha addresses this by turning the imperative problem into a functional one. At the heart of Kaocha is the abstraction of a testable, a Clojure map with an :id, a :type, and any other information that characterizes this test. When you (run testable) you get a similar map back, but this time containing test results: the amount of assertions that ran, errors, failures, captured exceptions, captured output to stdout, and so forth.

run is by no means pure, since a test can cause (and verify) any number of side effects, but tooling authors can choose to ignore the side effects. All they care about is captured and returned as pure data.

Testables are built up hierarchically. You can run a test-suite, which will run any namespaces it contains, which in turn will run each test var. The top level testable is called the test-plan.

(def test-plan
  {:tests [{:id :unit
            :type :clojure.test
            :tests [{:id :kaocha.result-test
                     :type :ns
                     :tests [{:id :kaocha.result-test/totals-test
                              :type :var
                              ,,,}]}]}]})

(run test-plan)
;;=>
{:tests [{:id :unit
          :type :clojure.test
          :tests [{:id :kaocha.result-test
                   :type :ns
                   :tests [{:id :kaocha.result-test/totals-test
                            :type :var
                            :count 1
                            :fail 0
                            :pass 1
                            :error 0
                            ,,,}]}]}]}

Kaocha is still built upon the foundations of clojure.test, and uses the same “reporter” abstraction for providing real time progress info. This isn’t always easy, as mentioned before clojure.test makes it easy to step on each other’s toes when extending or replacing the reporter. Kaocha tries to prevent this by providing its own conventions for extensions that are more readily composable. You can add additional reporters, instead of replacing the reporter wholesale. You can make Kaocha aware of custom event types through keyword hierarchies, and by implementing multimethods you can influence how custom events are rendered.

See the docs about custom reporters for more info.

More than one type of tests

Long gone are the days that clojure.test was the only game in town. What if you prefer to write your tests with midje, fudje, expectations, selvage, or cucumber?

Some of these stick to the clojure.test conventions, and can be used with any of the above test runners, others do not. Midje in particular is interesting here, as it provides its own version of more or less everything we mentioned: syntax, test runner, mocking/stubbing and more, and it does it in a very idiosyncratic way, deliberately moving away from clojure.test.

So you can’t for instance use Midje with bat-test, or circleci.test, you can only use Midje with Midje. Conversely you can’t easily use features of Midje without adopting it wholesale.

What’s worse, imagine you want to use one testing library for your unit tests, and another for your integration tests, now you’re dealing with separate tools, each with their own configuration and interface.

Kaocha tries to solve this by providing a test type (or test suite) abstraction. To add support for a given test type to Kaocha you implement two multimethods, load and run. Getting these right isn’t trivial, as they need to do the double book keeping mentioned above, invoking the reporter as things are happening, and collecting and returning the results in a functional way. What they don’t need to worry about is all the rest of the test runner machinery. They don’t need to provide a CLI or REPL interface, they don’t need to think about functionality for filtering or skipping tests, for profiling or output capturing, all of that is provided by Kaocha.

Kaocha does not need to know about your test type up front, it just needs to be on the classpath and follow certain naming conventions. This means that anyone can implement a new test type and push it to clojars, encouraging experimentation and innovation.

Besides supporting command line use (kaocha.runner), or REPL interaction (kaocha.repl), Kaocha can also be invoked directly by tooling (kaocha.api). Perhaps in the future editor integrations based on Kaocha will gain support for all these different test types without any extra work.

Moar features

People have many different approaches to writing and running tests, and this leads to different opinions about the features a test tool should support. One person’s essentials is an other person’s bloatware.

Kaocha tries to address this through plugins. Like test types these can be released by anyone, they are just Clojure libraries that follow certain naming conventions. Because of the functional, data-driven foundation that Kaocha provides plugin authors can ignore a lot of the details of running tests, and focus on the problem they are trying to solve.

For example the kaocha-junit-xml plugin provides support for writing out a junit.xml file with test results.

The guts of this plugin look like this. It implements three “hooks”, one to add an extra command line argument, one that transforms the test configuration, and that runs after all tests have finished.

The first two establish a common pattern, allowing configuration through tests.edn that can be overridden by supplying command line arguments.

(defplugin kaocha.plugin/junit-xml
  "Write test results to junit.xml"

  (cli-options [opts]
    (conj opts [nil "--junit-xml-file FILENAME" "Save the test results to a Ant JUnit XML file."]))

  (config [config]
    (if-let [target (get-in config [:kaocha/cli-options :junit-xml-file])]
      (assoc config ::target-file target)
      config))

  (post-run [result]
    (when-let [filename (::target-file result)]
      (write-junit-xml filename result))
    result))

write-junit-xml takes the test result, a nested map with the details of every single test, and transforms it into XML. Quite a difference from clojure.test.junit, which imperatively spits out the XML bit by bit as it goes along.

(defn write-junit-xml [filename result]
  (with-open [f (io/writer (io/file filename))]
    (binding [*out* f]
      (clojure.xml/emit (test-result->xml result)))))

Note also that this plugin will work with any test type, not just clojure.test, because they all produce the same Kaocha test-result data structure.

Many of the features I have planned for Kaocha will be provided as plugins. Those that rely on extra third party dependencies will be deployed as separate jars. Maximum features, minimum bloat!

Some other plugin ideas

  • Cloverage integration
  • Detecting side effects (a la circleci.test)
  • Configurable global pre/post steps
  • Configuring dynamic bindings in tests.edn (e.g. clojure.core/*print-length*)
  • Configuration for test.check (report-trials, report-shrinking, default-test-count)

Where are we now?

I’ve been working steadily on Kaocha since April, initially focusing on the base data structures, abstractions, and common functionality. I also put some effort into ergonomics, making sure the tool is pleasant to use. Most recently this led to deep-diff, which was split out into its own library.

The core functionality and extension points are there, including test types, reporters and plugins, as well as essential features like watching/autoloading, skipping and filtering, profiling, randomizing test order, and more.

clojure.test is the only test type that’s well supported so far, there’s a proof of concept of a Midje test type.

The release process has been streamlined so I can push out new versions quickly, you can expect new releases multiple times per week. Kaocha’s own tests run on CircleCI on a mix of Java and Clojure versions, and documentation is published on cljdoc.

What’s next?

I’ll be funded by Clojurists Together from November through January, and I do intend to make it count.

I have a long list of things I want to work on. I’ll start turning these into Github issues in the coming days so people can follow along.

Some of the things I mentioned in my Clojurists Together application have already been implemented, like junit.xml output, and pretty diffing of test failures, which got pulled out into a standalone deep-diff library.

Boot support is high on the list, with that all three major build tools will be supported. Cloverage integration is also a priority, partly because I want to get a better view on Kaocha’s own coverage.

I’ll also start working on supporting more test types. The current test type abstraction hasn’t been fully put to the test yet, it will be interesting to see how well it can support other testing libraries, or whether its design needs to evolve.

The big one of these will be a ClojureScript test type. One of my dreams with Kaocha is to have a dead simple way to drop a tests.edn into your project and start testing both Clojure and ClojureScript code. This will be a major task as I explained elsewhere, as it involves running the ClojureScript compiler, collecting metadata from the compiler, spinning up or connecting to a JavaScript runtime, invoking tests in a JS/CLJS harnass, and collecting the result.

Parallelization is one of the few features of Eftest and bat-test that Kaocha does not yet support. It hasn’t been high on the list because Kaocha by default randomizes the test order, providing a random seed that can be used to recreate a test run. This is a useful feature for catching unintended dependencies between tests, but it relies on test order being deterministic, which with parallelism is no longer the case. Still I understand why people would want to trade this determism in for higher throughput, and so I do intend to bring in parallelism, but it might require some reworking of Kaocha’s core.

There are also a bunch of smaller incremental improvements, like providing nicer feedback in case of misconfiguration. My current thinking is to get a lot of the smaller tasks out of the way in the first few weeks in order to get some momentum, and then start tackling ClojureScript support.

Towards the end of the three month program I intend to start working on documentation, and on making sure the project is easy to contribute to.

Meanwhile I’d love to hear your feedback. Did you try Kaocha yet? What did you think? Are there features missing that are blocking you from switching?

You can contact me directly, or comment on this post on r/clojure.

More blog posts

The Art of Tree Shaping with Clojure Zippers

This is a talk I did for the “Den of Clojure” meetup in Denver, Colorado. Enjoy!

Captions (subtitles) are available, and you can find the transcript below, as well as slides over here.

For comments and discussion please refer to this post on r/Clojure.

Two Years of Lambda Island, A Healthy Pace and Things to Come

It’s been just over two years since Lambda Island first launched, and just like last year I’d like to give you all an update about what’s been happening, where we are, and where things are going.

To recap: the first year was rough. I’d been self-employed for nearly a decade, but I’d always done stable contracting work, which provided a steady stream of income, and made it easy for me to unplug at the end of the day.

Lambda Island was, as the Dutch expression goes, “a different pair of sleeves”. I really underestimated what switching to a one-man product business in a niche market would mean, and within months I was struggling with symptoms of burnout, so most of year one was characterised by trying to keep things going and stay afloat financially, while looking after myself and trying to get back to a good place, physically and mentally.

D3 and ClojureScript

This is a guest post by Joanne Cheng (twitter), a freelance software engineer and visualization consultant based in Denver, Colorado. She has taught workshops and spoken at conferences about visualizing data with D3. Turns out ClojureScript and D3 are a great fit, in this post she’ll show you how to create your own visualization using the power of D3 and the elegance of ClojureScript.

I use D3.js for drawing custom data visualizations. I love using the library, but I wanted to try one of the several compile to JavaScript options, and I decided to look into ClojureScript. It ended up working out well for me, so I’m going to show you how I created a D3.js visualization using ClojureScript!

What we’re visualizing

Reloading Woes

Setting the Stage

When doing client work I put a lot of emphasis on tooling and workflow. By coaching people on their workflow, and by making sure the tooling is there to support it, a team can become many times more effective and productive.

An important part of that is having a good story for code reloading. Real world projects tend to have many dependencies and a large amount of code, making them slow to boot up, so we want to avoid having to restart the process.

The Bare Minimum, or Making Mayonnaise with Clojure

Making Mayonnaise

Imagine you have a grandfather who’s great at making mayonnaise. He’s been making mayonnaise since before the war, and the result is truly excellent. What’s more, he does this with a good old fashioned whisk. He’s kept his right arm in shape throughout decades just by beating those eggs and oil and vinegar.

Now he’s bought himself a handheld electric mixer after hearing his friend rave about hers, but after a few tries he gives up and goes back to his whisk. He says he just can’t get the same result. This seems slightly odd, so the next time you go over you ask him to show you how he uses the mixer.

Clojure Gotchas: "contains?" and Associative Collections

When learning a programming language we rarely read the reference documentation front to back. Instead someone might follow some tutorials, and look at sample code, until they’re confident enough to start a little project for practice.

From that point on the learning process is largely “just in time”, looking up exactly the things you need to perform the task at hand. As this goes on you might start to recognize some patterns, some internal logic that allows you to “intuit” how one part of the language works, based on experience with another part.

Developing this “intuition” — understanding this internal logic — is key to using a language effectively, but occasionally our intuition will be off. Some things are simply not obvious, unless someone has explained them to us. In this post I will look at something that frequently trips people up, and attempt to explain the underlying reasoning.

Dates in Clojure: Making Sense of the Mess

Update 2018-11-27: while most of this article is still relevant, I no longer recommend using JodaTime as the main date/time representation for new projects. Even existing projects that aren’t too invested in JodaTime/clj-time should consider migrating to java.time and clojure.java-time across the board.

You can always count on human culture to make programming messy. To find out if a person is a programmer just have them say “encodings” or “timezones” and watch their face.

In Java, and hence Clojure, there are half a dozen different ways to represent dates and times, which can lead to confusion, needless type coercion, and inconsistent code. In this post I’ll give you a quick lay of the land, and some tips to make it all a bit easier to deal with.

Clojure Gotchas: Surrogate Pairs

tl;dr: both Java and JavaScript have trouble dealing with unicode characters from Supplementary Planes, like emoji 😱💣.

Today I started working on the next feature of lambdaisland/uri, URI normalization. I worked test-first, you’ll get to see how that went in the next Lambda Island episode.

One of the design goals for this library is to have 100% parity between Clojure and ClojureScript. Learn once, use anywhere. The code is all written in .cljc files, so it can be treated as either Clojure or ClojureScript. Only where necessary am I using a small amount of reader conditionals.

Simple and Happy; is Clojure dying, and what has Ruby got to do with it?

The past week or so a lot of discussion and introspection has been happening in the Clojure community. Eric Normand responded to my one year Lambda Island post with some reflections on the size and growth of the community.

And then Zack Maril lamented on Twitter: “I’m calling it, clojure’s dying more than it is growing”. This sparked a mega-thread, which was still raging four days later. A parallel discussion thread formed on Reddit. Someone asked if their were any Clojure failure stories. They were pointed at this talk from RubyConf 2016, which seemed to hit a lot of people right in the feels, and sparked a subthread with a life of its own.

Meanwhile Ray, one of the hosts of the defn podcast reacted to the original tweet: “I’m calling it: Clojure is alive and well with excellent defaults for productive and sustainable software development.” This sparked another big thread.

Loading Clojure Libraries Directly From Github

Did you ever fix a bug in an open source library, and then had to wait until the maintainer released an updated version?

It’s happened to me many times, the latest one being Toucan. I had run into a limitation, and found out that there was already an open ticket. It wasn’t a big change so I decided to dive in and address it. Just a little yak shave so I could get on with my life.

Now this pull request needs to be reviewed, and merged, and eventually be released to Clojars, but ain’t nobody got time for that stuff. No sir-ee.

Lambda Island Turns One, The Story of a Rocky Ride

One year ago to date I launched Lambda Island, a service that offers high quality video tutorials on web development with Clojure and ClojureScript. It’s been quite a ride. In this post I want to look back at the past year, provide some insight into how this experience has been for me, and give you a glimpse of what the future has in store.

This story really starts in December 2015. After three years of doing contract work for Ticketsolve I decided it was time for a change. I have been self-employed for many years, but I knew that sooner or later I wanted to try my hand at selling a product, rather than selling my time.

In January and February I took some time for soul-searching, and recharging. I went to speak at RubyConf Australia, and got to hang out with some old friends around Australia and New Zealand. Once back in Berlin I got busy creating Lambda Island.

Writing Node.js scripts with ClojureScript

In the two most recent  Lambda Island episodes I covered in-depth how to create command line utilities based on Lumo, how to combine them with third party libraries, and how to deploy them to npmjs.com.

However there’s a different way to create tools with ClojureScript and distribute them through NPM, without relying on Lumo. In this blog post I want to quickly demostrate how to do just that.

To recap, Lumo is a ClojureScript environment based on Node.js, using bootstrapped (self-hosted) ClojureScript. This means the ClojureScript compiler, which is written in Clojure and runs on the JVM, is used to compile itself to JavaScript. This way the JVM is no longer needed, all you need is a JavaScript runtime to compile and run ClojureScript code, which in this case is provided by Node.js. On top of that Lumo uses nexe, so Lumo can be distributed as a single compact and fast executable binary.

Announcing lambdaisland/uri 1.0.0

I just released lambdaisland/uri, a pure Clojure/ClojureScript URI library. It is available on Github and Clojars.

This is a small piece of the code base that powers lambdaisland.com. It’s inspired by Ruby’s Addressable::URI, the most solid URI implementation I’ve seen to date, although it only offers a small part of the functionality that library offers.

It’s written in pure Clojure/ClojureScript, with only minimal use of .cljc reader conditionals to smooth over differences in regular expression syntax, and differences in core protocols. It does not rely on any URI functionality offered by the host, such as java.net.URL, so it’s usable across all current and future Clojure implementations (Clojure, ClojureScript, ClojureCLR).

re-frame Subscriptions Got Even Better

Up until recently, to use re-frame subscriptions in Reagent views, you had to use a form-2 component.

A form-2 component is a function that returns another function, which does the actual rendering of the component to hiccup. In contrast, a form-1 component renders the hiccup directly.

;; form-1
(defn todo-item [todo]
  [:div.view
   [todo-checkbox (:id todo) (:completed todo)]
   [:label {:unselectable "on"} title]
   [:button.destroy {:on-click #(dispatch [:todos/remove (:id todo)])}]])

;; form-2
(defn todo-item [todo]
  (fn [todo]
    [:div.view
     [todo-checkbox (:id todo) (:completed todo)]
     [:label {:unselectable "on"} title]
     [:button.destroy {:on-click #(dispatch [:todos/remove (:id todo)])}]]))

Game Development with Clojure/ClojureScript

This weekend it’s Ludum Dare again, the world’s longest running game jam. The idea is that, alone or with a team, you build a game in a weekend based on a certain theme.

We got a little team together here in Berlin, and so I’ve been reviewing what options there are for someone wanting to build a game in Clojure or Clojurescript.

The good news is there are plenty of options, as you’ll see from the list below. You can do desktop games, browser based games with canvas or webgl, and you can even create Unity 3D games, all from your comfortable Clojure parentheses.

Union Types with Clojure.Spec

Elm and other statically typed languages have a great feature called Union Types (also called Sum Types or Algebraic Data Types).

Here’s an example taken from Elm. Suppose your system used to represent users as integers, maybe just an auto-incrementing primary key, but then switched to UUIDs represented as strings.

To correctly model this situation, you need a way to create a type that can be either an integer or a string, that’s what union types give you.