Lambda Island

Testing libraries and frameworks are indispensable, but there’s no magic to how they work. Through implementing your own mini testing framework, you’ll learn how clojure.test does things under the hood.

While the main plot demonstrates how to write a function test-first, you’ll also meet several supporting characters, like preconditions, dynamic bindings, and even a macro. Finally the mask comes off. It was clojure.test all this time!

Transcript

A great exercise to do in any language is creating your own unit testing framework. While these frameworks can get pretty advanced, you actually need very little to be able to define and run tests.

In fact all you need to get started is one simple function, “assert equal”, which verifies that two values are the same.

If they are then nothing happens, if they’re not, then it throws an AssertionError, which is a built-in Error type, and we’ll give it a nice message: “I expected to get this thing, but instead I got that thing”.

(defn assert= [x y]
  (when (not= x y)
    (throw (AssertionError. (str "Expected " (pr-str y)
                                 " but got " (pr-str x))))))

(assert= 5 5)
;;=> nil

(assert= 5 6)
;;=> Unhandled java.lang.AssertionError
;;=> Expected (= 5 6) but got (not= 5 6)

And that’s all you need, now you can start writing tests. A test in this case means a piece of code that verifies the behavior of some other piece of code.

Let’s try it out by writing a function that converts Hiccup syntax to the format used by clojure.xml. Besides the actual implementation, I’ll also write a second function, which will act as the test.

(defn hiccup->cljxml [h]
  )

(defn test-hiccup->cljxml []
  )

In Hiccup syntax an HTML element is represented as a vector, whereas in clojure.xml it’s a map with :tag, :attrs, and :content keys.

;; hiccup
[:p {:class "info"} "🖐"]

;; clojure.xml
{:tag :p :attrs {:class "info"} :content '("🖐")}

I’ll test-drive this implementation, that means I first add a failing test case, and then add just enough code to make it pass. Then I add a second test case, make it pass, a third, and so on. This cycle continues until I can no longer come up with failing test cases, in which case either my imagination falls short, or the implementation is complete.

A simple first case could be: converting an empty paragraph. After calling hiccup->cljxml I should get a map back with a :p tag, and empty attributes and content.

(defn test-hiccup->cljxml []
  (assert= (hiccup->cljxml [:p]) {:tag :p, :attrs {}, :content ()}))

How much code do I need to make this pass? Not a lot. In fact I can just copy over this map and call it a day. There, all good.

This may look like cheating, but it’s actually a fine way to start. If that’s not good enough yet then show it by adding another test case.

So for instance when I convert a div, then I expect a :tag of :div, but instead I’m getting :p, now I can improve the code, and verify that the tests pass.

This demonstrates an important rule: always watch a new test fail before you make it pass. It is deceptively easy to write tests that test the wrong thing, or that don’t test anything at all. By watching it fail first and then pass, you know that you’re really testing what you think you are.

(defn hiccup->cljxml [h]
  {:tag (first h), :attrs {}, :content ()})

(defn test-hiccup->cljxml []
  (assert= (hiccup->cljxml [:p]) {:tag :p, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:div]) {:tag :div, :attrs {}, :content ()}))

(test-hiccup->cljxml)

Ok, let’s try another one. I’m adding another paragraph but this time with some text inside, and making sure that the text shows up as content.

Now run the test to watch it fail. Now you can update the function. The content is this case is everything but the first element of the vector, so you could use either rest or next, but if the content is empty then I want it to return an empty list instead of nil, so I’ll use rest.

Does that work? Apparently not, seems I forget an exclamation mark there. Ok, and now the tests all pass again.

(defn hiccup->cljxml [h]
  {:tag (first h), :attrs {}, :content (rest h)})

(defn test-hiccup->cljxml []
  (assert= (hiccup->cljxml [:p]) {:tag :p, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:div]) {:tag :div, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:p "hello!"]) {:tag :p, :attrs {}, :content '("hello!")}))

(test-hiccup->cljxml)

Now it starts getting interesting, because the second element can also be a map of attributes, and right now it’s being treated as a child element and added to content.

I’ll use a let block to keep this readable, check if the second element is a map, and then either use it as the attributes map, or use an empty map.

Similarly for the content you want to either drop the first element, or the first two.

Alright, looks good.

(defn hiccup->cljxml [h]
  (let [attrs (if (map? (second h))
                (second h)
                {})
        content (if (map? (second h))
                  (rest (rest h))
                  (rest h))]
    {:tag (first h), :attrs attrs, :content content}))

(defn test-hiccup->cljxml []
  (assert= (hiccup->cljxml [:p]) {:tag :p, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:div]) {:tag :div, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:p "hello!"]) {:tag :p, :attrs {}, :content '("hello!")})
  (assert= (hiccup->cljxml [:p {:class "info"} "hello!"])
           {:tag :p, :attrs {:class "info"}, :content '("hello!")}))

(test-hiccup->cljxml)

These tests have already been useful for developing this feature, but the really cool thing about them is that now you can make changes with confidence. This code could already be cleaned up a bit, so let’s do that first. You can always use the tests to make sure you didn’t break anything.

This map? check is being done twice, how about you pull that into the let block and call it attrs?. That already makes this first if short enough that I would just put it on a single line.

Now in this second if there’s this (rest (rest h)), that’s a bit ugly, might be better to just use drop. Now suppose you rewrite this other call to (rest ) as (drop 1), then you can move the drop outside the if. I think that looks a lot better.

(defn hiccup->cljxml [h]
  (let [attrs?  (map? (second h))
        attrs   (if attrs? (second h) {})
        content (drop (if attrs? 2 1) h)]
    {:tag (first h), :attrs attrs, :content content}))

Now let’s try some nesting, I’ll take this paragraph again, but this time add a :strong tag inside. Now the clojure.xml version should contain as content a list with a single map, which represents the :strong tag.

Let’s see what the test says, it seems it’s just putting the hiccup vector inside :content, without converting it. That’s not so hard to fix, let’s here map over hiccup->cljxml so any child elements get converted as well.

Now it’s throwing a different error, it seems there’s a string somewhere that’s being treated as a sequence, and so Clojure is looping over the individual characters, which are then in turn treated as a sequence again, and that’s when things blow up.

What’s happening is we’re mapping over the child elements assuming they’re vectors, but they could also be strings. To fix that let’s make hiccup->cljxml a bit more defensive, it will only convert vectors, anything else just passes through.

Now it should work, but seems we’re still doing something wrong. Ah yes, my mistake, in the test I had a quote inside a quoted list. There, that’s better.

(defn hiccup->cljxml [h]
  (if (vector? h)
    (let [attrs?  (map? (second h))
          attrs   (if attrs? (second h) {})
          content (drop (if attrs? 2 1) h)]
      {:tag (first h), :attrs attrs, :content (map hiccup->cljxml content)})
    h))

(defn test-hiccup->cljxml []
  (assert= (hiccup->cljxml [:p]) {:tag :p, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:div]) {:tag :div, :attrs {}, :content ()})
  (assert= (hiccup->cljxml [:p "hello!"]) {:tag :p, :attrs {}, :content '("hello!")})
  (assert= (hiccup->cljxml [:p {:class "info"} "hello!"])
           {:tag :p, :attrs {:class "info"}, :content '("hello!")})
  (assert= (hiccup->cljxml [:p [:strong "hello"]])
           {:tag :p,
            :attrs {},
            :content '({:tag :strong,
                        :attrs {},
                        :content ("hello")})}))

(test-hiccup->cljxml)

So are we done? Not really. So far we’ve only tested the happy path, but it’s also important to test for things that can go wrong, and to make sure you are getting the errors you would expect to get.

Unfortunately we don’t yet have an assertion for testing if a function throws an error. There’s also not that much that can go wrong here, assuming you pass in valid input, meaning a vector or a string. But what if you don’t? In some cases it will throw a cryptic exception, in others you’ll get a result back but it will be nonsense. Neither is very helpful.

It might be better to reject invalid input outright. By failing early you prevent invalid results from propagating through the system. It also gives you an opportunity to give useful feedback to the programmer.

A good way to do this is with a precondition. These are checks that are performed by Clojure whenever your function is called.

To do this, add a map with a :pre key right after the argument vector. The value is a vector with checks. In this case h needs to be a vector or a string.

Now if you pass in something else, it fails instantly, and you can see from the error what went wrong.

(defn hiccup->cljxml [h]
  {:pre [(or (vector? h) (string? h))]}
  ,,,)

So far we’ve gotten plenty of milage out of that three line testing framework. If you write more tests then you can group them in a test-suite function, and call that from an automated build or CI server.

(defn test-suite []
  (test-hiccup->cljxml)
  (test-other-thing)
  (test-yet-another-thing)
  ,,,)

Now let’s look at some of the shortcomings. A first one is that each test needs to be added manually to the test-suite function, or it won’t run on our build machine. Invetably someone will forget adding their tests, and no-one will know when things break. It would be better if tests were detected automatically across namespaces.

A second problem with this approach is that the test suite always “fails fast”. It stops running at the first failure, so you don’t know if it’s one test failing or a dozen, all you can do is fix one at a time and re-run the tests. It would be better if you could get a nice overview of how which tests passed and which failed.

Which brings us to the third shortcoming: reporting. A good test suite speaks to you. It shows in a single glance if all is fine, or if the world is on fire. It also tells you clearly what the problem is, and where it happened.

Let’s try to implement some of these features. To automatically find all tests you’ll need a bit of metaprogramming.

With ns-interns you can get all the vars of a namespace, in particular it gives you a map, with symbols as keys, and the var itself as the value.

(ns-interns *ns*)
;;=>
;; {assert= #'hiccup2cljxml.core/assert=
;;  test-hiccup->cljxml #'hiccup2cljxml.core/test-hiccup->cljxml
;;  hiccup->cljxml #'hiccup2cljxml.core/hiccup->cljxml}

To find all the vars in the system you can first get all the namespaces with all-ns, then call ns-interns for each, and concatenate the results.

(->> (all-ns)
     (mapcat ns-interns)
     )

Now let’s filter that down so you only keep the functions that have a name starting with “test-“. Each entry in the sequence is a pair, with the first one being the var name as a symbol. Turn that into a string so you can match it with a regular expression.

The test function is there, but so are half a dozen other functions. I guess you’ll need a finer brush for this.

(->> (all-ns)
     (mapcat ns-interns)
     (filter #(->> % first str (re-find #"^test-"))))

;;=> ([test-all-vars #'clojure.test/test-all-vars] [test-var #'clojure.test/test-var] [test-vars #'clojure.test/test-vars] [test-ns #'clojure.test/test-ns] [test-var #'cider.nrepl.middleware.test/test-var] [test-nss #'cider.nrepl.middleware.test/test-nss] [test-vars #'cider.nrepl.middleware.test/test-vars] [test-result #'cider.nrepl.middleware.test/test-result] [test-ns #'cider.nrepl.middleware.test/test-ns] [test-hiccup->cljxml #'hiccup2cljxml.core/test-hiccup->cljxml])

Or you can take a different approach, and use some metadata to make it clear which vars are tests, and which aren’t. Clojure actually has a standard way of doing this, using a :test key in the metadata. This should point at a function which raises an error when the test fails.

The function test-hiccup->cljxml is already such a function, to make it clear that it’s a test, give it :test metadata, with as value the #'test-hiccup->cljxml var. This is similar to what clojure.test does with deftest

(defn
  ^{:test #'test-hiccup->cljxml}
  test-hiccup->cljxml
  []
  ,,,)

With that you can now write a test runner. Start again with all the vars in all the namespaces, but this time keep only the ones with :test metadata. keep is like doing map, and then filtering out any nil values, so you get the :test metadata back for all the vars that have it, ignoring the others.

Now you can loop over the tests to run them one by one. Let’s wrap that in a function called run-all-tests. If you run that it just returns nil, because all the tests pass. Let’s break this test for a second to see if that actually works. Nice!

(defn run-all-tests []
  (doseq [test (->> (all-ns)
                    (mapcat ns-interns)
                    (keep #(-> % second meta :test)))]
    (test)))

(run-all-tests)

Writing tests has become a bit more tedious though, with all that extra metadata stuff, but with a little macro that’s easy to fix.

This is the first defmacro in a Lambda Island episode. Most of the time I try to avoid them and just use functions, which is why I never covered them before, but this is a perfect use case. If you’ve never written a macro yourself then don’t worry too much about the syntax.

In particular dealing with metadata in macros looks a bit different than what you’re used to, and because var is a special form it takes some extra care to correctly do a var lookup, but in the end you can see that essentially all this macro does is expand into a defn form, so defcheck defines a function.

Let’s use it to write a test, just something simple that obviously fails to try it out. Now running the test-a-thing function runs the test, as you would expect.

(defmacro defcheck [name & body]
  `(defn ~(with-meta name {:test `(var ~name)}) []
     ~@body))

(defcheck test-a-thing
  (assert= 5 6))

(test-a-thing)

If you are dealing with macros then macroexpand-1 is your friend, it shows you exactly what the generated code looks like. You only see the code forms themselves though, and not their metadata, but if you look at the var you can see that the test metadata is there.

Now run-all-tests finds the test, which is what we were after.

(macroexpand-1
 '(defcheck test-a-thing
    (assert= 5 6)))
;;=> (clojure.core/defn test-a-thing [] (assert= 5 6))

(meta #'test-a-thing)
;;=> {:test #'user/test-a-thing, :arglists ([]), :line 11, :column 2, :file "*markdown-clojure*", :name test-a-thing, :ns #namespace[user]}

(run-all-tests)
;;=> Unhandled java.lang.AssertionError
;;   Expected (= 5 6) but got (not= 5 6)

Now let’s stop our tests from bailing out as soon as an assertion fails, and while we’re at it let’s try to give better reporting as well.

First add a special var called test-results. It has these “earmuffs”, and it’s marked as dynamic. Dynamic vars are a bit different from regular vars, you use them in combination with binding.

Wrap the code that runs all tests in this binding macro. This is where *test-results* gets its value. Any code that runs inside binding will see this new value, but any code outside of it doesn’t. This way it acts a bit like a global: calls to assert= can access it to add stuff to the *test-results*, but at the same time its scope is limited. Code that runs earlier, later, or on a different thread doesn’t get to tamper with the results.

(declare ^:dynamic *test-results*)

(defn run-all-tests []
  (binding [*test-results* (atom {:pass 0, :fail 0, :assertions 0, :messages []})]
    ,,,))

Now instead of having assert= raise an exception, it updates the *test-results*. It keeps a count of how many assertions were run, how many of them passed, and how many failed. In case of failure it also adds a failure message.

(defn assert= [x y]
  (swap! *test-results* update :assertions inc)
  (if (= x y)
    (swap! *test-results* update :pass inc)
    (do
      (swap! *test-results* update :fail inc)
      (swap! *test-results* update :messages conj
             (str "Expected " (pr-str y) " but got " (pr-str x))))))

For this to work you’ll also have to give the map some initial values for these keys.

(defn run-all-tests []
  (binding [*test-results* (atom {:assertions 0 :tests 0 :pass 0 :fail 0 :messages []})]
    (doseq [test (->> (all-ns)
                      (mapcat ns-interns)
                      (keep #(-> % second meta :test)))]
      (test))))

Let’s pull this repetitive bit out into its own function, there that looks a bit cleaner. Finally let’s also keep track of the total number of tests.

(defn update-report! [& args]
  (apply swap! *test-results* update args))

(defn assert= [x y]
  (update-report! :assertions inc)
  ,,,)

(defn run-all-tests []
  (binding [*test-results* {,,,}]
    (doseq [test (,,,)]
      (update-report! :tests inc)
      (test))))

Finally you’ll need a function that prints out the actual test results. So after all the tests have run, derefernce the *test-results* atom, and pass it to a new print-report function.

Here in the argument list you can destructure the result map, I’ll just copy these keys over from here. First print out any messages, since you’re just looping over them for side effects you could use doseq or run!. Finally print out a little summary: we ran this many tests with this many assertions, and of these so many failed.

(defn- print-report! [{:keys [assertions tests pass fail messages]}]
  (run! #(println % "\n\n") messages)
  (println "Ran" tests "tests containing" assertions "assertions.")
  (println fail "failures."))

(defn run-all-tests []
  (binding [*test-results* (atom {:assertions 0 :tests 0 :pass 0 :fail 0 :messages []})]
    (doseq [test (->> (all-ns)
                      (mapcat ns-interns)
                      (keep #(-> % second meta :test)))]
      (update-report! :tests inc)
      (test))
    (print-report! @*test-results*)))

There, for a testing framework that was cobbled up in ten minutes that’s not half bad. Now when you run the tests the results are printed to standard output, so keep an eye on the REPL.

Let’s also see what happens if a test fails. Cool, that works.

There’s still plenty that could be improved here, and at this point maybe it’s time to say it’s been fun and switch over to an existing framework that is a bit more complete and actively maintained, such as clojure.test, which comes bundled with Clojure itself.

What I showed you isn’t all that different from what clojure.test does, and I figured that building something up step by step would be more interesting than me just browsing through the clojure.test code and pointing at things. Instead I would recommend you have a look yourself at the clojure.test source code. It’s only a few hundred lines of code and includes lots of documentation.

Using clojure.test is straightforward as well. Most of the time you’ll only need these four macros: deftest, testing, is, and are

deftest is a drop-in replacement for our own defcheck. That was easy.

Instead of having special purpose assertion macros like assert=, now you can use the is macro, giving it some kind of check that returns true or false. So here that’s simply an equality check. is is smart enough to generate a descriptive failure message if the two values aren’t equal.

With testing you can group multiple assertions together. This way you provide some extra documentation to the person reading your tests, and these descriptions also show up in any failure messages, so you get a bit of extra context about what went wrong.

(deftest test-hiccup->cljxml
  (testing "single tags"
    (is (= (hiccup->cljxml [:p]) {:tag :p, :attrs {}, :content ()}))
    (is (= (hiccup->cljxml [:div]) {:tag :div, :attrs {}, :content ()}))
    (is (= (hiccup->cljxml [:p "hello!"]) {:tag :p, :attrs {}, :content '("bonjour!")})))

  (testing "nested tags"
    (is (= (hiccup->cljxml [:p {:class "info"} "hello!"])
           {:tag :p, :attrs {:class "info"}, :content '("hello!")}))
    (is (= (hiccup->cljxml [:p [:strong "hello"]])
           {:tag :p,
            :attrs {},
            :content '({:tag :strong,
                        :attrs {},
                        :content ("hello")})}))))

If you have a bunch of similar is forms in a row, you can make that a bit more concise with are.

The first two arguments to are together form a “template” you first give it a list of variable names, in this case hiccup and cljxml, followed by an expression that uses those variables. In this case it checks if converting the hiccup to clojure.xml yields the right result.

After that you can put one or more pairs, one for each case you want to test. Here, I’ll align them so it’s a bit more clear. Each pair of values will be filled into the template, and the result checked with is. This version is identical to the previous one, but you get rid of all the clutter, making it easier to add more test cases.

(deftest test-hiccup->cljxml
  (testing "single tags"
    (are [hiccup cljxml] (= (hiccup->cljxml hiccup) cljxml)
      [:p]          {:tag :p, :attrs {}, :content ()}
      [:div]        {:tag :div, :attrs {}, :content ()}
      [:p "hello!"] {:tag :p, :attrs {}, :content '("bonjour!")}))
  ,,,)

To run the tests, you use clojure.test’s version of run-all-tests. Have a look at the REPL again to see the test report.

(clojure.test/run-all-tests)

FAIL in (test-hiccup->cljxml) (hiccup.clj:16)
single tags
expected: (= (hiccup->cljxml [:p "hello!"]) {:tag :p, :attrs {}, :content (quote ("bonjour!"))})
  actual: (not (= {:tag :p, :attrs {}, :content ("hello!")} {:tag :p, :attrs {}, :content ("bonjour!")}))

Ran 1 tests containing 5 assertions.
1 failures, 0 errors.

Leiningen has built-in support for clojure.test. Just make sure your tests are inside the test directory, and then run the tests with lein test.

Boot also supports clojure.test through a third party add-on.

If you’re using Emacs with CIDER then it makes more sense to use CIDER’s test runner. The CIDER manual has a section explaining all it can do.

Introduction to Clojure Testing