This episode is for subscribers only. Sign Up or Log in to watch it.

35. Generative Testing with test.check

Published 21 September 17

Generative testing, also called Property Based testing, is a powerful technique able to expose some of the most obscure bugs. You’ll learn how to create and compose generators, and how to define properties that together can verify many aspects of your code.

Show notes

clojure.test.check is a library for testing your code with lots of random data. You first specify certain properties of your program, and test.check will generate data and verify that the properties hold. Let’s see how that looks in practice.

First take a regular unit test. This one tests the normalize function from lambdaisland.uri by feeding it a string, and checking the result.

(ns lambdaisland.uri.normalize-test
  (:require [lambdaisland.uri :as uri]
            [lambdaisland.uri.normalize :as n]
            [clojure.test :refer [deftest testing is are]]))

(deftest normalize-test
  (is (= (str (n/normalize (uri/uri "http://foo.bar?q=💃")))
         "http://foo.bar?q=%F0%9F%92%83")))

That’s a useful test, but that test alone isn’t enough to inspire confidence, so you go on and add more and more of these test cases, each time coming up with some data that tests a specific aspect of the implementation. This works alright, but coming up with good test cases is limited by your patience and imagination.

It’s not just that you won’t come up with enough test cases, it’s also that you will tend to test “reasonable” data, missing many edge cases.

(deftest normalize-test
  (are [in out] (= (str (n/normalize (uri/uri in))) out)
    "http://foo.bar?q=💃" "http://foo.bar?q=%F0%9F%92%83"
    "http://foo.bar/./foo/../bar/baz" "http://foo.bar/bar/baz"
    "HTTP://foo.bar" "http://foo.bar"
    "http://foo.BAR" "http://foo.bar"
    "" ""
    "#" "#"
    ":::Av?%f" ":::Av?%25f"))

Wouldn’t it be great if you could generate test cases on the fly, testing them by the hundreds or thousands?

Broad overview of the API

(require '[clojure.test.check              :refer [quick-check]]
         '[clojure.test.check.clojure-test :refer [defspec]]
         '[clojure.test.check.properties   :refer [for-all for-all*]]

         ;; ~70 generators and generator combinators
         '[clojure.test.check.generators :as gen])

Generators

There really is a lot cramped together in the generators namespace, which makes the API docs a bit difficult to navigate. To get a better sense of what’s there I’ve categorized the various functions. If you’re ready to dig deeper I encourage you to read up on the functions under “Combinators” and “Modifiers”.

Scalars

any, any-printable, simple-type, simple-type-printable,

boolean