Generative Testing with test.check

Subscribe to gain access to this episode

Subscribe Login

or try out one of our freebies instead.

Generative testing, also called Property Based testing, is a powerful technique able to expose some of the most obscure bugs. You’ll learn how to create and compose generators, and how to define properties that together can verify many aspects of your code.

Transcript

clojure.test.check is a library for testing your code with lots of random data. You first specify certain properties of your program, and test.check will generate data and verify that the properties hold. Let’s see how that looks in practice.

First take a regular unit test. This one tests the normalize function from lambdaisland.uri by feeding it a string, and checking the result.

(ns lambdaisland.uri.normalize-test
  (:require [lambdaisland.uri :as uri]
            [lambdaisland.uri.normalize :as n]
            [clojure.test :refer [deftest testing is are]]))

(deftest normalize-test
  (is (= (str (n/normalize (uri/uri "http://foo.bar?q=💃")))
         "http://foo.bar?q=%F0%9F%92%83")))

That’s a useful test, but that test alone isn’t enough to inspire confidence, so you go on and add more and more of these test cases, each time coming up with some data that tests a specific aspect of the implementation. This works alright, but coming up with good test cases is limited by your patience and imagination.

It’s not just that you won’t come up with enough test cases, it’s also that you will tend to test “reasonable” data, missing many edge cases.

(deftest normalize-test
  (are [in out] (= (str (n/normalize (uri/uri in))) out)
    "http://foo.bar?q=💃" "http://foo.bar?q=%F0%9F%92%83"
    "http://foo.bar/./foo/../bar/baz" "http://foo.bar/bar/baz"
    "HTTP://foo.bar" "http://foo.bar"
    "http://foo.BAR" "http://foo.bar"
    "" ""
    "#" "#"
    ":::Av?%f" ":::Av?%25f"))

Wouldn’t it be great if you could generate test cases on the fly, testing them by the hundreds or thousands?

Show Notes

Broad overview of the API

(require '[clojure.test.check              :refer [quick-check]]
         '[clojure.test.check.clojure-test :refer [defspec]]
         '[clojure.test.check.properties   :refer [for-all for-all*]]

         ;; ~70 generators and generator combinators
         '[clojure.test.check.generators :as gen])

Example property-based test

(ns lambdaisland.sorted-test
  (:require [clojure.test.check.generators :as gen]
            [clojure.test.check.properties :as prop]
            [clojure.test.check.clojure-test :refer [defspec]]))

(def vector-of-int (gen/vector gen/int))

(defspec first-cant-be-more-than-last-test
  (prop/for-all [list vector-of-int]
    (let [sorted (sort list)]
      (or (empty? list) (<= (first sorted) (last sorted))))))

Generators

There really is a lot cramped together in the generators namespace, which makes the API docs a bit difficult to navigate. To get a better sense of what’s there I’ve categorized the various functions. If you’re ready to dig deeper I encourage you to read up on the functions under “Combinators” and “Modifiers”.

Types of properties

Most properties fall in these categories. When looking to test a function, go over this list to get ideas for properties to check.

Round Trip

Do an operation and its inverse.

Alternative Implementation

Verify against a reference implementation.

Same Destination, Different Route

Re-order operations without changing the result.

Idempotence

When doing something twice is the same as doing it once.

Invariants (size, contents, consistency)

Some things are always true. Some things never change.

“Hard to do, easy to verify”

I don’t know what you did there, but I can see that it’s right.