List Comprehension with clojure.core/for

A small macro with a lot of power! We start with a quick recap of Clojure’s looping constructs, and then go over the various features and use cases of for, backed by examples taken directly from popular open source projects.

Transcript

The clojure.core namespace is like a great treasure trove. Even experienced Clojure programmers will still find unexplored gems in there, and even familiar functions and constructs may have more to offer than first surmised.

Take the humble for macro, you don’t have to be familiar with for. I’m sure many developers are perfectly content shipping impressive products without reaching for it once. Others may use it many times but superficially, never realizing the full depth of its hidden powers. So this episode is for pros and beginners alike. Pay attention, you might learn more than you bargained for.

But first let’s see where for fits into the bigger picture of Clojure mapping constructs.

	lazy-seq	nil (side effects)
function	map	run!
macro	for	doseq

To process a collection sequentially, one element at a time, Clojure offers four options: map, run!, for, and doseq. That doesn’t mean you need all four. You can pick one from each column, and happily pretend the others don’t exist.

So you could learn map and run!, and forget that for and doseq ever existed, or vice versa. My impression is that in general people prefer to stick to map and doseq, and ignore for and run!. Don’t ask me why. Maybe the syntax provided by the doseq macro is more reminiscent of imperative looping constructs in other languages.

But I’m getting ahead of myself, let’s quickly run through all four to see what they do.

Map takes a function and a sequence, and returns a lazy sequence, applying the given function over each element of the input sequence. So if you increment the numbers from 0 to 4, you get 1 to 5. Easy enough.

(map inc (range 5))
;; => (1 2 3 4 5)

If you replace map with run! then the result becomes nil. That doesn’t seem very useful.

(run! inc (range 5))
;; => nil

However when you replace inc with a side-effectful function like println then you do see something happening. run! will still call the function with each successive element, but then it throws away the result.

(run! println (range 5))
;; 0
;; 1
;; 2
;; 3
;; 4
;; => nil

Could you use map and throw away the result yourself? Not really. map is lazy, meaning it only kicks into gear when the result is used (or as they say in Clojure-land, when the lazy sequence is realized).

(do
  (map println (range 5))
  nil)
;; <nothing gets printed>
;; => nil

for is like map, returning a lazy sequence. It also has several extra tricks up its sleeve, which we’ll get to in a bit.

The big difference is that for is a macro which introduces some new syntax. Now instead of passing a function you give it a vector containing a variable name and a collection, followed by a bit of code which then uses that variable.

(for [x (range 5)]
  (inc x))
;; => (1 2 3 4 5)

Now what for is to map, doseq is to run!.

In other words, doseq is a syntactic construct which does the same thing as run! (although again with some extra functionality built-in, as you’ll see).

This looks a lot like the for version, but like run! this one is eager rather than lazy, and returns nil rather than a sequence.

(doseq [x (range 5)]
  (println x))
;; 0
;; 1
;; 2
;; 3
;; 4
;; => nil

For completeness I should mention mapv, a variant of map which returns a vector, rather than a sequence. There’s no such thing as a “lazy vector”, so mapv always returns a fully realized collection. In other words: while map is lazy, mapv is eager.

This means that you could use mapv for side effects, similar to run! or doseq, but it’s certainly not recommended, as it would be quite surprising to someone reading your code, including your later self.

mapv ;; function | eager | returns vector

(mapv inc (range 5))

Now that you know where for fits into the language let’s zoom in and see what tricks it has in store. For this I’ve combed through dozens of open source libraries. Each time I’ll first show a basic example to illustrate a certain feature or use case, and then follow it up with

The first thing we’ll look at is destructuring. Hopefully you’ve come across destructuring before, as this is not an episode about destructuring, but I’ll give you a few examples so you know what I’m talking about. The official Clojure guide on destructuring is a good reference if you want a more in-depth refresher. You can find the link in your show notes.

Say I have this function which splits a URL into its scheme, domain name, and path.

(defn url-parts [url]
  (next (re-matches #"(\w+)://([^/]+)(.*)" url)))

(url-parts "https://lambdaisland.com/episodes")
;; => ("https" "lambdaisland.com" "/episodes")

You could use that result like this, assigning the result to local variable parts, and pulling out each part with nth.

(let [parts (url-parts "https://lambdaisland.com/episodes")]
  {:scheme  (nth parts 0)
   :domain (nth parts 1)
   :path   (nth parts 2)})

But instead you can do something like this using a vector of variable names, so the individual parts get pulled out and assigned to variables in one go. This is known as destructuring. The vector with the symbols scheme, domain, and path is known as a binding form, it specifies how certain local variables will be bound.

(let [[scheme domain path] (url-parts "https://lambdaisland.com/episodes")]
  {:scheme  scheme
   :domain domain
   :path   path})

Or, say I wrap this into a function, so this is now a function which returns a map, and after calling the function I want to pull out specific values from that map.

In this case I destructure using a map as the binding form, specifying the keys I want to pull out.

(defn url-map [url]
  (let [[scheme domain path] (url-parts url)]
    {:scheme  scheme
     :domain domain
     :path   path}))

(let [{:keys [scheme domain path]} (url-map "https://lambdaisland.com/episodes")]
  (str scheme "://" domain "/robots.txt"))

So far for this brief recap of destructuring, I know for many of you this is your bread and butter. The point is that you can use destructuring pretty much anywhere where you assign a local variable, including in for blocks.

(def urls ["http://gaiwan.co/"
           "http://lambdaisland.com/"
           "http://clojureverse.org/"])

(for [{:keys [scheme domain]} (map url-map urls)]
  (str scheme "://" domain "/favicon.ico"))

Here’s an example from Codox by James Reeves, a documentation tool that’s also used to build cljdoc.

This functions takes a sequences of maps, each with information about a namespace. It first groups all namespaces by name, in case the same namespace is declared in more than one file, for instance one clj and one cljs, and then combines each group into a single map again.

The result of group-by is a map. Under the hood this map gets turned into a sequence of “map entries”. These are two element tuples consisting of a key and a value, which you can generally treat as two element vectors, meaning you can destructure them with a vector.

;; codox/src/codox/main.clj:31
(defn- merge-namespaces [namespaces]
  (for [[name namespaces] (group-by :name namespaces)]
    (assoc (first namespaces) :publics (mapcat :publics namespaces))))

Here’s another example from Chestnut, where I’m using for together with into to update a map, converting the keys of the map from strings to keywords.

;; chestnut/src/leiningen/new/chestnut.clj:193
(defn load-props [file-name]
  (with-open [^java.io.Reader reader (clojure.java.io/reader file-name)]
    (let [props (java.util.Properties.)]
      (.load props reader)
      (into {} (for [[k v] props] [(keyword k) v])))))

It’s probably easier to follow along if I pull this into its own function and add a few newlines. Notice how using for with a map creates a sequence of pairs, which are then turned back into a map with into.

(defn keywordize-map-keys [m]
  (into {}
        (for [[k v] m]
          [(keyword k) v])))

You could also do this differently using point-free style. into optionally takes a transducer as its second argument, which you can create by only passing one argument to map.

The function we’re mapping over is created using juxt, to pull out each map entry’s key and value seperately. Finally I compose key together with the keyword function, to convert the string keys to keywords.

So now the map transducer will produce two element vectors, the first element being the map entry’s key, converted to a keyword, the second being the value, unchanged. into finally takes care of turning these pairs back into a map.

I’m aware this style will look unfamiliar to many people, but it’s a perfectly idiomatic way to achieve the same effect, and as your Clojure journey progresses I recommend getting used to both reading and writing this style of code.

(defn keywordize-map-keys2 [m]
  (into {}
        (map (juxt (comp keyword key) val))
        m))

Any binding form that you can use in function arguments or on the left-hand side of a let assignment you can also use inside for, for example you can add type hints. For instance this example from the Crux database tells Clojure that we’re dealing with instance of java.io.File, which helps the Clojure compiler to emit better Java bytecode, eliminating runtime type reflection.

;; crux/src/crux/io.clj:202
(for [^File file files]
  (let [in (DataInputStream. (io/input-stream file))
        cleaner-action (fn []
                         (.close in)
                         (.delete file))
        seq ((fn step []
               (lazy-seq
                (try
                  (when-let [x (nippy/thaw-from-in! in)]
                    (cons x (step)))
                  (catch Exception e
                    (cleaner-action)
                    (if (or (instance? IOException e)
                            (instance? IOException (.getCause e)))
                      nil
                      (throw e)))))))]
    [seq cleaner-action]))

In the examples we’ve seen so far the binding vector passed to for always contained two elements, the first one a variable or destructuring form, and the second one a collection. But you can also pass in multiple collections, each with their own binding form.

Now for will go over every possible combination, so if you take two collections with two elements each, then the result is a sequence with four results. This is in mathematical set theory is known as a “Cartesian Product”.

(for [x (range 2)
      y [:A :B]]
  [x y])
;; => ([0 :A] [0 :B] [1 :A] [1 :B])

You could add even more collections if you wanted to.

(for [x (range 2)
      y [:A :B]
      z [:+ :-]]
  [x y z])
;; => ([0 :A :+]
;;     [0 :A :-]
;;     [0 :B :+]
;;     [0 :B :-]
;;     [1 :A :+]
;;     [1 :A :-]
;;     [1 :B :+]
;;     [1 :B :-])

Looping over multiple collections can be useful for processing all cells of a 2-dimensional matrix. Say you have a representation of a chess board, and you want to turn that into a list of positions of the pieces, then pass a range for each dimension and you’re set.

This example turns this square board into a list of piece positions.

(def board
  [[:R :N :B :K :Q :B :N :R]
   [:P :P :P :P :P :P :P :P]
   [:_ :_ :_ :_ :_ :_ :_ :_]
   [:_ :_ :_ :_ :_ :_ :_ :_]
   [:_ :_ :_ :_ :_ :_ :_ :_]
   [:_ :_ :_ :_ :_ :_ :_ :_]
   [:p :p :p :p :p :p :p :p]
   [:r :n :b :k :q :b :n :r]])

(for [x (range 8)
      y (range 8)]
  [(str (get "ABCDEFGH" y) (inc x))
   (get-in board [x y])])

Or here’s another fun, card-game inspired example. Based on suites and ranks I first contstruct a deck of 52 cards, and then shuffle it and hand a player a hand of seven cards.

Note that here I’m using for as the first argument of a “thread-last” macro, this is not uncommon, if you want to apply sequence operations afterwards. But if you’re processing the result of for with filter or map then chances are there’s a better way, which we’ll get to.

(->> (for [suite ["♠" "♥" "♦" "♣"]
           rank  (concat (range 2 11)
                         ["A" "J" "Q" "K"])]
       (str rank suite))
     shuffle
     (take 7))
;; => ("3♠" "A♥" "Q♣" "J♠" "6♦" "7♣" "2♦")

From my own experience and looking at the examples I found I’d say that for most people these strictly combinatorial problems are perhaps not that common, but what is fairly common is processing nested collections, returning a flat result.

For each collection you can refer back to binding forms established by previous collection. Here’s an example of what you can do with that:

First I go over all namespaces known to Clojure, and for each namespace I look at every public var. For each var I take its metadata keys, and return them one by one. The result is a long sequence of keywords, which I then stick into a map to remove the duplicates.

You can’t really call this a cartesian product anymore, instead each succesive binding is used to further drill down.

(into #{}
      (for [ns  (all-ns)
            var (vals (ns-publics ns))
            k   (keys (meta var))]
        k))
;;=>
#{:redef :no-doc :private :protocol :added :ns :name :special-form :file
  :potemkin/body :static :inline-arities :skip-wiki
  :nrepl.middleware/descriptor :column :author :const :dynamic :line :macro
  :deprecated :declared :url :style/indent :tag :arglists :see-also :doc :forms
  :inline}

Let’s go back to the chess board example. It’s only been a minute and I’ve already forgotten what this actually returns. Let’s add a let so this is a bit more readable.

(for [x (range 8)
      y (range 8)]
  (let [pos   (str (get "ABCDEFGH" y) x)
        piece (get-in board [x y])]
    [pos piece]))

So far so good, but for already has let’s capabilities built in, I just need to move this into the for vector, and use a keyword for :let.

(for [x (range 8)
      y (range 8)
      :let [pos   (str (get "ABCDEFGH" y) x)
            piece (get-in board [x y])]]
  [pos piece])

So this is interesting, so far on the left hand side we always had a variable or other binding form, but you can also use a few keywords there, namely :let, :while, and :when. With these in hand for turns into quite a powerful list comprehension construct.

Say you are only interested in the position of the kings and queens, you could use :when to limit the results. This expression is checked on every iteration of the loop, and only if it evaluates to true is the body executed and the result added to the resulting sequence.

(for [x    (range 8)
      y    (range 8)
      :let [pos   (str (get "ABCDEFGH" y) x)
            piece (get-in board [x y])]
      :when (#{:K :Q :k :q} piece)]
  [pos piece])

So that’s :let and :when, the third keyword that for understands is :while. Just like with :when the predicate is checked on every iteration. As soon as it returns false, for stops and returns. This can be especially useful when dealing with infinite lazy sequences.

Here I’ve rewritten this code to use an infinite range. It uses modulo to turn that into a row and column, and it continues as long as it finds pieces. In other words: it stops when it falls off the board.

Then :when filters out the kings and queens, and finally I calculate the position in another :let block. You could do this in the first :let block, but then the position string would be created for every piece. By putting it at the end it’s only done for the kings and queens.

(for [idx    (range)
      :let   [x     (mod idx 8)
              y     (quot idx 8)
              piece (get-in board [x y])]
      :while piece
      :when  (#{:K :Q :k :q} piece)
      :let   [pos (str (get "ABCDEFGH" y) x)]]
  [pos piece])

It’s pretty neat that you can stack these different modifiers like this, for a language that is otherwise very regular and lean in syntax this is a rare exception, where you are given a sort of DSL to express linearly what otherwise would have been deeply nested forms. It’s also a tribute to the power of LISP, where a simple macro can deliver new and beautifully concise syntax.

You can even add more collections to traverse after using some of the keywords. Take the following example: this finds all metadata keys used in Clojure’s built-in namespaces, which have a string as their associated value.

It loops over all namespaces, gets the namespace name and checks if it starts with the string “clojure”. Then it loops over all the vars in each namespace, for each var gets the metadata, which is a map, so it destructures each map-entry to its key and value, and finally checks if the value is a string.

Prett cool, right?

(into #{}
      (for [ns    (all-ns)
            :let  [n (str (ns-name ns))]
            :when (str/starts-with? n "clojure")
            var   (vals (ns-publics ns))
            [k v] (meta var)
            :when (string? v)]
        k))

Most of the time it doesn’t get as convoluted as this, but it’s good to know just how freely you can stack and combine these different options.

Let’s look at another real-world example. Here’s a function from cljdoc, with a for expression which uses destructuring and a :let. Here you see a common use case for the for macro: Hiccup syntax.

When writing HTML templates this version using for looks much nicer than trying to do the equivalent with map.

Both the Hiccup library that you would use on the server side, as well as ClojureScript libraries like Reagent or Sablono all know how to deal with sequences inside hiccup, so you also don’t need to awkwardly flatten things into vectors with into.

;; cljdoc/render/api.clj:104
(defn namespace-list [{:keys [current]} namespaces]
  (let [base-params (select-keys (first namespaces) [:group-id :artifact-id :version])
        namespaces  (ns-tree/index-by :namespace namespaces)]
    [:div.mb4
     (layout/sidebar-title "Namespaces")
     [:ul.list.pl2
      (for [[ns level _ leaf?] (ns-tree/namespace-hierarchy (keys namespaces))
            :let [style {:margin-left (str (* (dec level) 10) "px")}]]
        [:li
         [:a.link.hover-dark-blue.blue.dib.pa1
          {:href (routes/url-for :artifact/namespace :path-params (assoc base-params :namespace ns))
           :class (when (= ns current) "b")
           :style style}
          (->> (ns-tree/split-ns ns)
               (drop (dec level)))]])]]))

There is something to watch out for though, an issue you can quite easily run into specically when using Reagent and Re-frame, caused by for’s laziness.

Here’s a basic example to illustrate the problem. This component takes a list of user ids, and renders a table of their names and email addresses, which it both gets through a re-frame subscription.

These subscriptions are what Reagent calls “reactions”. Reagent will track when they are dereferenced, so that when a user’s email address changes, the right components are updated.

The problem here is that by the time user-table finishes, the dereferencing hasn’t happened yet, because for is lazy. By the time the sequence returned by for is realized we are no longer inside user-table, so reagent won’t know this component relies on these name and email subscriptions.

(defn user-table [ids]
  [:table
   [:thead
    [:tr [:th "Name"] [:th "Email"]]]
   [:tbody
    (for [id ids]
      [:tr
       [:td @(re-frame/subscribe [:user/name id])]
       [:td @(re-frame/subscribe [:user/email id])]])]])

There are several ways around this, you could use doall, effectively forcing the sequence to be realized before returning, or you can use into, generating Hiccup that doesn’t contain the lazy sequence.

[:tbody
 (doall
  (for [id ids] 
    [:tr
     [:td @(re-frame/subscribe [:user/name id])]
     [:td @(re-frame/subscribe [:user/email id])]]))]

(into [:tbody]
      (for [id ids]
        [:tr
         [:td @(re-frame/subscribe [:user/name id])]
         [:td @(re-frame/subscribe [:user/email id])]]))

Finally what I’ll sometimes do is use backticks and “~@” or “unquote-splice”. I find syntax quotes can be quite nice for working with Hiccup, assuming everyone on your team is familiar with them.

`[:tbody
  ~@(for [id ids]
      [:tr
       [:td @(re-frame/subscribe [:user/name id])]
       [:td @(re-frame/subscribe [:user/email id])]])]

Once again I thought this would be a very short episode, but it turns out that no matter which part of Clojure you zoom into, there’s always a lot to talk about. I hope this episode did deliver something useful to pros and beginners alike. Let us know! We’re always happy to hear from our learners!

Finally remember that all of the syntax we talked about is also available with doseq, including the :let, :when, and :while keywords. Talking about getting two for the price of one!

Show Notes

browse source code

Cheat sheet

(for [<binding-form> <coll-expr> ...
      :let [<binding> <expression>
            ...]
      :when <predicate>
      :while <predicate>]
  <body>)