47. Interceptors, part 1, concepts
FreebiePublished 26 February 19
The Interceptor pattern was originally introduced by Pedestal, but has since been adopted by several other projects in the Clojure world. It’s an extremely useful design tools to be familiar with, and while they may seem a bit strange at first, Interceptors are suprisingly straightforward.
This episode introduces the interceptor concept, including the context map, queue, and stack. Clojure’s persistent queues are explored, and to round off we look at how interceptors deal with error handling.
Namespaces
Cheat sheet
Interceptor:
{:name ::my-interceptor
:enter (fn [ctx] ctx)
:leave (fn [ctx] ctx)
:error (fn [ctx exception] ctx)}
Enqueue/execute
(require '[lambdaisland.interceptor :refer [enqueue execute]])
(-> {:initial :context}
(enqueue [interceptor-1 interceptor-2])
(execute))
Links
Interceptors are a programming pattern first introduced by the Pedestal library, where it is used to implement HTTP request handling. Since then there have been several independent implementations. re-frame uses a version of interceptors for its event handling, Finnish Clojure shop Metosin released a highly optimized version called Sieppari, Eric Normand came up with some variations on the theme in his own implementation.
Despite that interceptors have seen limited adoption outside of a few narrow use cases tied to specific libraries. This is a shame, because interceptors are uniquely suitable for modelling all kinds of processes, including dynamic dispatching and asynchrony.
In this episode I’ll first explain interceptors in their own right using a simplified implementation specifically written to help demonstrate the concepts involved. Once you’ve gotten a good grasp of the fundamentals I’ll go over some of the specific features in Pedestal’s implementations.
To follow along, clone the GitHub repository for this episode, you can find the
link in the show notes, and then open the interceptor-demo
namespace, and
start a REPL in your editor of choice.
I’ve already included the lambdaisland.interceptor
namespace, pulling in all
public functions. The first two, enqueue
and execute
, are really all you
need, but the others will come in handy to see step by step how things work
under the hood.
(ns lambdaisland.interceptor-demo
(:require [lambdaisland.interceptor :refer [enqueue
execute
process-enter
process-error
process-leave
enter-1
leave-1
error-1
ctx-summary]]))
Let’s look at an example to get a birds eye view of what things look like.
When dealing with interceptor there’s always a single map called the context
that gets passed around throughout the process. This snippet starts with a
context
map containing a :count
of 0
. It then enqueues two interceptors,
the first will increases the counter by 1, and second by 10. It then calls
execute
on the context, so the interceptors actually run. The end result is
again the context map, but updated by the two interceptors.
(-> {:count 0}
(enqueue [{:enter (fn [ctx]
(update ctx :count + 1))}
{:enter (fn [ctx]
(update ctx :count + 10))}])
execute)
;;=>
{:count 11,
:lambdaisland.interceptor/queue <-()-<,
:lambdaisland.interceptor/stack ()}
So these two maps in enqueue
are what we call interceptors. They can
optionally have a :name
, as well as :enter
, :leave
, and :error
functions.
These different functions are executed at different parts of the process. First
each interceptor’s :enter
functions is called, passing the context from one to
the next. Once execute
has processed all :enter
functions it will shift into
reverse, calling each interceptor’s :leave
function in reverse order.
If an error occurs while processing :enter
or :leave
, then execute
will
switch to a different track, calling the :error
function of any interceptor that
still needs to leave.
Don’t worry if that still sounds a little abstract, I’ll run you through the process step by step so you can see for yourself what’s happening.
{:name ::my-first-interceptor
:enter (fn [ctx] ctx)
:leave (fn [ctx] ctx)
:error (fn [ctx ex] ctx ex)}
Let’s take a step back and drop the call to execute
. I’ll also give these
interceptors names so you can identify them. The first one is called :add-1
,
the second :add-10
.
Next I’m enqueueing these two interceptors onto the context map, but I’m not yet executing them. I’m just setting up the battle plan, adding these two interceptors to the context, so I can execute them later on.
(-> {:count 0}
(enqueue [{:name :add-1
:enter (fn [ctx]
(update ctx :count + 1))}
{:name :add-10
:enter (fn [ctx]
(update ctx :count + 10))}]))
;; =>
;; {:count 0,
;; :lambdaisland.interceptor/queue
;; <-({:name :add-1,
;; :enter #function[lambdaisland.interceptor-demo/eval13368/fn--13369]}
;; {:name :add-10,
;; :enter #function[lambdaisland.interceptor-demo/eval13368/fn--13371]})-<}
Now the context has gotten an extra key: :lambdaisland.interceptor/queue
. The
printed representation has these ASCII art arrows at the front and back,
indicitating that this is a Clojure PersistentQueue
. Queues don’t get the same
attention that maps, vectors, and sets get, but they are just as much a part of
Clojure’s immutable data structures.
A queue is very similar to a Clojure list, the difference is that with a list elements are added and removed from the front, so they behave as a kind of stack, whereas with a queue elements are added to the back, and removed from the front. You can also say that lists are “Last in first out” or LIFO, whereas queues are “First in first out” or FIFO.
(-> {:count 0}
(enqueue [{:name :add-1
:enter (fn [ctx]
(update ctx :count + 1))}
{:name :add-2
:enter (fn [ctx]
(update ctx :count + 10))}])
:lambdaisland.interceptor/queue
type)
;;=> clojure.lang.PersistentQueue
Unlike vectors or maps there’s no reader syntax for creating queue literals,
instead you start with the empty queue, and add elements onto it with conj
.
The arrows at the front and back show the direction in which the elements move
through the queue. Each call to conj
inserts an element at the tail end, and
each call to pop
removes an element from the front. pop
returns the new
queue, to get the element at the front just call first
.
(-> clojure.lang.PersistentQueue/EMPTY
(conj :a) ;; => <-(:a)-<
(conj :b) ;; => <-(:a :b)-<
(conj :c) ;; => <-(:a :b :c)-<
(pop) ;; => <-(:b :c)-<
(conj :d) ;; => <-(:b :c :d)-<
(conj :e) ;; => <-(:b :c :d :e)-<
(pop) ;; => <-(:c :d :e)-<
(pop) ;; => <-(:d :e)-<
(pop) ;; => <-(:e)-<
(pop) ;; => <-()-<
)
Ok so you have a context including a queue of interceptors, let’s start executing some of them.
The execute
function is pretty straightforward, it calls process-enter
,
process-leave
, and process-error
, corresponding with the three stages that
execute
goes through.
(defn execute
"Execute the context.
This assumes interceptors have been enqueued with [[enqueue]]. It will run
through the `:enter` stage of all interceptors in the queue, and then through
their `:leave` stage in reverse order.
If at some point an exception is raised, then it will skip the rest of the
`:enter` or `:leave` stage, and instead run the `:error` stage of all
interceptors that still had to leave."
[context]
(-> context
process-enter
process-leave
process-error))
Each of these you can further split into individual steps. For instance
process-enter
processes the :enter
stage of all interceptors in the queue by
recursively calling enter-1
, passing the context from one to the next. The
rest of the code here is just error and boundary checking.
Knowing this you can call enter-1
directly to see what things look like after
entering the first interceptor.
process-leave
and process-enter
similarly call leave-1
and error-1
, so
you can call these fine grained functions from the REPL to see step by step what
happens.
(defn process-enter
"Process the `:enter` stage of all interceptors on the queue.
Run through all interceptors on the queue, executing their `:enter` stage and
moving them from the queue to the stack, until the queue is empty or an error
occured."
[{::keys [queue] :as context}]
(if (seq queue)
(let [new-context (enter-1 context)]
(if (::error new-context)
new-context
(recur new-context)))
context))
Let me first refactor this a bit. I’ll create a helper function so I can easily
create a few interceptors to test with. The interceptor adds one amount to the
count in the :enter
stage, and another amount in the :leave
stage. That way
if I pick my numbers right it’s easy to see which parts have executed and which
haven’t.
(defn make-interceptor [x y]
{:name (keyword (str "add-" x "-" y))
:enter (fn [ctx]
(update ctx :count + x))
:leave (fn [ctx]
(update ctx :count + y))})
Now add them both to the queue, ready to be executed. The count so far is still zero, since none of the interceptors have actually run.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)]))
;;=>
;; {:count 0,
;; :lambdaisland.interceptor/queue
;; <-({:name :add-1-2,
;; :enter #function[lambdaisland.interceptor-demo/make-interceptor/fn--13426],
;; :leave #function[lambdaisland.interceptor-demo/make-interceptor/fn--13428]}
;; {:name :add-10-20,
;; :enter #function[lambdaisland.interceptor-demo/make-interceptor/fn--13426],
;; :leave
;; #function[lambdaisland.interceptor-demo/make-interceptor/fn--13428]})-<}
This is a bit noisy though. To better see what’s going on I’ve added a
ctx-summary
function, which replaces the interceptor maps with just their
names, and it replaces the :lambdaisland.interceptor/queue
key with just
:queue
. Now you can see at a glance what’s in the queue.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)])
ctx-summary)
;;=> {:count 0, :queue <-(:add-1-2 :add-10-20)-<}
Time to process the first :enter
stage using enter-1
. Now several things
have happened. The :count
has increased by 1, so it seems like the :enter
stage of the first interceptor has run. This interceptor has now been removed
from the queue, instead added onto a list called “stack”.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)])
enter-1
ctx-summary)
;;=> {:count 1, :queue <-(:add-10-20)-<, :stack (:add-1-2)}
Call enter-1
again and now the second interceptor’s :enter
stage has
increased the :count
to eleven. The second interceptor also got moved from the
queue to the stack, so the queue is now empty, while the stack contains both
interceptors, but in reverse order, since stacks are “first in last out”.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)])
enter-1
enter-1
ctx-summary)
;;=> {:count 11, :queue <-()-<, :stack (:add-10-20 :add-1-2)}
Now that the queue is empty, the :leave
stage will kick in, so let’s call
leave-1
and see what happens.
Rather than processing the queue, the leave
stage processes the stack, popping
interceptors off the stack, and running their :leave
functions.
Notice that the last interceptor to :enter
was the first to :leave
. They’re
kind of like sandwiches in that way, and while they’re quite different from Ring
middleware, you can see that in this sense they’re a little similar, since each
interceptor can “wrap” the interceptors that follow.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)])
enter-1
enter-1
leave-1
ctx-summary)
;;=> {:count 31, :queue <-()-<, :stack (:add-1-2)}
Finally run the :leave
stage of the last interceptor. Now all interceptors have
finished executing, the stack is empty, and the count is 33.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)])
enter-1
enter-1
leave-1
leave-1
ctx-summary)
;;=> {:count 33, :queue <-()-<, :stack ()}
This is the same result as if you would have called execute
directly, running
through the whole interceptor process.
(-> {:count 0}
(enqueue [(make-interceptor 1 2)
(make-interceptor 10 20)])
execute
ctx-summary)
;;=> {:count 33, :queue <-()-<, :stack ()}
So far so good, now let’s look at some of the more interesting things you can do with interceptors. One interesting thing to note is that the full context is available to each interceptor, including the queue and stack, and so both can be manipulated. This means for instance that an interceptor can queue extra interceptors.
In this example I’m starting out with just a single interceptor, but this
interceptor calls enqueue
again to add ten more interceptors to the queue.
(-> {:count 0}
(enqueue [{:name :enqueue-more
:enter
(fn [ctx]
(enqueue ctx (repeat 10 (make-interceptor 1 0))))}])
enter-1
ctx-summary)
This is commonly used in web applications to do routing. A router interceptor will inspect the request, and enqueue appropriate interceptors to handle the given URL and request method.
(-> {:request {:uri "/hello"}}
(enqueue [router])
execute)
Another thing you can do with the queue is empty it (or simply remove it), to
stop any interceptors left in the queue from executing. This way the :enter
stage finishes, and :leave
kicks in, calling :leave
on any interceptors left
in the stack. This is common enough that Pedestal has a helper function for it
called terminate
.
(defn terminate [ctx]
(dissoc ctx :lambdaisland.interceptor/queue))
Can you follow what’s going on here? The :enqueue-more
interceptor uses its
:enter
stage to enqueue a hundred identical interceptors. Each of them
increases the count by one, unless the count is already greater than twenty,
then it calls terminate
to remove the remainder of the queue, effectively
cutting any remaining processing of the :enter
stage short.
(-> {:count 0}
(enqueue [{:name :enqueue-more
:enter
(fn [ctx]
(enqueue ctx (repeat 100
{:name :add-1
:enter
(fn [ctx]
(if (> (:count ctx) 20)
(terminate ctx)
(update ctx :count inc)))})))}])
execute)
Now let’s see what happens when an error occurs. I’ll first create this interceptor, aptly called “boom!” which simply throws an exception when its enter stage gets called.
(def boom!
{:name :BOOM!
:enter (fn [ctx]
(throw (ex-info "Oops!" {:very :sorry})))})
The handle-error
interceptor doesn’t have an :enter
or :leave
function,
but instead has an :error
function. Note that unlike :enter
and :leave
this one takes two arguments, the second being the caught exception.
This error handler interceptor will flip the sign of the count, so we can see that something went wrong.
(def handle-error
{:name :handle-error
:error (fn [ctx ex]
(update ctx :count * -1))})
Next I’ll enqueue the error handler first. Then I’ll put some regular interceptors that increase the count, then the interceptor that blows up, and finally another count interceptor.
(-> {:count 0}
(enqueue [handle-error
(make-interceptor 1 2)
(make-interceptor 10 20)
boom!
(make-interceptor 100 200)])
)
Ok, now let’s walk through this step by step to see what happens. First run the
:enter
stage of the first three interceptors. So far things are
straightforward, the first didn’t have an :enter
stage, the second and third
increased the :count
to eleven. Now the exploding interceptor is up next.
(-> {:count 0}
(enqueue [handle-error
(make-interceptor 1 2)
(make-interceptor 10 20)
boom!
(make-interceptor 100 200)])
enter-1
enter-1
enter-1
ctx-summary)
;;=>
{:count 11,
:queue <-(:BOOM! :add-100-200)-<,
:stack (:add-10-20 :add-1-2 :handle-error)}
Now run the next one. You’ll notice two things: the queue is gone, and there’s
an :error
key added to the context.
{:count 11,
:stack (:BOOM! :add-10-20 :add-1-2 :handle-error),
:error
{:stage :enter,
:interceptor :BOOM!,
:exception-type :clojure.lang.ExceptionInfo,
:very :sorry}}
At this point execute
will unwind the stack, just like when processing the
leave
stage, but instead of calling :leave
, it will call each interceptor’s
:error
key.
The same happens if an exception happens during the :leave
stage. Any
remaining interceptors on the stack will have their :error
stage called. Note
that an exception in an :enter
stage can be handled by an :error
function on
the same interceptor, whereas if the exception happens in a :leave
function,
then the current interceptor’s :error
handler doesn’t run, because the
interceptor is no longer on the stack.
(-> {:count 0}
(enqueue [handle-error
(make-interceptor 1 2)
(make-interceptor 10 20)
boom!
(make-interceptor 100 200)])
process-enter
process-error
ctx-summary)
So far I’ve stuck to my own minimalist implementation to demonstrate the basic concepts involved in dealing with interceptors. It’s a great implementation to study, it’s only a hundred-odd lines of straightforward code, but it does lack some of the features that you get with Pedestal, key among them the ability to “go async”.
I’ll explore Pedestal’s interceptors in detail in the next episode, stay tuned!