On Extensibility
by Laurence Chen
For a long time, I had a misunderstanding about Clojure: I always thought that the extensibility Clojure provides was just about macros. Not only that, but many articles I read about Lisp emphasized how Lisp’s macros were far more advanced than those in other languages while also extolling the benefits of macros—how useful it is for a language to be extensible. However, what deeply puzzled me was that the Clojure community takes a conservative approach to using macros. Did this mean that Clojure was less extensible than other Lisp languages?
Later, I realized that my misunderstanding had two aspects:
First, Clojure has at least two extension mechanisms: Macros and Protocols. When using the Clojure language, we often do not clearly distinguish between core syntax and core library functions. In other words, if the extensible parts are core library functions, the experience for future users is almost the same as extending core syntax. More specifically, many parts of Clojure’s core library are constructed using Protocol/Interface syntax. This syntax serves as the predefined extension points in the core library, meaning that core library functionality can also be extended.
Second, I used to mix up “extensibility” and “extensibility mechanisms.” I always focused on “Oh, I discovered another language, database, or software that supports plugins. That’s great! It has an extensibility mechanism, so it can be extended.” However, an extensibility mechanism is just a means to achieve extensibility. But what exactly is extensibility? What problems does it solve, and what benefits does it bring? I never really thought through these questions.
Here is a proposed definition of extensibility:
Given a module or a segment of code developed based on certain external assumptions, when these external assumptions change, without modifying the existing code, the behavior of the module can be enhanced or altered through a predefined mechanism, allowing it to adapt to new requirements.
According to this definition, the benefits of extensibility are:
- Cost savings. If no modifications are needed, there is no need to worry about breaking existing functionality or regression issues.
- Reduced complexity. The ability to extend or modify a module’s behavior through predefined mechanisms eliminates the need to copy entire modules and make modifications, saving a significant amount of code.
- Empowering users. Even though the module has already been developed, it can still be modified. This is particularly useful when module developers and users belong to different organizations or teams, as it provides great flexibility, allowing users to self-serve.
Next, let’s look at some real-world examples to better understand extensibility in practice.
Macro
Let’s first examine some common built-in Macros:
->: Transforms linear syntax into nested parentheses, effectively introducing a new DSL (domain-specific language).comment: Ignores a block of code.with-open: Grants a block of code access to a resource that is automatically closed when leaving the block.with-redefs: Temporarily redefines global variables within a block of code.with-in-str: Temporarily binds*in*to a specificStringReaderwithin a block of code.
Macros can be roughly categorized into two types:
- Non with-style Macros
- With-style Macros
Non with-style Macros
These Macros typically accept arguments in the form of & body, internally parsing body and transforming its evaluation strategy.
For example, consider core.async/go:
(go
(println "Before")
(<! (timeout 1000))
(println "After"))
The go Macro transforms body into a state machine to execute it asynchronously. It doesn’t just wrap a block of code but actually rewrites it.
The code passed as an argument to these Macros often introduces new syntax or semantics, effectively extending the Clojure language itself by adding new DSLs.
With-style Macros
In contrast, some Macros accept arguments in the form of a b c & body and internally reference ~@body. These Macros do not dissect the statements inside body; instead, they inject additional processing before or after body executes. Because they preserve the original structure of body, they are particularly suited for resource management, context setting, and similar scenarios.
The embedkit library contains an inspiring with-style Macro that treats authentication state as a form of context.
with-refresh-auth: Refresh the authentication state and retry the request if the request fails with a401error.
(defmacro with-refresh-auth [conn & body]
`((fn retry# [conn# retries#]
(Thread/sleep (* 1000 retries# retries#)) ; backoff
(try
~@body
(catch clojure.lang.ExceptionInfo e#
(if (and (= 401 (:status (ex-data e#)))
(instance? clojure.lang.Atom conn#)
(< retries# 4))
(do
(reset! conn# (connect @conn#))
(retry# conn# (inc retries#)))
(throw e#)))))
~conn
0))
;; Use with-refresh-auth to wrap the do-request
(defn mb-request [verb conn path opts]
(with-refresh-auth conn
(do-request (request conn verb path opts))))
Protocol
Many Clojurians, including myself, have struggled to grasp when to use Protocols—they can feel abstract and difficult to apply. The best explanation I’ve found is from ask.clojure.org:
Protocol functions are better as SPI to hook in implementations than in API as functions consumers call directly.
If, like me, you don’t immediately grasp what an SPI (service provider interface) is, studying the buddy-auth library can help.
buddy-auth is a commonly used authentication library for web applications. Users can extend it by adding new authentication mechanisms without modifying its source code.
To define an authentication mechanism, one must implement the IAuthentication Protocol using reify.
For example, http-basic-backend is a basic authentication mechanism that implements IAuthentication:
(defn http-basic-backend
[& [{:keys [realm authfn unauthorized-handler] :or {realm "Buddy Auth"}}]]
{:pre [(ifn? authfn)]}
(reify
proto/IAuthentication
(-parse [_ request]
(parse-header request))
(-authenticate [_ request data]
(authfn request data))
...
)
When using buddy-auth, the wrap-authentication middleware is added to the Ring handler. This middleware ultimately calls proto/-parse and proto/-authenticate.
Looking at this diagram, you might think, “Isn’t this just the Strategy Design Pattern?” Indeed, in this pattern, Strategy corresponds to the Service Provider Interface, allowing authentication to be swapped in buddy-auth without modifying any code.
Summary
| Extensibility Mechanism | Macro (non with-style) | Macro (with-style) | Protocol |
|---|---|---|---|
| Extends | Clojure language itself | Code passed to the Macro | Modules |
| Behavior Modification Mechanism | Parses and rewrites body | Wraps body | Design and replace |
| Predefined Extension Points | None | None | Requires Protocols in the module design |
| Degree of Extensibility | High | Low | Medium (only specified parts are replaceable) |
If we were to name these three mechanisms:
- Non with-style Macros: syntax-rewriting extension
- With-style Macros: contextual extension
- Protocols: replace-based extension
Replace-based extension is relatively easy to grasp and common across programming languages. Contextual extension, while involving meta-programming, remains accessible. Syntax-rewriting extension, on the other hand, fundamentally alters the language itself, making it the domain of compilation experts.
Clojure provides excellent extensibility, offering diverse mechanisms that allow extensions at the language, code block, core library, and user-defined module levels. If you want to elevate your programming skills, consider how to design software for extensibility—it will make your software feel more Clojurish.
Note:
- In this article, you can consider “module” and “library” as synonymous. To me, a library is simply a published module.
- “Interface” and “Protocol” can also be regarded as synonymous. While there are subtle differences between them, there is no distinction in their usage within this article.
