Lambda Island Blog2022-11-04T19:45:28+00:00Arne Brasseurhttps://lambdaisland.com/blogAnnouncing our Clojurists Together Planshttps://lambdaisland.com/blog/2022-11-04-clojurists-together-20222022-11-04T19:45:28+00:00<<>><p><em>By Alys Brooks</em></p><p>In case you haven’t been hanging out in our <a shape="rect" href="https://discord.gg/fkU9xXj363">Discord</a> or <a shape="rect" href="https://clojurians.slack.com/archives/C1DV21X9P">Slack
channel</a> or
following <a shape="rect" href="https://www.clojuriststogether.org/news/q3-2022-funding-announcement/">Clojurists Together
news</a>, we’ve been
accepted into the latest round of Clojurists Together! Many thanks to Clojurists
Together and the backers who provide their budget. We’re also using
OpenCollective funds to match our Clojurists Together grant—many thanks to our sponsors on OpenCollective!</p><p>You may have heard <a shape="rect" href="https://www.clojuriststogether.org/news/september-2022-monthly-update/">some updates from fellow
awardees</a>
already—we opted for a delayed start and have been gearing up to make the most
of our funds over the next six months.</p><p>We’ve been triaging old issues (over 90), and as a part of that process, we’ve already
closed several stale or already completed issues, fixed 8 issues, and identified about 30
more that are good candidates for Clojurists Together. We’re not committing to finishing each and
every one, but we do want to make a significant dent in the pile, focusing
particularly on documentation, usability, and important fixes.</p><p>Probably the most exciting changes are merging and refining the long-awaited <a shape="rect" href="https://github.com/lambdaisland/kaocha/pull/234">Kaocha
parallelization feature</a> and making Kaocha-cljs2 significantly easier to get
started with.</p><p>We’re tracking our progress on <a shape="rect" href="https://github.com/orgs/lambdaisland/projects/2">a GitHub
project</a>. We’ll probably post
at least one or two updates here on the Lambda Island blog. We’ll also be
releasing improvements as we develop them, so you can also follow our projects
to stay in the loop:</p><ul><li><a shape="rect" href="https://github.com/lambdaisland/kaocha">kaocha</a></li><li><a shape="rect" href="https://github.com/lambdaisland/kaocha-cljs">kaocha-cljs</a></li><li><a shape="rect" href="https://github.com/lambdaisland/kaocha-cucumber">kaocha-cucumber</a></li><li><a shape="rect" href="https://github.com/lambdaisland/kaocha-junit-xml">kaocha-junit-xml</a></li><li><a shape="rect" href="https://github.com/lambdaisland/kaocha-cloverage">kaocha-cloverage</a></li><li><a shape="rect" href="https://github.com/lambdaisland/kaocha-cljs2">kaocha-cljs2</a></li><li><a shape="rect" href="https://github.com/lambdaisland/chui">chui</a></li><li><a shape="rect" href="https://github.com/lambdaisland/funnel">funnel</a></li><li><a shape="rect" href="https://github.com/lambdaisland/funnel-client">funnel-client</a></li><li><a shape="rect" href="https://github.com/lambdaisland/deep-diff2">deep-diff2</a></li><li><a shape="rect" href="https://github.com/lambdaisland/facai">facai</a></li><li><a shape="rect" href="https://github.com/lambdaisland/faker">faker</a></li></ul></<>>The REPL is Not Enoughhttps://lambdaisland.com/blog/2022-06-23-the-repl-is-not-enough2022-08-11T21:55:59+00:00<<>><p><em>By Alys Brooks</em></p><p>The usual pitch for Clojure typically has a couple ingredients, and a key one
is the REPL. Unfortunately, it’s not always clear on what ‘REPL’ means. Sometimes the would-be Clojure advocate breaks down what the letters mean—R for read, E for eval, P for print, and L for loop—and how the
different stages connect to the other Lispy traits of Clojure, at least.
However, even a thorough understanding of the mechanics of the REPL fails to capture what we’re usually getting at: The promise of interactive development.</p><p>To explore why this is great, let’s build it up from (mostly) older and more familiar
languages. If you’re new to Clojure, this eases you in. If Clojure’s one of your
first languages, hopefully it gives you some appreciation of where it came from.</p><h3 id="casino-repl-interactive-code-execution">Casino REPL: Interactive code execution</h3><p>The first level is being able to interactively enter code and run it at all.
You might be surprised to learn that (technically) Java has a REPL. Similar
tools exist for other static languages, like C and C#.</p><p>These can be handy for figuring out an obscure or forgotten bit of syntax or
playing around with an API.</p><p>These REPLs are typically afterthoughts and have limitations on what you can
define. In <code>jshell</code>, for example, you can define static methods but no classes.</p><h3 id="license-to-eval-full-access-to-the-language">License to Eval: Full Access to the Language</h3><p>The next step is basically no-compromises eval/execute—all the constructs of the
language are available. Most dynamic languages offer this, as do static
functional languages like Haskell and OCaml. Shells, including <code>bash</code> or
PowerShell, also have this level of capability.</p><p>These languages are generally high-level and may even resemble the pseudocode
you wrote or referenced, so using these REPLs to try out ideas and do some quick
testing can be a fluent experience. After all, shells were the main interface to
computers for several decades, and have remained in the toolbox of power users,
system administrators, developers, and
<a shape="rect" href="https://www.youtube.com/watch?v=YOQUVpEGyqg&t=39s">quartermasters</a> since.</p><p>Still, you run into some disadvantages:</p><ul><li>These REPLs start in a blank slate, but most development is in the context of an
existing program, often a very large one. You have to use imports to bring
in the relevant code, and it may take quite a bit of typing.</li><li>What you write in the REPL is often ephemeral. Ephemeral in the quality sense
is actually okay—writing one (or two, or three) versions in the REPL to
throw away isn’t bad. But it’s also ephemeral in a more literal sense. Once
the REPL ends, your code is either lost or not in a convenient format.
Recent history is typically just a few up arrow presses away, but to find
earlier code you have to either search or wade through typos, uses of <code>doc</code>
and other inspection, and design dead ends.</li></ul><h3 id="from-devtools-with-love-adding-context">From Devtools With Love: Adding Context</h3><p>Going beyond a sequential experience of entering code and seeing the result
takes us another step toward understanding what our code is doing. Actually,
it’s really two steps:</p><ol><li>Going beyond text to include graphs, images, animations, and widgets.</li><li>Showing the current state at all times.</li></ol><p>Web developers and data scientists have taken the lead here. Every major desktop
browser has a suite of tools for inspecting not only the JavaScript code but also the interface (the DOM) it generates or alters. Similarly, RStudio and Spyder are data science-oriented IDEs that keep a running process and allow you to see the values of any currently defined variables.</p><p>Some supercharged REPLs and REPL sidekicks exist for Clojure:</p><ul><li><a shape="rect" href="https://github.com/binaryage/dirac">Dirac</a> tweaks Chrome’s DevTools to
accomodate ClojureScript</li><li><a shape="rect" href="https://vlaaad.github.io/reveal/">Reveal</a> adds the ability to explore and
visualize data structures returned by the forms you’re evaluating.</li><li><a shape="rect" href="https://github.com/djblue/portal">Portal</a>, inspired by Reveal, similarly lets
you explore a variety of data types.</li></ul><p>Along similar lines, re-frame applications can use re-frame10x, which allows for
stepping back and forward to see the state of application.</p><p>Notebooks are another way of moving past the textual paradigm. Notebooks let you have inline diagrams, images, graphs, and even widgets. They also allow you to embed explanatory text and diagrams—the promised literate programming all the way from the 1970s. Some notebooks add a variable inspector and debugger, blurring the line between IDE and notebook. <a shape="rect" href="https://github.com/nextjournal/clerk">Clerk</a> brings these to Clojure.</p><h3 id="the-clojurian-with-the-golden-form-out-of-the-textual-repl">The Clojurian With the Golden Form: Out of the Textual REPL</h3><p>Clojure’s base REPL already has the strengths of the dynamic, expressive
languages mentioned in <code>License to Eval</code> (especially if you add tools and quality-of-life libraries from the previous section). However, many
Clojure developers find they are most productive if they can evaluate code as
they edit source code.</p><p>These are often done through advanced REPLs like nREPL, pREPL, and <a shape="rect" href="https://nrepl.org/nrepl/alternatives.html#comparison">their
alternatives</a>. (Rich Hickey
<a shape="rect" href="https://groups.google.com/g/clojure-dev/c/Dl3Stw5iRVA/m/IHoVWiJz5UIJ">has argued “REPL” is a misnomer</a> at least in nREPL’s case.)</p><p>Fully realized, Clojure forms and values become lingua franca allowing you to control, inspect, and redefine a variety of systems, as you
send code from your editor, terminal, or notebook to a REPL, a local instance of your
program, a browser, a node backend, or even a production instance.
Unfortunately, most of these require some setup. In particular, getting a ClojureScript REPL is
a multistage process, much like modern rockets, and prone to failure, much like
early rockets.</p><p>These advantages transcend the basic command-line evaluation that “REPL” often
suggests, so listing the REPL among Clojure’s advantages actually undersells the
feature if you don’t explain what it can actually do.</p><h3 id="sessions-are-forever-common-lisp">Sessions are Forever: Common Lisp</h3><p>The Clojure interactive development is not at the apex. Common Lisp went even
further by persisting state between sessions and letting you examine the state.</p><p>Perhaps the most noticeable is that these save where you left off. This makes it
easier to build up your program over time, at the cost of some state ambiguity.
If you wrote a function <code>process-input</code>, renamed it to the
<code>canonicalize-user-commands</code>, fixed a bug, and refactored it, <code>process-input</code> would still be
hanging around, with subtle differences.
Arguably working from an editor is a better fit for making changes to
long-running systems or collaborating with other programmers, but being able to persist sessions would be nice for smaller
programs or experimentation.
In addition to Common Lisp, Smalltalk, and R also remember where you left off.</p><p>Common Lisp has another super power: conditions and restarts. When your program
fails, it’s paused at the moment everything went wrong, allowing you to try to
recover, explore what went wrong, or even redefine things and resume
execution like nothing happened.</p><p>In Clojure and most other languages, an error,
exception, or panic basically shuts everything down. You can see the stack at
the moment of failure, but you can’t interact with it.
Rich Hickey was inspired by Common Lisp and the lack of the condition system is not
because he thought they weren’t valuable. As he explains in <a shape="rect" href="https://dl.acm.org/doi/abs/10.1145/3386321"><em>A History of
Clojure</em></a>,</p><blockquote><p>I experimented a bit with emulating Common Lisp’s conditions system, but I was swimming upstream.</p></blockquote><p>Some Clojurians have decided to try swimming upstream, but since these libraries aren’t
widely used, you’ll have to think carefully about how they’ll interact with
libraries that rely on Clojure’s native exceptions.</p><h3 id="conclusion">Conclusion</h3><p>Common Lisp being the endpoint of our journey puts the lie to my blog
post’s structure. Like most stories of only-increasing progress, this one isn’t completely true. Interactive development
<em>hasn’t</em> simply gotten better and better over time. We’ve lost ground in some
areas even as we’ve gained ground in others.</p><p>Still, we’re in a good place with Clojure. As the recent introduction of Clerk
demonstrates, there’s still interest in improving the interactive development
experience in Clojure.</p><h3 id="appendix-all-the-james-bond-clojure-puns-i-could-regrettably-not-fit-in-this-post">Appendix: All the James Bond-Clojure Puns I Could, Regrettably, Not Fit in this Post</h3><ul><li>Dr. Nil</li><li>Dyn Another Day</li><li>Live and <code>let</code> Die</li><li>From nREPL with Love</li><li>The <code>spy</code> Who Loved Me</li><li>You Only <code>defonce</code></li><li>MoonREPL</li></ul></<>>Setting up M1 Mac for Clojure developmenthttps://lambdaisland.com/blog/2022-05-09-clojure-setup-m1-mac-arm2022-05-09T13:14:03+00:00<<>><p>By Mitesh (@oxalorg)</p><p>I recently bought an M1 Macbook Air with 16 GB RAM. Below is a log of how
and what I downloaded on my machine for a full stack Clojure development experience.</p><p>The M1 is an ARM chip and thus the software which runs on it must be compatible
with the ARM instruction set instead of the x86 instruction set.</p><p>Fortunately I’ve had no problems moving from Linux/WSL to the M1 full time. There
were a few minor hurdles, but nothing has been a deal breaker yet.</p><p>The first thing I did was setup my Apple ID. Then I wanted to download my most
used apps which I did (without brew):</p><ul><li>Slack</li><li>Discord</li><li>OBS</li><li>Dropbox</li><li>Firefox developer edition</li><li>Docker for Mac</li><li>HomeBrew - <a shape="rect" href="https://brew.sh/">https://brew.sh</a></li></ul><p>HomeBrew (or simply “brew”) is a package manager for macOS. I haven’t had a
great experience dealing with some of the brewisms, but it has a huge
repertoire and it just works.</p><p>So let’s start brewing</p><pre><code class="language-bash"># mpv the best media player
brew install —cask mpv
brew install --cask keepassxc # save your ℀
brew install --cask flux # save your 👀
brew install --cask maccy # clipboard manager
brew install youtube-dl
brew install jq wget coreutils shellcheck tree cloc
</code></pre><p>What is <code>--cask</code> here? Cask is a way to install GUI application using
homebrew.</p><h4 id="compatibility-note">Compatibility note</h4><p>In the earlier days I think it was necessary to install x86 binaries in a
specially rosseta2 terminal, but I’ve never had to do that.</p><p>For example you must have seen people asking you to do this before running brew
install on specific software:</p><pre><code class="language-bash">/usr/bin/arch -x86_64 zsh
</code></pre><p>But I’ve never had to do this. I think the newer versions of MacOS automagically
detect x86 binaries and run them under rosseta2 without having to do anything at
all.</p><p>I’ve never once worried about this incomaptibility except for not being able to
run certain Docker containers (which is also quite rare).</p><h4 id="java--clojure">Java & Clojure</h4><p>It’s really simple to install JDK, Clojure CLI tools, and leiningen
using homebrew.</p><pre><code class="language-bash"># java/jdk, we need this for clojure
brew install openjdk
brew install leiningen
brew install maven
brew install clojure/tools/clojure
</code></pre><p>I chose to not install <code>brew install openjdk@17</code> explicitly, because leiningen
and maven depend on the latest openjdk so that gets installed anyways. If you
want a specific version like say JDK v11, then you can also do something like
<code>brew install openjdk@11</code>.</p><p>Then put the path of openjdk@11 first in your <code>$PATH</code> var in shell so clojure picks that up.</p><p>If the above does not work for you, try running this with sudo perms and see if that works as a fix:</p><pre><code class="language-bash">sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk
</code></pre><h3 id="babashka">Babashka</h3><p>Babashka has become such a huge part of my everyday workflow nowadays. So let’s install that ASAP</p><pre><code class="language-bash">brew install borkdude/brew/babashka
</code></pre><h3 id="node--npm">Node & NPM</h3><p>I mainly work on Clojurescript with shadow-cljs which only needs node
and npm installed.</p><pre><code class="language-bash">brew install node@14
</code></pre><h4 id="fonts">Fonts</h4><p>I use Iosevka as my choice of font, which is easy to install using brew.</p><p>Unfortunately there’s a rendering performance issue with Iosevka on
Emacs + MacOS. It was a hard bug to find, but switching to fira-code
fixes those performance issues.</p><pre><code class="language-bash">brew tap homebrew/cask-fonts
brew install —cask font-iosevka
brew install —cask font-fira-code
</code></pre><h4 id="emacs-on-mac-m1">Emacs on mac m1</h4><p>There are a lot of ways to install emacs on the m1, but I found <code>emacsmacport</code>
to be the best in the sense that it works out of the box and as expected 🙂</p><pre><code class="language-bash">brew tap railwaycat/emacsmacport
brew install --cask emacs-mac
ln -s ~/projects/dotfiles/emacs.d ~/.emacs.d
</code></pre><p>Then I changed the <code>CAPSLOCK</code> to <code>CTRL</code> to avoid the emacs pinky and also
because who uses caps lock 😂</p><p>Go to System Preferences → Keyboard → Caps and select it as “control modifier”</p><p><strong>Corgi Emacs</strong></p><p>My choice of emacs configuration layer is <a shape="rect" href="https://github.com/corgi-emacs/corgi">Corgi Emacs</a>,
it’s a lightweight version of spacemacs/doom aimed at Clojure users. So do give it a try!</p><h4 id="terminal">Terminal</h4><p>A lot of Mac users prefer to use iTerm2, and it has lots of benefits. But this
time around I didn’t really find a need for it, so I’m just using the built-in
<a shape="rect" href="http://terminal.app">terminal.app</a></p><p>macOS now ships with zsh as the default shell. So the first thing I did was
install oh-my-zsh and link my <code>.zshrc</code> dotfiles</p><pre><code class="language-bash">sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
ln -s ~/projects/dotfiles/shell/zshrc ~/.zshrc
</code></pre><p>Now my zsh config started complaining that it didn’t find tmux. Any terminal
I open launches a tmux instance automatically. This way I almost completely
live in <code>tmux</code> and the <a shape="rect" href="http://terminal.app">terminal.app</a> works just fine
for that! So let’s install <code>tmux</code> and some other terminal helper/utilities</p><pre><code class="language-bash">brew install tmux
brew install tmuxp
brew install ripgrep
brew install fzf
brew install fd
brew install direnv
</code></pre><p>Then I generate an ssh key. Folks remember to use ed25519 algorithm
as RSA is now being deprecated.</p><pre><code class="language-bash">ssh-keygen -t ed25519
</code></pre><h4 id="docker">Docker</h4><p>Docker for Mac was installed using their website and it runs as expected.
I haven’t had any issues uptil now as most of the images I run are well supported for ARM.</p><h4 id="terraform">Terraform</h4><p>A lot of terraform plugins had to be updated to newer versions which support
Darwin ARM builds.</p><pre><code class="language-bash">brew install hashicorp/tap/terraform
</code></pre><h4 id="conclusion">Conclusion</h4><p>All in all I am extremely happy with this machine. The battery life is amazing,
there is no FAN noise, it never heats up, and software runs and builds much
faster than my ~9 year old custom built PC.</p><p>One thing I hate is that it only has 2 ports and can only drive a single
external monitor. Both of these are minor annoyances for me, but not enough
to give up on this laptop.</p><p>Ref: my <a shape="rect" href="https://github.com/oxalorg/dotfiles">dotfiles repo</a>. Thanks</p></<>>Making Lambda Island Freehttps://lambdaisland.com/blog/2022-04-25-making-lambda-island-free2022-04-25T10:20:38+00:00<<>><p>Six years after launching Lambda Island we’ve decided to flip the switch and make all content free. That’s right, all premium video content is now free for anyone to enjoy. Go learn about <a shape="rect" href="https://lambdaisland.com/collections/react-reagent-re-frame">Reagent and re-frame</a>, <a shape="rect" href="https://lambdaisland.com/collections/ring-compojure">Ring and Compojure</a>, or <a shape="rect" href="https://lambdaisland.com/collections/clojure-foundations">Clojure Fundamentals</a>.</p><p>If you currently have an active subscription then it will be automatically cancelled in the coming days. You will no longer be charged. No further action is necessary.</p><p>Lambda Island is how the <a shape="rect" href="https://gaiwan.co">Gaiwan Team</a> gives back to the Clojure community. We have released over thirty different <a shape="rect" href="https://github.com/lambdaisland/open-source">open source projects</a>, we write articles <a shape="rect" href="https://lambdaisland.com/blog/">on our blog</a>, post videos on our <a shape="rect" href="https://www.youtube.com/lambdaisland">YouTube channel</a>, we moderate the <a shape="rect" href="https://discord.gg/pCuZBDRruW">Lambda Island discord community</a>, and provide a home for the <a shape="rect" href="https://clojureverse.org/">ClojureVerse forum</a>.</p><p>We do all of this for free because we want to see Clojurists and the Clojure community at large thrive. However our best stuff, the meticulously produced tutorial videos that Lambda Island started with, was still locked away behind a paywall. Which is a shame, because we feel more people need to see them. Since the start feedback on these videos has almost been unanimously positive, and so we want the world to see them. If we want to find a wider audience then there was really only one option, to make it all free.</p><p>We want to send a deep and heartfelt thank you to all the people who have supported us over the years. Some subscribers have been with us for the whole six years, and we know quite a few of you are continuing to pay, not because of the videos, but because of all the other stuff we do. That is truly humbling. There will still be multiple ways to support us going forward. The easiest way as an individual to support us financially is to sign up for the <a shape="rect" href="https://opencollective.com/lambda-island">Lambda Island Open Collective</a></p><p>Over the years Lambda Island has evolved from being primarily known for its videos and documentation, to being mostly known for its high quality open source tools and libraries, like the <a shape="rect" href="https://github.com/lambdaisland/kaocha">Kaocha</a> test runner, <a shape="rect" href="https://github.com/corgi-emacs/corgi">Corgi</a> editor, <a shape="rect" href="https://github.com/lambdaisland/ornament">Ornament</a> styling library, <a shape="rect" href="https://github.com/lambdaisland/embedkit">Embedkit</a> dashboard and BI engine, and many small but useful utility libraries like <a shape="rect" href="https://github.com/lambdaisland/uri">uri</a>, <a shape="rect" href="https://github.com/lambdaisland/fetch">fetch</a>, <a shape="rect" href="https://github.com/lambdaisland/glogi">glogi</a>, <a shape="rect" href="https://github.com/lambdaisland/classpath">classpath</a>, <a shape="rect" href="https://github.com/lambdaisland/edn-lines">edn-lines</a>, and <a shape="rect" href="https://github.com/lambdaisland/data-printers">data-printers</a>.</p><p>If your company is relying on Lambda Island open source then we’d <a shape="rect" href="mailto:arne@gaiwan.co">love to talk to you</a>. We want Lambda Island libraries and tools to be known for stability, reliability, and quality. A brand that inspires trust, and that you can build upon with confidence. But these things take maintenance, to ensure compatibility with an evolving ecosystem, to patch critical bugs and preempt security flaws. As a small team we constantly need to make choices of what to prioritize, which usually means that we work on what the needs are of current clients. But we don’t want other companies who use our libraries to become second class citizens. This is why we are working on a “Pro” program for Lambda Island Open Source. A way for companies to buy into this prioritization, to get advice on how to integrate and get the most value out of our offerings, and to get a direct line for support questions. If that sounds interesting or you want to learn more then please do <a shape="rect" href="mailto:arne@gaiwan.co">get in touch!</a>.</p></<>>What Is Behind Clojure Error Messages?https://lambdaisland.com/blog/2022-04-07-Clojure-Error-Messages2022-04-07T11:57:28+00:00<<>><p>by Ariel Alexi and Arne Brasseur</p><p>Have you ever looked at your REPL and asked yourself “What am I supposed to understand from this?”. This is not just a junior thought, but one also common to more advanced programmers. What makes it hard is that Clojure error messages are not very informative. This is why a lot of work was put into improving these error messages.</p><p>The errors seem a bit strange at first, but you can get used to them and develop a sense of where to look. Then, handling errors will be much less difficult.</p><p>In this post, we’ll look into four different types of errors.</p><h4 id="have-no-fear-of-the-stack-trace">Have No Fear of the Stack Trace</h4><p>When you get an error in Clojure, the stack trace will probably be the first place you will look to try to understand what happened. The problem with the stack trace is that you usually get a huge stack trace, hence it can be very intimidating if you are not familiar with it. But with the right strategy, you can get a lot of information out of it.</p><p>If the error happened inside a library you are calling, then it propably looks something like this:</p><pre><code>Exception .... (1)
at java...
at java...
at clojure...
at clojure...
at some_library.... (3)
at some_library....
at your_own_code.... (2)
at your_own_code....
at ...
... (lots more after this)
</code></pre><p>Stack traces are upside down; the most recent stack frame is at the top. In this case, our <code>your-own-code</code> namespace is calling the <code>some-library</code> namespace, which calls a Clojure function, which relies on Java. Some of these parts may be missing, but generally this is the pattern you’ll see. I’ve marked the most important places to look for.</p><p>(1) Look at the exception type and message. Do you understand what it means? We’ll give you some tips to understand these further down.</p><p>(2) Find the first line of the stack trace involving your own code, this will tell you a file and line number. Go look at that line. What is it doing? How could that relate to the error you got?</p><p>(3) Find the top-most line before that doesn’t start with “java” or “clojure”, this could be your own code, or library code, but it’s another important part of the puzzle. What is that line doing?</p><p>The rest of the stack trace is usually not that important, but don’t dismiss it just yet. Maybe you see that you have a <code>nil</code> somewhere causing a <code>NullPointerException</code>. But where did that <code>nil</code> come from? Look at the lines below of the ones pointed out, and try to understand where the value is coming from.</p><p>Most errors in Clojure can be classified into patterns and contain interfaces. There are many different patterns for errors, some of them were documented in <a shape="rect" href="https://github.com/yogthos/clojure-error-message-catalog">this repo</a> by members of the community. Let’s try to understand what we see by focusing on the errors type and error description which contains the patterns.</p><h4 id="how-to-recognize-errors-by-spotting-certain-patterns">How to Recognize Errors by Spotting Certain Patterns</h4><p>Let’s get down to business and tackle some of the confusing error messages you might encounter:</p><p><strong>1) Pattern #1: <code>X can not be cast to Y</code></strong></p><ul><li>Explanation: You have something of type X, but the function you are calling expects it to be of type Y. Clojure tries to cast X to be of type Y, but fails.</li><li>Pattern example: <code>Java.lang.String cannot be cast to clojure.lang.IFn.</code>
In this case, it’s not possible because the JVM doesn’t know how to turn a String into an IFn (Interface Function).</li><li>Code example: A number is being used where a function is expected.</li></ul><pre><code class="language-clojure">user=> (def x 10)
user=> (x)
Execution error (ClassCastException) at user/eval138 (REPL:1).
class java.lang.Long cannot be cast to class clojure.lang.IFn
</code></pre><p>Here it helps to learn to recognize some of the interface names that Clojure uses. <code>IFn</code> means a function, or something that acts like a function, like a keyword. <code>ISeq</code> is a sequence, like a <code>list</code> or the result of <code>filter</code> or <code>map</code>. <code>IPersistentCollection</code> means any Clojure collection, like a vector, map, or set.</p><p>A particular case that you might come across is Clojure complaining it can’t cast to <code>java.util.concurrent.Future</code>. This means you are trying to dereference something that isn’t dereferencable.</p><pre><code class="language-clojure">user=> @3
Execution error (ClassCastException) at user/eval7 (REPL:0).
class java.lang.Long cannot be cast to class java.util.concurrent.Future
</code></pre><p>You don’t actually need a future to replace the number in this case, it could be an atom, a delay, or any other reference type, so don’t the let “Future” in that error message throw you.</p><p><strong>2) Pattern #2: <code>Don't know how to create Y: from X</code></strong></p><ul><li>Explanation: This is very similar, Clojure tries to convert into X into Y, but can’t.</li><li>Pattern example: <em><code>IllegalArgumentException Don't know how to create ISeq from: java.lang.Long.</code></em>
In this case, it was expecting a collection but got a number, hence converting from a number (<em>Long</em>) into a collection was not possible.</li><li>Code example: Trying to iterate over a number, instead of a collection.</li></ul><pre><code class="language-clojure">user=> (reduce (fn [x y] (into x (+ x 1))) [1 2 3])
Execution error (IllegalArgumentException) at user/eval1$fn (REPL:1).
Don't know how to create ISeq from: java.lang.Long
</code></pre><p><strong>3) Pattern #3: <code>Wrong number of args (X) passed to: Y</code></strong></p><ul><li>Explanation: the function Y expected to get a different number of arguments instead of the X number of arguments we gave.</li><li>Code example: In this case, the function expects a single collection, but we are passing it 3 numbers.</li></ul><pre><code class="language-clojure">user=> (first 1 2 3)
Execution error (ArityException) at user/eval3 (REPL:1).
Wrong number of args (3) passed to: clojure.core/first--5402
</code></pre><p><strong>4) Pattern #4: <code>X - failed: even-number-of-forms? at: [:bindings] spec: :clojure.core.specs.alpha/bindings</code></strong></p><ul><li>Explanation: This happens during Clojure’s compilation phase, when you have a form like a <code>let</code> binding vector, which is expected to contain an even number of entries. You can see here that internally Clojure uses <code>clojure.spec</code> to validate these, which is why this error message looks a little different.</li><li>Code example: In this case, it was expected to have even numbers in the binding of the <code>let</code> but we gave it an odd number. The bindings vector should contain pairs of names and values, so it should always be even.</li></ul><pre><code class="language-clojure">user=> (let [a b 2] (+ a b))
Syntax error macroexpanding clojure.core/let at (REPL:1:1).
[a b 2] - failed: even-number-of-forms? at: [:bindings] spec: :clojure.core.specs.alpha/bindings
</code></pre><p>A related message is <code>Map literal must contain an even number of forms</code>, which happens when you evaluate something like <code>{:hello}</code>. The <code>:hello</code> key needs to have a value associated with it. If you meant to create a set, add a #: <code>#{:hello}</code>.</p><h4 id="final-thoughts">Final thoughts</h4><p>After reading those few examples and explanations, I hope you feel like next time you will know who is your main suspect when you investigate those errors.</p><p>From my standpoint, I think that Clojure error messages could be more informative and less threatening especially for those who are new to the language. But in the same breath, remember that writing code is 10% writing new code lines and 90% debugging and refactoring. At least now you know where to start when doing the 90%.</p></<>>Improve your code by separating mechanism from policyhttps://lambdaisland.com/blog/2022-03-10-mechanism-vs-policy2022-03-11T11:50:21+00:00<<>><p>by Arne Brasseur</p><p>Years ago when I was still a programming baby, I read about a concept, a way of
thinking about code, and it has greatly influenced how I think about code to
this day. I’ve found it tremendously helpful when thinking about API design, or
how to compose and layer systems.</p><p>Unfortunately I’ve found it hard at times to explain succinctly what I mean.
I’ve also realized that I’ve built up my own mental framework around this
concept that goes I think beyond the original formulation. So, in the hope of
inspiring someone else, to maybe elicit some interesting discussion, and also
just so that I have a convenient place on the web to link to, here’s my take on
“policy” vs “mechanism”.</p><p>It’s a concept I first came across some 15 years ago in <a shape="rect" href="https://www.linuxtopia.org/online_books/programming_books/art_of_unix_programming/ch01s06_3.html">The Art of UNIX
Programming</a>,
where it’s referred to as the “Rule of Separation”. I’ve always been hesitant to
refer to this book, because the author is a highly problematic character, but
it’s the only place where I’ve seen this concept explained, and I would really
like it to be more widely known. Wikipedia has a page on
<a shape="rect" href="https://en.wikipedia.org/wiki/Separation_of_mechanism_and_policy">Separation of mechanism and policy</a>, but
it mostly talks about security policies, rather than software development in
general.</p><p>The basic idea is very simple: make a clear distinction between code that
implements primitive functionality (mechanism), and code that implements
application and business logic (policy).</p><p>Mechanism are things like parsing JSON, sending an email, or GUI rendering
primitives; policy are things like a checkout flow, or a bulk data processing
task that reads and writes from an S3 bucket, or the implementation of the look
and feel of your application.</p><p>The distinction here isn’t so much in <em>what</em> the code does, as in <em>how</em> it is
written. There is a qualitative difference between pure Mechanism code vs Policy
code, and a lot of real-world code ends up somewhere in the middle, thus
creating a spectrum. I believe that pushing your code to live closer to the
edges of that spectrum will improve maintainability, and make your application
easier to evolve over time.</p><p>Here are some of the ways I think mechanism code differs from policy code:</p><table id="mech-vs-policy-table"><thead><tr><th colspan="1" rowspan="1" style="text-align: left">Mechanism Code</th><th colspan="1" rowspan="1" style="text-align: left">Policy Code</th></tr></thead><tbody><tr><td colspan="1" rowspan="1" style="text-align: left">Unopinionated</td><td colspan="1" rowspan="1" style="text-align: left">Opinionated</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Context-free</td><td colspan="1" rowspan="1" style="text-align: left">Contextual</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Individual pieces</td><td colspan="1" rowspan="1" style="text-align: left">Integrated whole</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Simple</td><td colspan="1" rowspan="1" style="text-align: left">Easy</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Explicit parameters</td><td colspan="1" rowspan="1" style="text-align: left">Hard coded values and defaults</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Functional</td><td colspan="1" rowspan="1" style="text-align: left">Procedural</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Abstract</td><td colspan="1" rowspan="1" style="text-align: left">Concrete</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">High level of reuse (rewiring)</td><td colspan="1" rowspan="1" style="text-align: left">Reuse through copy-paste-edit</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">“Model”</td><td colspan="1" rowspan="1" style="text-align: left">“Controller”</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Low level</td><td colspan="1" rowspan="1" style="text-align: left">High level</td></tr><tr><td colspan="1" rowspan="1" style="text-align: left">Easy to test</td><td colspan="1" rowspan="1" style="text-align: left">Tedious to test</td></tr></tbody></table><style>
#mech-vs-policy-table tr:nth-child(2n) td, #mech-vs-policy-table tr:nth-child(2n) th {
background-color: #ddd
}
#mech-vs-policy-table th, #mech-vs-policy-table td {
padding-right: 1em;
padding-top: 0.2em;
padding-bottom: 0.2em;
padding-left: 0.2em;
}
</style><p>Mechanism code solves one specific problem, or implements a specific primitive,
in a generic, unopinionated way. It doesn’t make assumptions about the context it
will be called from, or how it will be combined with other primitives. All of
those decisions are left to the caller.</p><p>It is truly “Simple” code (in the <a shape="rect" href="https://www.youtube.com/watch?v=SxdOUGdseq4">Simple Made
Easy</a> sense), but is often not
“Easy”. Because it tries to be unopionated and makes no assumption you need to
tell it exactly what you want. Everything it depends on needs to be passed in
explicitly. There is a level of compromise possible here where default values
are provided for the most common case. This does make it less “purely” mechanism
code, since this way aspects of your policy may start trickling in to your
mechanisms, but it can be the pragmatic thing to do.</p><p>Ideally I want my mechanism code to be functional, in other words: data in, data
out. Even if the underlying mechanism is actually side-effectful, I still like
to see a functional API that essentially takes instructions as input, and
returns a status report as output.</p><p>Policy code, on the other hand, can be very procedural. First do this, then do
that, and if that goes wrong, then do this other thing. It’s fine if stuff is
“hard coded”, it just needs to be clear and high level enough that it’s easy to
change. So where mechanism code tends to be highly abstracted, policy code is
the opposite, it deals very concretely with a specific use case. Mechanism code
gets reused by calling it with different parameters, and wiring it up in
different ways. Policy code doesn’t typically see actual reuse, the nearest you
have is finding a piece of code that does something somewhat similar, copying it
over, and then adjusting it to the task at hand.</p><p>So the rate of change is very different. Good mechanism code can live on for
years with hardly any changes. You may fix a bug from time to time, or make it
more efficient, or perhaps extend it to support more features, but the core
thing that it does stays the same.</p><p>Policy code on the other hand is really just a bunch of opinions. This is how we
decided that things should currently work. Expect to be tweaking and changing
this almost constantly.</p><p>Obviously mechanism code makes a prime candidate for extraction into a library,
and indeed many of the mechanisms we rely on we simply get by pulling in the
right library. Part of assessing whether you want to adopt a certain library can
be to look at how much it applies this philosophy of separation. As I mentioned
before, a lot of code ends up somewhere in the muddy middle ground, being
somewhat generic and abstracted, but still making a lot of assumptions that
might be true for the library author, but not necessarily for other library
consumers.</p><p>Extracting mechanism code into a library can be a valuable exercise because it
allows you to consider the code in isolation, to think about its API, its input
domain, and so forth. You can test it really rigorously to make sure you have a
piece of kit you feel confident about, and then later leverage it worry-free in
your policy code.</p><p>You get a lot of value out of writing tests for mechanism code, especially with
techniques like property-based testing where you can make assertions over a
large part of the possible inputs. Since the code tends to be fairly stable you
don’t have the problem of constantly having to adjust your tests. The main thing
you do over time is shaving off edge cases, patching them up and adding a
regression test.</p><p>Testing policy code is a lot harder, it often integrates multiple application
layers, from UI to database, so there is more setup, state, timing concerns, and
more chance of flaky tests. And since the code is expected to change
continuously, maintaining these kind of tests can add a significant maintenance
burden. That is not to say you should never test your policy code, but think
carefully about the trade-offs you are making. Maybe a smaller amount of tests
covering the things that are most likely to cause issues will give you more bang
for your buck.</p><p>I hope this exposition about mechanism vs policy was helpful. If this resonates
with you then please come join the conversation <a shape="rect" href="https://discord.gg/pCuZBDRruW">on Discord</a>.</p></<>>How to control the metacognition process of programming?https://lambdaisland.com/blog/2022-02-17-the-fg-command2022-03-10T11:23:18+00:00<<>><p>by Laurence Chen</p><p>There is a famous quote: Programming is thinking, not typing. After doing this job long enough, sometimes, I feel I am doing too much typing. No matter whether humans are thinking or typing, they use their brains, so I believe the word “typing” is a metaphor: typing refers to the kind of activities that we are doing unconsciously, doing through our muscle memory, not through our deliberate attention. Many of us, with enough experience, use quite a lot of muscle memory. On the other hand, do we use enough thinking when we need to?</p><p>We, humans, make a lot of decisions in our daily life: a decision like buying a car, or a decision like naming a function. Evolution makes us become the kind of animal that we can delegate certain parts of decisions down to our subconscious minds, which we may refer to as the background process in our brain. Therefore, we can focus our mind on something that matters. Here is the problem: what if the subconscious mind is not doing a good job? Most of the time, it is not a big deal, and escalating the issue back to foreground thinking can solve it. For example, when you press <code><localleader> e e</code> to evaluate a form, but mistakenly evaluating the wrong form. You detect that, then you use your foreground thinking to decide what to do next. However, sometimes, when you know that the subconscious mind can not handle the job well before you begin the job, what can you do to prevent your subconscious mind from taking over control? Do we have any mechanism similar to the linux command <code>fg</code>, which can bring a process from background to foreground?</p><p>I feel somewhat embarrassed to confess that, although I have been programming for ten years, sometimes when I encounter a bug, my brain first goes to the panic mode for hours, and then goes back to the analytic mode. Why? My theory is that when I developed my skills as a junior developer, I solved my problems in the panic mode: crazy goolging, trial and error, and etc. Year over year experience has trained my behaviors and muscle memory to debug in the panic mode. However, now I want to fix that. There are two methods that can help me control my subconscious mind:</p><ol><li>Explain the code to another person. (Rubber Duck Method)</li><li>Ask myself some pre-made questions. (Drucker Method)</li></ol><h3 id="remote-work-taught-me-the-rubber-duck-method">Remote work taught me the Rubber Duck Method</h3><p>When working with the Gaiwan team, we are working fully remotely. Sometimes, when I want to find someone to discuss, I need to wait. Several times, when I encounter a bug and I want to ask for help, I write down all the details about the code, the environment, and how I have attempted to solve the bug. After I paste my bug report in the Gaiwan’s communication channel and take a 15 minutes break, the magical thing just happens: the muse comes to my mind. I solve it quickly and edit the message that I have already posted. The Rubber Duck Method really works!</p><p>Actually, I think the idea of thinking by explaining has different names: Rubber Duck, Literate programming, Feynman Technique, etc. They are all similar things.</p><h3 id="right-question-is-like-the-fg-command-on-your-brain">Right question is like the fg command on your brain</h3><p>Now, when I encounter a bug, I ask myself <a shape="rect" href="https://lambdaisland.com/blog/2021-09-21-the-obstacles-of-effectively-debugging">three questions</a>:</p><ol><li>Do I use the scientific method to chase this bug?</li><li>Do I have the correct system view of this problem scope?</li><li>Do I have the necessary telemetry tool?</li></ol><p>I have similar issues when I write a function, write a module, or prepare the deployment: <strong>Typing too much and thinking too little</strong>.</p><p>Here are some questions I made for myself, but still alpha version:</p><h3 id="questions-for-writing-a-function">Questions for writing a function:</h3><ol><li>Should I keep command and query separate?</li><li>Do I write the <strong>contingent designs</strong> for errors like <code>try/catch</code> and logs?</li><li>Do I write the <strong>preventive designs</strong> for errors like <code>pre</code> condition or <code>assert</code>?</li><li>Should the function name describe <a shape="rect" href="https://leanpub.com/elementsofclojure"><strong>the purpose</strong></a>?</li><li>Do I add some suitable doc strings on functions?</li></ol><h3 id="questions-for-writing-a-module">Questions for writing a module:</h3><ol><li>Should I explicitly specify the APIs of this module, and make these APIs apparently different from internal functions?</li><li>Has all the unnecessary dead code been removed?</li><li>Do I design or use suitable Clojure records to model some <strong>immutable concepts</strong> of the domain problem?</li></ol><h3 id="questions-for-integration-and-deployment">Questions for integration and deployment</h3><ol><li>Do I add appropriate tests on domain functions?</li><li>Do I design appropriate feedback messages on setup scripts?</li><li>Can some manual setup steps be replaced with automation scripts?</li></ol><p>The question of using records needs more explanation: For some immutable concepts like <a shape="rect" href="https://lambdaisland.com/blog/27-02-2017-announcing-lambdaisland-uri">uri</a>, <a shape="rect" href="https://github.com/lambdaisland/deja-fu">date</a>, or <a shape="rect" href="https://github.com/lambdaisland/embedkit/blob/main/src/lambdaisland/embedkit.clj#L74">connection</a>, to represent them with some well-designed records will make the implementation details hidden in the proper layer. These immutable concepts tend to have a fixed set of related operations, which can be modeled using <code>Protocol</code>.</p><p>I found that both explaining or asking questions can help me effectively. They help me train my subconscious mind and the more analytic thinking I use the better divergent thinking I get. Are you also using some questions? Tell us your <code>fg</code> command.</p></<>>Unboxing the JDKhttps://lambdaisland.com/blog/2022-01-27-2022-01-27-unboxing-the-jdk2022-06-23T18:19:15+00:00<<>><p><em>By Alys Brooks</em></p><p>It’s easy to forget the Java Development Kit is, in fact, a kit. Many Clojure
developers, myself included, rarely work with commands like <code>java</code> directly,
instead using <code>lein</code>, <code>boot</code>, or <code>clojure</code>. Often we don’t even use the Java
standard library directly in favor of idiomatic wrappers.</p><p>There are a lot of advantages to staying in the Clojure level. Often, Clojure-specific
tools ergonomically support common practices like live-reloading, understand
Clojure data structures, and can tuck away some of the intermediate layers of
Clojure itself that aren’t a part of your application.</p><p>But the Java tools are still useful. So, in the spirit of a YouTube unboxing
video, let’s take a look at what’s in the JDK box. We’ll be looking at OpenJDK
16, since later versions of the JDK remove some legacy tools, like <code>appletview</code>,
which aren’t useful to Clojure developers today anyway.</p><p>The tools in post are listed roughly in order of importance.</p><h3 id="the-usual-suspects">The usual suspects</h3><p>The two tools Java developers use most are <code>java</code> and <code>javac</code>. There’s a good
chance you’re familiar with these (and if you are, feel free to skip ahead!) from doing
Java development.</p><p><code>java</code> starts the JVM, the program that converts Java bytecode into commands
that platform can run natively, collects garbage, profiles and optimizes the
running program, and provides information to monitoring tools. When you pass JVM
arguments to tools like the Clojure CLI, they’re sent to <code>java</code>.</p><p><code>javac</code> compiles Java source code into Java bytecode. Clojure projects don’t
actually use <code>javac</code> unless they include Java source code; Clojure has its own
compiler that emits Java bytecode using a subset of <a shape="rect" href="https://asm.ow2.io/">ASM</a>.</p><p>You’re probably also familiar with <code>javadoc</code>, even if you haven’t heard of it.
As the default documentation software for Java, it’s responsible for all of the
Java Standard Library docs, plus many third party libraries (You may also
remember it for generating pages with HTML frames well into the 00s.) Clojure
doesn’t have an <em>official</em> documentation tool, but options exist:
<a shape="rect" href="https://github.com/tomfaulhaber/autodoc">Autodoc</a>, which is used for the
<code>clojure.</code> namespaces plus a few others, and <a shape="rect" href="cljdoc.org">Cljdoc</a>, which is used
by many open source Clojure(Script) libraries, including Lambda Island’s.</p><h3 id="monitoring">Monitoring</h3><p>Regardless of what monitoring to do, start by running <code>jps</code> to identify the
process ID.
Unfortunately, most CLojure apps will be listed as <code>main</code> because that’s the
standard entry point. If <code>jps</code> is ambiguous, you can also use <code>jcmd</code>, which will
list the process ID and the command instead.</p><p>From that ID, a world of tools opens up:</p><ul><li>Running <code>jstack ID</code> will show you all the running threads in that process.
This is particularly handy when your application is taking longer to run
than expected.</li><li><p>Running <code>jcmd ID</code> plus commands let you find out specific information about
your running process, like GC stats: <code>jcmd ID gc.info</code>.</p></li><li><code>jconsole</code> has much of the same monitoring, just in a GUI format.</li></ul><h3 id="miscelleanous-helpers">Miscelleanous helpers</h3><p><code>jshell</code> (as in “java shell”, although I can’t help reading it as “JS hell”) is
a Java REPL. While the Clojure REPL makes a pretty good Java REPL,
thanks to its interop capabilities, if you’re interested in how something works
in Java, jshell can be a handy choice.</p><p><code>jdeps</code> shows dependencies of JARs, class files, or folders containing them.</p><p><code>jdb</code> is a commandline debugger for Java.</p><h3 id="two-jar-tools">Two JAR tools</h3><p>While you can create JARs with <code>jar</code>, it’s probably a better idea to leave the
JAR creation to your build tool and use <code>jar</code> solely to examine the contents.
Note that JARs are actually zip files, so you can actually use any tool that
opens zip files.</p><p><code>jarsigner</code>, well, signs JARs.</p><h3 id="experimental-tools">Experimental tools</h3><p>The following are experimental, so while they may be useful for debugging, you
probably don’t want to build them into scripts. They’re also slightly redundant
compared with <code>jcmd</code>, <code>jconsole</code>, and <code>visualvm</code>, but you may prefer their
format instead:</p><ul><li><code>jinfo</code> dumps a lot of details about the current jvm, including the class path, JVM
arguments (including the defaults), and more.</li><li><code>jstat</code> and <code>jstatd</code> provide many of the same monitoring options but using
sampling.</li><li><code>jmap</code> shows statistics about all the classes loaded by a program and also
lets you dump your program’s heap. It may be
useful as an initial look into your application’s memory usage, but isn’t
helpful for deep exploration on its own, since there’s not much detail. You
can open the dumps in <code>jhat</code> or <code>visualvm</code> for further analysis.</li><li><code>jhat</code> lets you analyze heap dumps through a very spartan web browser interface.</li></ul><p><code>jaotc</code> is an experimental ahead-of-time compiler. It’s also on its
<a shape="rect" href="https://openjdk.java.net/jeps/410">way out</a>
so I won’t say much more about it. <a shape="rect" href="https://www.graalvm.org/">GraalVM</a> is the suggested alternative, and there’s <a shape="rect" href="https://github.com/clj-easy/graal-docs/blob/master/doc/external-resources.md">lots of information</a> and <a shape="rect" href="https://github.com/clj-easy/graalvm-clojure">examples</a> on using it for Clojure projects.</p><p><code>jlink</code> is a bit of a dark horse. It lets you create custom runtimes with just
the parts your application needs. This doesn’t make a lot of sense in scenarios
where more than run java program is running on a platform, but for Docker
images and other situations where your program runs with its own JRE, it might be worth it to slim down the JRE.</p><p><code>jrunscript</code> runs various script files. To run a script file written in a
language other than JavaScript, you need to provide a JAR. You probably already
have a good runtime for any scripting langauge you’d use, so this doesn’t seem
useful except in cases where you have the JDK available and little else.</p><p><code>jhsdb</code> is similar to <code>jdb</code> but for crashed JVMs.</p><h3 id="and-one-more-thing">And one more thing</h3><p>For many versions, the JDK contained <code>jvisualvm</code>, a Java profiler. It’s still
available, just not as a part of the JDK. It has a much nicer interface than
<code>jconsole</code> and more functionality, particularly profiling and sampling, so
consider downloading it if you’re doing significant performance analysis.</p><p>When running it on Linux, you may want to provide <code>--laf com.sun.java.swing.plaf.gtk.GTKLookAndFeel</code>—on my machine, the default “look and feel” selects a tiny font.</p><h3 id="packing-up">Packing up</h3><p>One interesting thing is the many ways these commands are modeled after C and
Unix generally: <code>jar</code> after <code>tar</code>, <code>jdb</code> after <code>gdb</code>, <code>javac</code> after <code>cc</code>, and so on. This isn’t a
game-changing insight, but is interesting historically. Considering Java’s place
in programming language history, it makes sense. In the mid-90s, virtual machines
, while not new, were much less common than they are today and C and C++ were much more
dominant, so drawing a comparison made things easier for people coming from C or
C++ and probably increased its credibility.
By the time Clojure emerged, tools like Ant and Maven were already widespread in the Java
ecosystem, so Leiningen had more predecessors to learn from.</p></<>>Lambda Island Open Source Update January 2022https://lambdaisland.com/blog/2022-01-13-lioss-update2022-01-13T12:41:15+00:00<<>><p><em>By Alys Brooks</em></p><p>Community contributors and Lambda Island team members have been busy working on new projects and improving
existing ones.</p><p>If you want to join in, check out our <a shape="rect" href="https://lambdaisland.com/blog/2021-01-21-call-for-contributions-to-lioss">blog
post</a>
calling for contributions, or jump right in to our <a shape="rect" href="https://github.com/lambdaisland/kaocha/issues?q=is%3Aopen+is%3Aissue+label%3A%22first+issue+only%22">first
issues</a>
list.</p><h3 id="a-versioning-tweak">A Versioning Tweak</h3><p>You might notice that library versions look a little different. We now increment
the minor version on each release. This should make it easier to tell how many
releases between your current version and the latest.
For example, Kaocha’s at <code>1.60.977</code>.</p><p>Apart from that our versioning policy is still the same: the “teeny” version is
just the numbers of commits, and the major version is either 0 for comparitively
young and immature projects, or 1 if we feel ready to make a strong commitment
regarding API stability. Note that even for “0” projects we don’t take making
breaking changes lightly, and always consider how widely used a project already
is.</p><h3 id="kaocha-parallel-tests-and-more">Kaocha: Parallel tests and more</h3><p>A major focus has been adding parallelization support. We’re waiting to
stabilize the feature (although you’re <a shape="rect" href="https://github.com/lambdaisland/kaocha/pull/234#issuecomment-907548179">invited to try it</a> and
provide feedback) before formally releasing it, but it’s getting close.</p><p>In the meantime, here are newly released improvements to Kaocha:</p><ul><li><p><a shape="rect" href="https://cljdoc.org/d/lambdaisland/kaocha/1.60.977/doc/2-installing#alternative-method-exec-fn">kaocha.runner/exec-fn</a> for use with Clojure CLI’s -X feature</p></li><li><p>New plugin <a shape="rect" href="https://cljdoc.org/d/lambdaisland/kaocha/1.60.977/doc/8-plugins#gc-profiling">gc-profiling</a> for measuring the memory usage of tests.</p></li><li><p>Added support for code using :as-alias</p></li></ul><p>We’ve also fixed many bugs, so running Kaocha should be smoother than ever.</p><h3 id="regal-re2-and-lazy-qualifiers">Regal: RE2 and Lazy Qualifiers</h3><p>Regal received two new major features:</p><ul><li><p>Support for <a shape="rect" href="https://github.com/google/re2">Google’s RE2 regex library’s syntax</a></p></li><li><p><a shape="rect" href="http://www.rexegg.com/regex-quantifiers.html#lazy">Lazy quantifiers</a></p></li></ul><h3 id="fetch">Fetch</h3><p>We’re <a shape="rect" href="https://www.youtube.com/watch?v=jjt9Qx9MBPk">making Fetch happen</a>, well not really.
We’ve fleshed out the Fetch library, meaning a larger portion of the native
Fetch API is available in our ergonomic wrapper:</p><ul><li>Added support for all options that <code>js/fetch</code> understands. Option values can
be supplied as keyword or string. <code>:headers</code> is expected to be a Clojure map
from string to string: <code>:headers</code>, <code>:redirect</code>, <code>:mode</code>, <code>:cache</code>,
<code>:credentials</code>, <code>:referrer-policy</code>.</li><li>Implemented encoding for <code>:content-type :form-encoded</code></li><li>Added optional EDN support. Enable it by requiring the <code>lambdaisland.fetch.edn</code> namespace, which registers the necessary multimethods.</li></ul><p>And one change in behavior: Supplying a body as a string will not encode it, but use the string unchanged
as the body.</p><h3 id="a-new-tool-classpath">A New Tool: Classpath</h3><p>In case you missed it, Arne wrote a <a shape="rect" href="https://lambdaisland.com/blog/2021-08-25-classpath-is-a-lie">blog post</a> diving deep into the classpath and
introducing the need for the <code>lambdaisland.classpath</code> library. In short, when you <code>require</code> a namespace, there’s a lot more
going on than simply loading the first <code>.class</code> or <code>.clj</code> that matches.</p><h3 id="witchy-minecraft">Witchy Minecraft</h3><p>Combining the creativity of Minecraft with the interactivity of Clojure can
yield some amazing results. Over a year ago we started the
<a shape="rect" href="https://github.com/lambdaisland/witchcraft">Witchcraft</a> project to explore that
space, and we’ve been shipping major improvements almost every month since. We
did a <a shape="rect" href="https://youtube.com/playlist?list=PLhYmIiHOMWoGyYsWmcQN0sG40BRjnNGM3">series of videos</a>
to demonstrate the kind of things you can do with it.</p><p>If that sounds cool then come hang out <a shape="rect" href="https://discord.gg/pCuZBDRruW">on our Discord</a>,
or join the “#minecraft” channel on Clojurians Slack.</p><p>The most recent development is that we are working at integrating with the
<a shape="rect" href="https://wiki.citizensnpcs.co/Citizens_Wiki">Citizens2</a> plugin, so you can
populate your world with unique and interesting characters. We’ve also added
some new entries to the
<a shape="rect" href="https://github.com/lambdaisland/witchcraft/tree/main/src/lambdaisland/witchcraft/gallery">gallery</a>,
like a <a shape="rect" href="https://github.com/lambdaisland/witchcraft/blob/main/src/lambdaisland/witchcraft/gallery/mob_spawner.clj">classic mob spawner</a>
and a <a shape="rect" href="https://github.com/lambdaisland/witchcraft/blob/main/src/lambdaisland/witchcraft/gallery/birchwood_lodge.clj">birchwood lodge</a>.</p><h3 id="dont-be-a-stranger">Don’t Be A Stranger</h3><p>We’d love to have your help on these projects. To reiterate, if you want to join in, check out our <a shape="rect" href="https://lambdaisland.com/blog/2021-01-21-call-for-contributions-to-lioss">blog
post</a>
calling for contributions, or jump right in to our <a shape="rect" href="https://github.com/lambdaisland/kaocha/issues?q=is%3Aopen+is%3Aissue+label%3A%22first+issue+only%22">first
issues</a>
list.</p><p>Even if you can’t or don’t want to help out, we’d love for you to
<a shape="rect" href="https://discord.gg/pCuZBDRruW">join</a> the Lambda Island Discord. We hope that
it will be an enjoyable and inclusive community for Clojurians of all experience
levels. (If the link expires, you can message Arne (Arne#3086), Mitesh
(ox#1485), or myself (alysbrooks#2195) for an invite).</p></<>>Making nREPL and CIDER More Dynamic (part 2)https://lambdaisland.com/blog/2021-11-24-making-nrepl-cider-more-dynamic-22021-11-24T09:19:41+00:00<<>><p>By Arne Brasseur</p><p>In <a shape="rect" href="https://lambdaisland.com/blog/2021-11-03-making-nrepl-cider-more-dynamic-1">part 1</a> I
set the stage with a summary of what nREPL is and how it works, how
editor-specific tooling like CIDER for Emacs extends nREPL through middleware,
and how that can cause issues and pose challenges for users. Today we’ll finally
get to the “dynamic” part, and how it can help solve some of these issues.</p><p>To sum up again what we are dealing with: depending on the particulars of the
nREPL client (i.e. the specific editor you are using, or the presence of
specific tooling like <code>refactor-clj</code>), or of the project (shadow-cljs vs vanilla
cljs), certain nREPL middleware needs to present for things to function as they
should. When starting the nREPL server you typically supply it with a list of
middlewares to use. This is what plug-and-play “jack-in” commands do behind the
scenes. For nREPL to be able to load and use those middlewares they need to be
present on the classpath, in other words, they need to be declared as
dependencies. This is the second part that jack-in takes care of.</p><p>This means that nREPL servers are actually specialized to work with specific
clients, which is a little silly if you think about it. You can’t connect with
vim-iced to a server that expects CIDER clients, or at least not without reduced
functionality.</p><p>Instead what we want is for the nREPL server to be client-agnostic. Once a client
connects it can then tell the server of its needs, and “upgrade” the connection
appropriately. It can even upgrade the connection incrementally, loading support
for extra operations when it first needs them.</p><p>Let’s unpack what is needed to make this a reality, we need to be able to</p><ul><li>add extra middleware to a running server or connection</li><li>add entries to the classpath at runtime</li><li>resolve and download (transitive) dependencies</li></ul><h4 id="add-middleware-to-a-running-server">Add Middleware to a Running Server</h4><p>Turns out this problem has already been solved! Yay! Over a year ago Shen Tian
implemented a <code>dynamic-loader</code> middleware (see <a shape="rect" href="https://github.com/nrepl/nrepl/pull/185">nrepl/nrepl#185</a>),
which provides an <code>add-middleware</code> operation.</p><pre><code class="language-clojure">(-->
id "23"
op "add-middleware"
session "33d052f6-04dd-4f0e-916f-ed94aa0188ec"
time-stamp "2021-11-19 09:40:42.092185482"
middleware ("cider.nrepl/wrap-apropos" "cider.nrepl/wrap-classpath" "cider.nrepl/wrap-clojuredocs" "cider.nrepl/wrap-complete" "cider.nrepl/wrap-content-type" "cider.nrepl/wrap-debug" "cider.nrepl/wrap-enlighten" "cider.nrepl/wrap-format" "cider.nrepl/wrap-info" "cider.nrepl/wrap-inspect" "cider.nrepl/wrap-macroexpand" "cider.nrepl/wrap-ns" "cider.nrepl/wrap-out" "cider.nrepl/wrap-slurp" "cider.nrepl/wrap-profile" "cider.nrepl/wrap-refresh" "cider.nrepl/wrap-resource" "cider.nrepl/wrap-spec" "cider.nrepl/wrap-stacktrace" "cider.nrepl/wrap-test" "cider.nrepl/wrap-trace" "cider.nrepl/wrap-tracker" "cider.nrepl/wrap-undef" "cider.nrepl/wrap-version" "cider.nrepl/wrap-xref")
)
(<--
id "23"
session "33d052f6-04dd-4f0e-916f-ed94aa0188ec"
time-stamp "2021-11-19 09:40:42.958165047"
status ("done")
)
</code></pre><p>If you are a CIDER user and are fairly up-to-date (this may require using
<code>master</code>) you can try this out today.</p><pre><code class="language-clojure">(add-hook 'cider-connected-hook #'cider-add-cider-nrepl-middlewares)
</code></pre><p>Install this hook, then connect to a vanilla nREPL server. You can run one in your project with:</p><pre><code class="language-shell">clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version "RELEASE"}}}' -M: -m nrepl.cmdline
</code></pre><p>However, chances are you’ll see something like this in the cider-repl buffer:</p><pre><code>WARNING: middleware cider.nrepl/wrap-trace was not found or failed to load.
WARNING: middleware cider.nrepl/wrap-macroexpand was not found or failed to load.
WARNING: middleware cider.nrepl/wrap-inspect was not found or failed to load.
...
</code></pre><p>That’s because the <code>dynamic-loader</code> middleware tries to <code>(require
'cider.nrepl)</code>, and fails. We need to first get <code>cider-nrepl</code> onto the
classpath.</p><h4 id="adding-entries-to-the-classpath-at-runtime">Adding Entries to the Classpath at Runtime</h4><p>I have written at great length recently about the classpath and classloaders,
(see <a shape="rect" href="https://lambdaisland.com/blog/2021-08-25-classpath-is-a-lie">The Classpath is a Lie</a>). Simply
adding entries to the classpath is fairly easy, Clojure’s <code>DynamicClassLoader</code>
has a public <code>addURL</code> method. <a shape="rect" href="https://github.com/tobias/dynapath">dynapath</a>
provides an abstraction around this. You need a few lines of code to check if
you have the right kind of classloader, and if not instantiate a new one, and
you’re good.</p><p>The harder part is controlling which classloader is in use at the point where
<code>require</code> gets called. Typically this is the “context classloader”, which is a
<em>mutable thread local variable</em>. As if mutability alone wasn’t tricky enough.
Inside an nREPL request you’re generally covered, since nREPL creates a
<code>DynamicClassLoader</code> for you for each session. It used to be a little too eager
about creating new classloaders, which I addressed in
<a shape="rect" href="https://github.com/nrepl/nrepl/pull/248">nrepl/nrepl#248</a>. However there is
still the issue mentioned in <em>The Classpath is a Lie</em>, which is that
<code>clojure.main</code> creates a new <code>DynamicClassLoader</code> for each call to <code>repl</code>, which
in nREPL means on every eval. We do some recursing to find the
<code>DynamicClassLoader</code> which sits directly above the system classloader, and use
that. This tends to give fairly predictable results. There has been talk of
forking <code>clojure.main/repl</code> for nREPL’s use, which would allow us to get rid of
this annoying behavior, which would help simplify things.</p><p>To try this at home first find the location of the <code>cider-nrepl</code> JAR. This is a
fat jar, it includes all its dependencies inlined and shaded with
<a shape="rect" href="https://github.com/benedekfazekas/mranderson">MrAnderson</a>, so we don’t need to
worry about resolving transitive dependencies.</p><pre><code>find ~/.m2 -name 'cider-nrepl-*.jar'
</code></pre><p>Now we end up with something like this.</p><pre><code class="language-clojure">;; remove the previous one if necessary
(pop 'cider-connected-hook)
;; install the new hook
(add-hook 'cider-connected-hook
(lambda ()
(cider-sync-tooling-eval
(parseedn-print-str
'(.addURL (loop ; shenanigans to find the "root" DCL
[loader (.getContextClassLoader (Thread/currentThread))]
(let [parent (.getParent loader)]
(if (instance? clojure.lang.DynamicClassLoader parent)
(recur parent)
loader)))
(java.net.URL. "file:/home/arne/.m2/repository/cider/cider-nrepl/0.27.2/cider-nrepl-0.27.2.jar"))))
(cider-add-cider-nrepl-middlewares)))
</code></pre><p>And there you go, you’ve successfully turned a vanilla nREPL connection into a
cider-nrepl connection. You can now make full use of CIDER’s capabilities!</p><h4 id="resolving-dependencies">Resolving Dependencies</h4><p>The previous solution assumes that you already have <code>cider-nrepl*.jar</code> on your
system, that you know where to find it, that it matches the CIDER version Emacs
is using (or at least is compatible, they no longer need to match exactly), and
that it doesn’t need any additional dependencies.</p><p>A more generic solution would allow you to simply provide dependency
coordinates, the type you supply in <code>deps.edn</code> or <code>project.clj</code>, and let Clojure
figure out and download what it needs. Something like this:</p><pre><code class="language-clojure">(add-hook 'cider-connected-hook
(lambda ()
(cider-sync-tooling-eval
(parseedn-print-str
`(update-classpath! ((cider/cider-nrepl . ,cider-injected-nrepl-version)))))
(cider-add-cider-nrepl-middlewares)))
</code></pre><p>This is called “dependency resolution”. It means taking a set of
artifact-name+version coordinates, trying to find the given versions in one or
more repositories (like Clojars or Maven Central), downloading their <code>.pom</code>
files to figure out any transitive dependencies (recursively), and finally
downloading the actual jars.</p><p>This requires a good deal of machinery, machinery that is not present in every
Clojure process out of the box. You could start your nREPL process with
<a shape="rect" href="https://github.com/clojure/tools.deps.alpha">tools.deps.alpha</a> as a dependency,
or lean on other libraries that are lower-level (org.apache.maven, Aether), or
higher level (lambdaisland/classpath, Pomegranate). In any case we need these to
be declared and loaded at boot time, if they are not present than we have a
chicken-and-egg problem, our connection upgrade is once again blocked.</p><p>It’s also worth pointing out that this is by far the most complicated part of
this whole endeavor. Adding tools.deps.alpha adds about 11MB of dependencies.
Maybe that’s fine, many apps will pull in hundreds of megabytes of dependencies,
what’s a dozen more? Still, people often have good reasons to keep their
dependencies to a minimum, so this is not a decision that nREPL can make for
them.</p><p>But we can sidestep the issue, in the case of CIDER we only need to download a
single jar. We can just do that and be done with it. In fact, CIDER already
contains code to do just that:</p><pre><code class="language-clojure">(add-hook 'cider-connected-hook
(lambda ()
(cider-sync-tooling-eval
(parseedn-print-str
`(.addURL (loop
[loader (.getContextClassLoader (Thread/currentThread))]
(let [parent (.getParent loader)]
(if (instance? clojure.lang.DynamicClassLoader parent)
(recur parent)
loader)))
(java.net.URL. ,(concat "file:" (cider-jar-find-or-fetch "cider" "cider-nrepl" cider-injected-nrepl-version))))))
(cider-add-cider-nrepl-middlewares)))
</code></pre><p>Alternatively we could shell out to a tool that can do this work for us. We
actually have a lot of choice there at this point. There’s of course <code>clojure</code>
(i.e. the Clojure CLI), but Babashka can do the same thing (<code>bb clojure -Sdeps
{} -Spath</code>), and there’s <a shape="rect" href="https://github.com/borkdude/deps.clj">deps.clj</a>,
available as standalone binaries, or as an Uberjar, which could be invoked in a
separate process/JVM, or loaded onto the classpath and invoked from Clojure
directly.</p><p>It would be neat to wrap this in a little library which looks for one of these
executables in some default places, and falls back to downloading <code>deps.clj</code>.
This way you could get the functionality of a 11MB dependency for perhaps a
hundred lines of Clojure, although this may seem like an unsavory approach to
some.</p><p>It’s probably clear by now that there’s more than one way to shear a sheep, and
you may be wondering why we don’t just go and hide all these details behind a
facade that <em>Just Works</em>™. To an extent that will probably happen, these are
early days and we are still figuring out how to best fit these pieces together.
But we’re also likely to find that there’s no one size that fits all, whether
it’s all tools that build on nREPL, or all users of those tools.</p><p>In an upcoming blog post I’ll be talking a lot more about <em>Mechanisms</em> vs
<em>Policies</em>. I think at this point it’s ok to focus on the mechanisms, make sure
we have the individual ingredients, and let people experiment with combining
them in the way that makes most sense to them and their project.</p><p>Speaking of ingredients, there’s one more piece we haven’t covered yet, the
Sideloader!</p><h4 id="the-role-of-the-sideloader">The Role of the Sideloader</h4><p>Besides implementing the <code>dynamic-loader</code> middleware, Shen Tian also implemented
another nREPL piece, meant to complement it, the Sideloader. Inspired by a
similar mechanism in <a shape="rect" href="https://github.com/Unrepl/unrepl">unrepl</a>, the Sideloader
is a special kind of ClassLoader which requests the resources it needs from a
connected nREPL client.</p><p>The way this works is that the client first installs the sideloader by invoking
the <code>sideloader-start</code> op. Now every time you <code>require</code> a namespace, access an
<code>io/resource</code>, or load a Java class, the nREPL client (i.e. your editor) gets a
<code>sideloader-lookup</code> message. If it is able to provide the requested resource,
then it responds by sending a <code>sideloader-provide</code> message back.</p><p>The idea is that instead of adding cider-nrepl to the classpath directly, we let
CIDER (Emacs) supply each namespace or resource to nREPL on demand, over the
network.</p><p>I’ve put considerable effort into the client side implementation of this, see
<a shape="rect" href="https://github.com/clojure-emacs/cider/issues/3037">clojure-emacs/cider#3037</a>,
and the half a dozen PRs linked from there. This is why the aforementioned
<code>cider-jar-find-or-fetch</code> exists, we download the <code>cider-nrepl</code> JAR from Emacs,
so that we can supply its contents piecemeal to the Clojure process.</p><p>You can try this out as well:</p><pre><code class="language-clojure">(add-hook 'cider-connected-hook #'cider-upgrade-nrepl-connection)
</code></pre><p>This will enable the sideloader, and inject the necessary middleware as before.</p><p>So far I find the results rather underwhelming. Doubly so when connecting to an
nREPL server on a different machine, which is the use case where this approach
would actually make the most sense. Round-tripping over the network, and
extracting file contents from the JAR from Emacs Lisp, then base64 encoding and
decoding them to go over the wire… it all adds a lot of overhead. Note that
this impacts <strong>all</strong> classloader lookups, since we only fall back to the system
classloader when the Sideloader has determined that CIDER is unable to supply
the given resource. It’s also worth nothing that for every <code>.clj</code> file that
needs to be loaded this way, we round-trip three times: once to look for an
<code>__init.class</code> file, once for a <code>.cljc</code>, and only then does Clojure look for the
<code>.clj</code> file.</p><p>There are two obvious ways to improve this, one is to only have the sideloader
active for a limited amount of time. You activate it, inject the necessary
middleware, and turn it back off. Currently this does not work because of
cider-nrepl’s deferred middleware. Most middleware only gets <code>require</code>d the
first time it is actually used, at which point the sideloader has long been
disabled.</p><p>What I’ve also experimented with is providing a list of prefixes, so that only
cider-nrepl’s namespaces are fetched via the sideloader, it helps of course, but
the results are still underwhelming.</p><p>So as it stands I’m not convinced the sideloader is going to be an important
piece in making this dynamic upgrading of nREPL connections a reality. I think
the approaches I’ve set out above, where we make sure any resources required are
present on the classpath directly, are much to be preferred. Faster and more
reliable.</p><p>I do however think the Sideloader could become a cool piece of kit for loading
files from the user’s code base, especially when connected to a remote process.</p><p>I run a Minecraft server on a cloud instance, and use
<a shape="rect" href="https://github.com/lambdaisland/witchcraft-plugin">witchcraft-plugin</a> to endow
Minecraft with Clojure superpowers, including an nREPL server. Locally I have my
<a shape="rect" href="https://github.com/plexus/cauldron">cauldron</a> repo where I do my Minecraft
creative coding. It’s a collection of repl sessions, and of namespaces with
utility functions. When I eval <code>(require 'net.arnebrasseur.cauldron.structures)</code>
then that currently fails, because this Cauldron repo isn’t present on the
server. I need to go into each namespace I need and manually eval them in
topological order before I can use them. Not great. In this case I think it
would be fantastic if CIDER could spoon feed the server the necessary
namespaces on request.</p><h4 id="conclusion">Conclusion</h4><p>With this post I hope to draw some attention to all the work that’s been
happening. We’ve been laying the groundwork for really improving the user
experience, now we need to figure out how to bring it all together in a way that
makes sense.</p><p>There’s a risk though that pushing for these changes will initially negatively
impact the user experience, because change is hard, and we can’t anticipate
everyone’s use case and needs.</p><p>So I expect “jack-in” to stay around, and to remain the default recommendation.
It’s not perfect, but it works well for the vast majority of users.</p><p>At the same time we want to invite power users and tooling authors, especially
those that have experienced the limitations and frustrations that come with the
current approach, to consider these alternatives. To try them out and report
back, so that we can shave off the rough edges, abstract away some of the
plumbing, and gradually make this ready for broad consumption.</p></<>>Making nREPL and CIDER More Dynamic (part 1)https://lambdaisland.com/blog/2021-11-03-making-nrepl-cider-more-dynamic-12021-11-03T17:14:39+00:00<<>><p><em>This first part is a recap about nREPL, nREPL middleware, and some of the
issues and challenges they pose. We’ll break up the problem and look at
solutions in part 2.</em></p><p>The REPL is a Clojurists <a shape="rect" href="https://lambdaisland.com/blog/2017-12-29-the-bare-minimum-clojure-mayonnaise">quintessential
tool</a>,
it’s what we use to do Interactive Development, the hallmark of the LISP style
of development.</p><p>In Interactive Development (more commonly but somewhat imprecisely referred to
as REPL-driven development), the programmer’s editor has a direct connection
with the running application process. This allows evaluating pieces of code in
the context of a running program, directly from where the code is written (and
so not in some separate “REPL place”), inspecting and manipulating the innards
of the process. This is helped along by the dynamic nature of Clojure in which
any var can be redefined at any point, allowing for quick incremental and
iterative experimentation and development.</p><p>This is why it’s essential to the Clojure development experience to have proper
editor support, a plugin which bridges the gap between where the code is written
and where the code is run. So we have CIDER for Emacs, Calva for VS Code,
Cursive for IntelliJ, Conjure or Iced for Vim, and so forth. Often these will
also leverage the same (or a parallel) connection into the process for other
editor affordances, like navigation and completion.</p><p>But for these editor plugins to connect to the Clojure process something needs
to be listening on the other side, accepting connections, allowing the
initiation of a program-to-program dialogue. The most common way to achieve this
is by leveraging the nREPL protocol, an asynchronous message-based network
protocol for driving interactive development. The application process is started
with an embedded nREPL server, so that the editor can connect as an nREPL
client.</p><h2 id="nrepl-server-and-middleware">nREPL Server and Middleware</h2><p>nREPL is an extensible protocol, the reference server implementation understands
certain core <em>operation types</em> like <code>"eval"</code>. More operations can be supported,
or existing operations can be modified or augmented, through nREPL middleware.
For example: the Piggieback middleware can intercept <code>"eval"</code> messages, and
forward them to a ClojureScript environment, rather than evaluating them in the
Clojure process itself.</p><p>Which middleware to use will mostly depend on the editor you are using. You’ll
typically find that the Clojure-specific functionality for a given editor is
partly implemented as a typical editor extension, for instance CIDER written in
Emacs LISP, or Calva written in Typescript, and partly as nREPL middleware,
providing the functionality the editor extension relies on. For instance, both
CIDER and Calva rely on functionality provided by cider-nrepl.</p><p>Other tooling can also mandate specific middleware. When using Shadow-cljs you
need to use Shadow’s particular nREPL middleware to enable ClojureScript eval,
rather than the standard Piggieback. The clj-refactor Emacs package requires the
refactor-nrepl middleware to be able to do its work.</p><p>This has sometimes lead to confusion or frustration. Why are some editor
features not working? Which middleware do I need in my situation? How do I make
sure (and verify) that it’s loaded? What if my colleague uses a different
editor, does that change our setup? And so forth.</p><h2 id="easy-not-simple-jacking-in">Easy, not Simple: Jacking In</h2><p>What we have seen in response is that editors have mostly tried to relieve users
from having to start up their application process, instead the editor does it
for you, providing all the right flags and dependencies to make sure nREPL gets
started, with all the necessary middleware in tow. For instance when I do
<code>cider-jack-in</code> in Emacs in the context of a Clojure CLI (<code>deps.edn</code>) based
project it does something like this:</p><pre><code class="language-clojure">/usr/local/bin/clojure \
-Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.9.0-beta3"}
cider/cider-nrepl {:mvn/version "0.27.2"}
refactor-nrepl/refactor-nrepl {:mvn/version "2.5.1"}}}' \
-m nrepl.cmdline \
--middleware '[shadow.cljs.devtools.server.nrepl/middleware
refactor-nrepl.middleware/wrap-refactor
cider.nrepl/cider-middleware]
</code></pre><p>First it provides Clojure CLI, based on clojure/tools.deps.alpha, with extra
dependencies: the nREPL server, and the cider-nrepl and refactor-nrepl
middlewares. This will cause tools.deps to find these artifacts (poms and jars)
on Clojars or Maven Central, download them, download any dependencies they rely
on, and then add all those jars to the classpath. (The lookup path where Clojure
can find namespaces to load, or the JVM looks for compiled classes).</p><p>Then it invokes the <code>nrepl.cmdline</code> namespace’s <code>-main</code> function, passing it a
<code>--middleware</code> flag, which tells it which middleware vars to load and add to the
server when it boots up.</p><p>Note that in this case it also includes the shadow-cljs middleware. If nREPL is
started by shadow-cljs itself then this is already provided, but in this case I
want to use Clojure CLI, so I need extra project-level configuration to tell
CIDER that I need this middleware. Scenarios like this show that despite the
ease of use that “jack-in” commands have brought, this whole thing will still
regularly cause issues for people.</p><pre><code class="language-clojure">;; Project-specific setup via .dir-locals.el
((nil .
(cider-preferred-build-tool . clojure-cli)
(cider-clojure-cli-global-options . "-A:dev")
(eval .
(progn
(make-variable-buffer-local 'cider-jack-in-nrepl-middlewares)
(add-to-list 'cider-jack-in-nrepl-middlewares "shadow.cljs.devtools.server.nrepl/middleware")))))
</code></pre><p>And for all its convenience, this jack-in approach where the editor plugin takes
care of starting the process, nREPL, adding the necessary middleware, and then
connecting once the server is up and running is still not suitable in every
scenario.</p><p>For one it means your Clojure process runs “inside” your editor, while some much
prefer to run these separate, with the Clojure process running in its own
terminal window. If you have specific requirements for how to start the process,
say a specific Java version, or providing <code>deps.edn</code> aliases, then now you need
to figure out how to make your editor do those things. It may seem a lot easier
to just take care of the startup yourself. Maybe your process is even running
somewhere else, inside a container, through WSL, or on a server in the cloud. In
that case the jack-in approach might be firmly out of reach.</p><p>If you still want to have the rich editor support you have come to depend on,
then you will have to take care yourself of making sure nREPL and all necessary
middleware is on the classpath, to load the necessary namespaces, to start the
nREPL server, and to add the right middleware.</p><h2 id="running-your-own-nrepl-server">Running Your own nREPL Server</h2><p>This isn’t all that hard, there was a time when jack-in wasn’t a thing, and so
we all did this. If you look at old <code>project.clj</code> files you’ll often find
<code>cider-nrepl</code> or <code>piggieback</code> in there as a dependency, and nREPL options to
include the right middleware. Leiningen comes bundled with its own nREPL server,
which is configured through keys inside the <code>defproject</code>, which contributes to
many <code>project.clj</code> files being so bulky and full of boilerplate.</p><pre><code class="language-clojure">(defproject ,,,
:dependencies [[cider/piggieback "..."]
[cider/cider-nrepl "..."]]
:repl-options
{:nrepl-middleware [cider.nrepl/cider-middleware,cider.piggieback/wrap-cljs-repl]})
</code></pre><p>But this causes its own issues. These dependencies can easily get out of sync
with what your editor needs. And what if your teammates use different editors?
Or don’t upgrade their editors at the same pace? It quickly gets messy.</p><p>So can we do better? I believe we can! What we really want is to get rid of
these editor and tooling specific nREPL connections. Instead we provide a
“vanilla” nREPL server which any client can connect to. The client can ask the
server which capabilities it has, and if it needs more functionality, then it
can “upgrade” its connection. So that’s the lodestar we are aiming for. In the
next article we’ll break this up into three sub-problems, and look at the
options we have available for solving them.</p></<>>The Obstacles of Effective Debugginghttps://lambdaisland.com/blog/2021-09-21-the-obstacles-of-effectively-debugging2021-09-22T05:11:43+00:00<<>><p>by Laurence Chen</p><div style="text-align: center;"><p><img width="900" src="https://img.lambdaisland.com/4edeefd5d3b21ca8405f0aee8ba0102b.png" alt="Diagram of how the system and its environment relate to the bug hypothesis" /></p></div><p>Have you ever had one of the following experiences before?</p><ol><li><p>Your job is in customer service, and your company’s client reported that they encountered certain troubles in using your products. You forwarded the client’s complaint to the service team, but the service team told you, “Our product has no bugs.”</p></li><li><p>You needed to use a complex gadget, and it requires 15 steps. You followed the manual, but it did not work well. What do you do next?</p></li><li><p>You needed to use a shell command, even one which you’ve used successfully before. However, when you use it, it doesn’t work. The console gives you some error messages which you do not understand. You searched StackOverflow, but no answer could really get you to progress.</p></li></ol><p>The above difficulties are typical difficulties that users using a new system will encounter, no matter whether that system is a software product, a complex gadget, or a shell command. I have implemented some systems and have been the users of many systems. Having strong feelings about this kind of situation, I believe that people who want to help others to use a system smoothly or people who want to help themselves need to understand “how to debug.”</p><h1 id="the-scientific-method">The scientific method</h1><p>The core idea of debugging is quite similar to the scientific method.</p><p>When scientists want to explain a certain natural phenomenon, they first propose a hypothesis, do experiments which help to verify the hopothesis, observe the results, and then falsify the hypothesis or modify the hypothesis by the experimental results. After several rounds of repetitions, they provide an explanation which has been verified: we call it a theory. Debugging is quite similar to the above activity. If we want to catch a bug inside a system, we also prepare a hypothesis first. The hypothesis is our guess to the bug.</p><p>Here in Taiwan science is one of the things that our education system emphasizes. However, in reality, when I have helped or taught others to debug, I realized that merely eplaining what the scientific method is was not enough to get the learners to acquire this skill. The results were usually quite bad.</p><p>The scientific method of debugging consists of 4 steps:</p><ol><li>hypothesis</li><li>experiment</li><li>observe</li><li>modify hypothesis by observation</li></ol><p>My observation is that there typically are two obstacles to applying the scientific method:</p><ol><li>Not doing experiments well</li><li>Not knowing how to make a good hypothesis</li></ol><h1 id="change-one-thing-at-a-time">Change one thing at a time</h1><p>Sometimes, even myself, I change three different places inside the system and then hope to see the system magically work. However, most of the time, this kind of effort does not get me to my destination faster.</p><p>There is an important property in the experimental part of the scientific method: controlled variables. Every experiment should be used to explain a hypothesis of why the system does not work. Therefore, it is fine that you change three places together if they are dependent and you need to change them together to verify one hypothesis. However, it is not effective to change three places when each place corresponds to a distinct hypothesis.</p><h1 id="a-good-hypothesis-seldom-appears-when-you-dont-understand-how-the-system-works">A good hypothesis seldom appears when you don’t understand how the system works</h1><p>“The errors inside a system usually have a relationship with the physical outer world.” This is something I keep reminding myself when I examine my hypothesis.</p><p>When I was in college, I once conducted a circuit experiment. After I followed the circuit map and created a circuit on breadboard by placing all the active and passive components, the output voltage was not correct. My hypothesis was that probably one of the resistors is broken, and given that examining each one is a little bit too much effort, I decided to reimplement the whole circuit again. Before I began, the teaching assistant showed me how to debug. He picked several points inside the circuit, and he measured the voltages. Then, he pointed out where the bug was.</p><p>In the above example, the younger me made a hypothesis by assuming that anything can break randomly. On the other hand, the teaching assistant, he made a hypothesis based on the mental model of the system and he focused on thinking about the interaction of the circuit components inside the system. Also, he used a multimeter to measure the voltage (the experiment and observe parts).</p><h1 id="debugging-in-our-daily-life">Debugging in our daily life</h1><p>I was once asked by my wife, “Do you think that debugging is an exceptional thing? Well, it depends.” I then replied, “Last time you drove your car to get it fixed by a car mechanic, that mechanic listened to the sound of the car and then made his diagnosis. Does this story sound similar to debugging?”</p></<>>The Classpath is a Liehttps://lambdaisland.com/blog/2021-08-25-classpath-is-a-lie2021-11-16T16:22:47+00:00<<>><p>by Arne Brasseur</p><p>A key concept when working with Clojure is “the classpath”, a concept which we
inherit from Clojure’s host language Java. It’s a sequence of paths that Clojure
(or Java) checks when looking for a Clojure source file (<code>.clj</code>), a Java Class
file (<code>.class</code>), or other resources. So it’s a lookup path, conceptually similar
to the <code>PATH</code> in your shell, or the “library path” in other dynamic languages.</p><p>The classpath gets set when starting the JVM by using the <code>-cp</code> (or
<code>-classpath</code>) command line flag.</p><pre><code>java -cp src:/home/arne/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar clojure.main
</code></pre><p>Entries on this “classpath” are either directories (like <code>src</code>), or JAR files
(like <code>clojure-1.10.3.jar</code>), which are really just zip files in disguise.</p><pre><code>unzip -l ~/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar
</code></pre><p>When you <code>require</code> a namespace, Clojure will look for a corresponding <code>.clj</code>,
<code>.cljc</code>, or <code>.class</code> file “on the classpath”. You can do the same by using
<code>clojure.java.io/resource</code>.</p><pre><code class="language-clojure">(require '[clojure.java.io :as io])
(io/resource "clojure/main.class")
;;=> #object[java.net.URL 0x3237dfe5 "jar:file:/home/arne/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar!/clojure/main.class"]
</code></pre><p>Intuitively we think of this performing something like the following pseudocode:</p><pre><code class="language-clojure">(some #(find-file-in-directory-or-jar % "clojure/main.class") the-classpath)
</code></pre><p>It’s a useful mental model. It is also wrong. No my sweet summer child, in the
world of ClassLoaders and URLClassPaths nothing is ever that straightforward.</p><p>The trouble starts when you want to do anything more than find a named resource
on the classpath. Perhaps you want to inspect the classpath, iterate over all
the files on the classpath, add or remove entries to or from the classpath. What
you find out is that in Java</p><ul><li>everything happens somewhere else</li><li>all the good bits are hidden from you</li></ul><h2 id="everything-happens-somewhere-else">Everything Happens Somewhere Else</h2><p>This is supposedly a quote from Adele Goldberg, one of the pioneers working at Xerox PARC:</p><blockquote><p>In Smalltalk, everything happens somewhere else.</p></blockquote><p>(I can’t find a good source to support this though, I’d be grateful if anyone is
able to trace this to a primary source.)</p><p>Java is no different. You don’t just <em>do</em> stuff. You ask an object to do it,
which asks another object, which delegates to its parent implementation, and so
forth. When you need to look up stuff on “the classpath”, you ask a
<code>java.lang.ClassLoader</code>.</p><pre><code class="language-clojure">(.getResource ^ClassLoader loader "clojure/main.class")
</code></pre><p><code>ClassLoader</code> is an abstract class with many descendants, including</p><pre><code>jdk.internal.loader.ClassLoaders$AppClassLoader
jdk.internal.loader.ClassLoaders$PlatformClassLoader
jdk.internal.loader.BootClassLoader
jdk.internal.loader.BuiltinClassLoader
jdk.internal.loader.SecureClassLoader
java.net.URLClassLoader
clojure.lang.DynamicClassLoader
</code></pre><p>Each class retains a reference to the classloader it was loaded with:</p><pre><code class="language-clojure">(.getClassLoader (class (fn [])))
;; => clojure.lang.DynamicClassLoader@4413660d
(.getClassLoader clojure.main)
;; => jdk.internal.loader.ClassLoaders$AppClassLoader@443b7951
(.getClassLoader java.sql.Time)
;; => jdk.internal.loader.ClassLoaders$PlatformClassLoader@4d131e92
(.getClassLoader String) ; more on this special case below
;; => nil
</code></pre><p>So we’ve established we need a classloader before we can do anything
classpath-y. Where do we get one? If you need access to a ClassLoader in Java
for some reason you typically just get the one that the class of <code>this</code> was
loaded with, and use that.</p><pre><code class="language-clojure">this.getClass().getClassLoader()
</code></pre><p>But we don’t have <code>this</code> in Clojure. Let’s maybe see which classloader Clojure
itself uses when it needs to require a namespace:</p><pre><code class="language-java">package clojure.lang;
public class RT {
static public ClassLoader baseLoader(){
if(Compiler.LOADER.isBound())
return (ClassLoader) Compiler.LOADER.deref();
else if(booleanCast(USE_CONTEXT_CLASSLOADER.deref()))
return Thread.currentThread().getContextClassLoader();
return Compiler.class.getClassLoader();
}
}
</code></pre><p>It first checks the <code>clojure.lang.Compiler/LOADER</code> dynamic var. From scouring
the code it seems this is used to set the loader internally during a specific
scope, but what this ultimately is used for I have no idea. It does give you a
way to override the classloader that Clojure uses, by giving that var a root
binding. This is something we do in Kaocha to allow us to add test directories
to the classpath at runtime, although I’m not sure this is recommended, and I
may reconsider how we do that after having leveled up considerably recently when
it comes to classpath shenanigans.</p><p>Next it uses the “context class loader”, if <code>USE_CONTEXT_CLASSLOADER</code> is true,
which by default it is. This one is interesting, it’s a thread-local, mutable
ClassLoader field, so you can <code>setContextClassLoader</code> as well as
<code>getContextClassLoader</code>.</p><p>What’s this for? According to <a shape="rect" href="https://stackoverflow.com/questions/1771679/difference-between-threads-context-class-loader-and-normal-classloader">this StackOverflow
post</a>
which has lots of juicy details “[it] exists only because whoever
designed the ObjectInputStream API forgot to accept the ClassLoader as a
parameter, and this mistake has haunted the Java community to this day”. Perhaps
a tad dramatic. Fact is that this is the ClassLoader Clojure looks at (under
typical circumstances). And since it’s mutable that gives us some options for
doing… interesting stuff.</p><h2 id="the-classloader-chain">The Classloader Chain</h2><p>ClassLoaders don’t come alone, they bring all their ancestors with them. Each
classloader has a reference to a parent.</p><pre><code class="language-clojure">(defn classloader-chain [cl]
(take-while identity (iterate #(.getParent %) cl)))
</code></pre><p>When evaluating this in Clojure you should see at least three entries, one
provided by Clojure, and two by Java. There is actually one more ClassLoader,
the BootLoader, but it’s built-in to the virtual machine, you don’t get to see
it. It’s represented by <code>nil</code>.</p><pre><code class="language-clojure">(classloader-chain (clojure.lang.RT/baseLoader))
;;=>
[clojure.lang.DynamicClassLoader@107842e8
jdk.internal.loader.ClassLoaders$AppClassLoader@443b7951
jdk.internal.loader.ClassLoaders$PlatformClassLoader@4d131e92]
</code></pre><p>(To keep the spacial metaphors straight, I’m going to refer to this list
interchangeably as the “stack” or “chain” of classloaders. The ones higher in
the list will be “up” the stack, the ones lower in this last are “down” the
stack.)</p><p>If you evaluate this using nREPL you may instead have gotten a much longer list,
with a whole bunch of <code>clojure.lang.DynamicClassLoader</code> instances at the top,
followed by the same <code>AppClassLoader</code> and <code>PlatformClassLoader</code>. This is
something I <a shape="rect" href="https://clojureverse.org/t/dynamically-adding-to-the-classpath-in-a-post-java-9-world-help/2520/6">had noticed before</a>,
and never had gotten a satisfying answer about, until <a shape="rect" href="https://danielsz.github.io/blog/2021-05-12T13_24.html">Daniel Szmulewicz recently blogged</a> about
it. I highly recommend reading his post, it makes a good complement for this
one, and will help deepen your understanding.</p><p>So what happens when you call <code>getResource</code> or <code>getResources</code> on Clojure’s
class loader? When you dig in you’ll see that neither <code>DynamicClassLoader</code> nor
its parent <code>URLClassLoader</code> implement <code>getResource</code>, so they inherit the base
implementation in <code>java.lang.ClassLoader</code>.</p><pre><code class="language-java">package java.lang;
public abstract class ClassLoader {
public URL getResource(String name) {
URL url;
if (parent != null) {
url = parent.getResource(name);
} else {
url = BootLoader.findResource(name);
}
if (url == null) {
url = findResource(name);
}
return url;
}
}
</code></pre><p>This checks in order:</p><ul><li>the parent classloader</li><li>the BootLoader</li><li><code>findResource</code> on <em>this</em> classloader</li></ul><p><code>findResource</code> is where classes like <code>URLClassLoader</code> implement their own logic
for finding resources, <code>getResource</code> wraps <code>findResource</code>, but with added logic
for traversing down the chain of parent classloaders.</p><p>What’s important to notice here is that <em>the parent is checked first</em>. This is a
key insight, it means that if any of the classloaders down the chain return a
resource, then the current classloader is simply bypassed.</p><p>And what’s that <code>BootLoader</code> stuff in the middle? It allows adding classpath
entries with <a shape="rect" href="https://www.eclipse.org/openj9/docs/xbootclasspath/">special <code>-Xbootclasspath</code>
flags</a>. These are always
checked first, so this lets you replace Java’s own classes with patched
versions.</p><p>Looking again at this <code>classloader-chain</code>, we now know that when looking for a
resource, these are checked bottom-to-top.</p><pre><code class="language-clojure">[clojure.lang.DynamicClassLoader@107842e8
jdk.internal.loader.ClassLoaders$AppClassLoader@443b7951
jdk.internal.loader.ClassLoaders$PlatformClassLoader@4d131e92
nil]
</code></pre><p>First the built-in “boot classloader” is checked, providing some core classes
that are baked into the JVM, like <code>String</code>, and fundemenatal things like
<code>java.base</code> or <code>java.logging</code>.</p><p>Then the <code>PlatformClassLoader</code> is checked. Here you find classes provided by
Java SE. This class loader was introduced with the module system introduced in
Java 9, and doesn’t use a classpath at all, but it knows how to load classes
from specific core modules like <code>java.sql</code> or <code>java.xml.dom</code>.</p><p>Then we get the <code>AppClassLoader</code>, also called the “system class loader”. This is
where “the classpath” (the one we passed to java with <code>-classpath</code>) gets
checked.</p><p>If none of those find the resource, then Clojure’s <code>DynamicClassLoader</code> gets a
turn. In this case it doesn’t do anything of its own, it simply inherits the
implementation from <code>URLClassLoader</code>, and by default it does not have any URLs
in the list that it checks.</p><p>So what’s it for? It implements methods like <code>findClass</code> and <code>loadClass</code>, to
return classes which are only defined in memory. This allows the compiler to
generate bytecode on the fly when you evaluate forms, without having to create
<code>.class</code> files on disk. So far we’ve mostly talked about locating resources via
classloaders, but as the name suggests their first use is to find “classes”,
i.e. to load byte code from disk and turn it into a class definition that the
JVM can work with.</p><pre><code class="language-clojure">(defn xxx [])
(.loadClass (clojure.lang.RT/baseLoader) "user$xxx")
;; user$xxx
(.loadClass (ClassLoader/getPlatformClassLoader) "user$xxx")
;; => java.lang.ClassNotFoundException
</code></pre><p>Here you see that when you define a function, it really creates a class (and an
instance thereof). Clojure’s classloader knows about this class, it can find it
in memory. Java’s classloader has no idea.</p><h2 id="so-what-do-you-do-with-this">So what do you do with this?</h2><p>Ok, that was already a lot of theory and nitty gritty details… why am I doing
this to myself? (and to you, dear reader)</p><p>One thing I like to be able to do is add dependencies to a project without
having to restart the REPL process every time. For a brief period in time this
worked wonderfully for me using <a shape="rect" href="https://github.com/clj-commons/pomegranate">Pomegranate</a>, but since Java 9 this
stopped working, and my <a shape="rect" href="https://clojureverse.org/t/dynamically-adding-to-the-classpath-in-a-post-java-9-world-help/2520">pleas for help</a>
largely fell on deaf ears.</p><p>Since then I found the <code>Compiler/LOADER</code> workaround which we use in Kaocha, and
there’s an experimental <a shape="rect" href="https://github.com/clojure/tools.deps.alpha/tree/add-lib3">tools.deps add-lib3 branch</a>
that adds this functionality directly to tools.deps.alpha. Cool beans!</p><p>Now let’s add a little twist. We have over a dozen Lambda Island open source
libraries at this point, and many depend on each other. It happens regularly
that you are working on one library, and halfway through you figure out you need
some related changes or additions in another library.</p><p>The typical thing to do is to change <code>deps.edn</code> to use a <code>:local/root</code>
reference, restart your REPL, and continue from there. When your REPL is quick
to restart and you don’t have much state to build up again then maybe that’s not
a big deal, but it can get pretty annoying.</p><p>We started running into the same problem with Nextjournal. We’re in the process
of extracting and releasing some of the modules that go into making Nextjournal,
the way we’ve already open-sourced <a shape="rect" href="https://github.com/nextjournal/clojure-mode">clojure-mode</a>.
(Keep an eye out for this, it’s good stuff!)</p><p>But that means going from monorepo bliss to a situation where there’s a lot more
overhead in maintaining and coordinating these things. On top of that for a big
application like Nextjournal restarting your REPL can take a little while, it’s
something we really like to avoid.</p><p>So we tried adding it with <code>add-lib</code>.</p><pre><code class="language-clojure">(require '[clojure.tools.deps.alpha.repl :as deps-repl]
'[clojure.java.classpath :as cp]
'[clojure.java.io :as io])
(deps-repl/add-lib {nextjournal.clojure-mode {:local/root "../clojure-mode"}})
(filter #(re-find #"clojure-mode" (str %)) (cp/classpath))
;; => ("/home/arne/Nextjournal/clojure-mode")
(io/resource "nextjournal/clojure_mode.cljs")
;;=> "/home/arne/.gitlibs/libs/nextjournal/clojure-mode/a83c87cd2bd2049b70613f360336a096d15c5518/src/nextjournal/clojure_mode.cljs"
</code></pre><p>Ok, what’s going on here? We’ve tried adding the <code>:local/root</code> version of
clojure-mode to the classpath. We can see it’s on there, and yet when we look
for something specific we see it’s still getting looked up in the gitlib
version. Frustrating!</p><p>Let’s first get some better insight into what’s happening, by looking at the
stack of classloaders again, and for each checking which locations it checks.</p><pre><code class="language-clojure">(defn classpath-chain
"Return a list of classloader names, and the URLs they have on their classpath"
[]
(for [cl (classloader-chain)]
[(symbol
(or (.getName cl)
(str cl)))
(map str (cond
(instance? URLClassLoader cl)
(.getURLs cl)
(= "app" (.getName cl))
(map #(File. ^String %)
(.split (System/getProperty "java.class.path")
(System/getProperty "path.separator")))))]))
</code></pre><p>Here we loop over the <code>(classloader-chain)</code> we had earlier. Some of them will be
instances of <code>URLClassLoader</code>, and for these we can simply ask them what URLs
they check.</p><p>For the app/system classloader the situation is a little different. Up to Java 8
this was also a <code>URLClassLoader</code>, but since Java 9 that’s no longer the case,
and there’s no good way to inspect its actual classpath. It contains a
<code>URLClassPath</code> instance, but it’s private. Remember, all the good bits are
always out of reach.</p><p>But we can look at how it’s being initialized:</p><pre><code class="language-java">package jdk.internal.loaders
public class ClassLoaders {
private static final BootClassLoader BOOT_LOADER;
private static final PlatformClassLoader PLATFORM_LOADER;
private static final AppClassLoader APP_LOADER;
static {
// ...
String cp = System.getProperty("java.class.path");
URLClassPath ucp = new URLClassPath(cp, false);
APP_LOADER = new AppClassLoader(PLATFORM_LOADER, ucp);
}
}
</code></pre><p>It takes the <code>java.class.path</code> system property as its path, so assuming no one
has changed the property since then, this gives us a way to find out what
classpath it’s looking at.</p><pre><code class="language-clojure">(classpath-chain)
;;=>
([clojure.lang.DynamicClassLoader@711fe6bb ("/home/arne/Nextjournal/clojure-mode")]
[app
("dev"
"test"
"src"
"resources"
"/home/arne/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar"
"/home/arne/.m2/repository/org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.jar"
"/home/arne/.m2/repository/org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.jar"
"/home/arne/.gitlibs/libs/nextjournal/clojure-mode/a83c87cd2bd2049b70613f360336a096d15c5518/src/nextjournal/clojure_mode.cljs")]
[platform ()])
</code></pre><p>As you can see <code>add-lib</code> added the new directory to Clojure’s
DynamicClassLoader, but the gitlib version is still part of the application
class loader. And since we know that loaders lower down the stack are checked
first, files in the gitlib will shadow files in the <code>:local/root</code>.</p><p>Why wasn’t <code>(cp/classpath)</code> telling us this? Turns out the current
implementation is flawed, as soon as the <code>DynamicClassLoader</code> contains a URL (as
is the case after calling <code>add-lib</code>, it completely ignores the system classpath,
even though in reality it is still checked (and even gets priority!).</p><p>Sadly we can’t change the system classloader. Its classpath may as well be set
in stone. But we can define our own classloader, one which plays by our own
rules.</p><p>The plan is as follows:</p><ul><li>Define our own subclass of <code>URLClassLoader</code></li><li>Install it directly above the bottom-most <code>DynamicClassLoader</code> (one
<code>DynamicClassLoader</code> is enough, the fact that there may be many is an
unfortunate side-effect of how Clojure and nREPL interact).</li><li>Have it first check its own paths, then those of its parent, and only then
delegate further to Java’s classloaders</li></ul><p>Note that the old version (the gitlib) will still be there, but we only look for
files there if they don’t exist in the <code>:local/root</code> version. This works fine if
you are only changing or adding files, but if you delete a file you may start
seeing it pick up the old version.</p><pre><code class="language-clojure">(defn priority-classloader
[cl urls]
(let [cp-files (map io/as-file urls)
find-resources (fn [^String name]
(mapcat (fn [^File cp-entry]
(cond
(and (cp/jar-file? cp-entry)
(some #{name} (cp/filenames-in-jar (JarFile. cp-entry))))
[(URL. (str "jar:file:" cp-entry "!/" name))]
(.exists (io/file cp-entry name))
[(URL. (str "file:" (io/file cp-entry name)))]))
cp-files))]
(proxy [URLClassLoader] [(str `priority-classloader) (into-array URL urls) cl]
(getResource [name]
(or (first (find-resources name))
(.findResource (.getParent this) name)
(.getResource (.getParent this) name)))
(getResources [name]
(java.util.Collections/enumeration
(distinct
(concat
(find-resources name)
(mapcat
enumeration-seq
[(.findResources (.getParent this) name)
(.getResources (.getParent (.getParent this)) name)]))))))))
</code></pre><p>Ok this is getting a little hairy. I was hoping I could just call
<code>(.findResource this ...)</code> to have it search the list of paths, but that’s not
working… I followed the trail from <code>URLClassLoader</code> to <code>URLClassPath</code> to
<code>Loader</code> (remember how in Java everything always happens somewhere else?), and
eventually gave up and implemented my own <code>find-resources</code> logic.</p><p>The important bits are here:</p><pre><code class="language-clojure">(or (first (find-resources name)) ; Search our own paths
(.findResource (.getParent this) name) ; Search DynamicClassLoader
(.getResource (.getParent this) name)) ; Search App/Platform/Boot
</code></pre><p>Now let’s try installing it (<code>root-loader</code> is helper to find the
<code>DynamicClassLoader</code> that sits immediately above the application classloader,
like I said, one is enough):</p><pre><code class="language-clojure">(.setContextClassLoader
(Thread/currentThread)
(priority-loader (root-loader) ["/home/arne/Nextjournal/clojure-mode"]))
</code></pre><p>That should do the trick, but it doesn’t, at least not when you evaluate this
from nREPL. The problem there is that nREPL captures the current context
classloader at the beginning of each <code>eval</code> operation, and restores it
afterwards. So we need to somehow set it “outside” of the current <code>eval</code>.</p><pre><code class="language-clojure">(let [thread (Thread/currentThread)]
(future
(Thread/sleep 100)
(.setContextClassLoader
thread
(priority-loader (root-loader (context-classloader thread) ["/home/arne/Nextjournal/clojure-mode"])))))
</code></pre><p>We use a future which closes over the current thread, and then update the
thread’s context classloader from the future, which runs on a different thread.</p><p>And… it works! Except that when you try to navigate with something like
<code>cider-find-var</code>, presumably because that uses a separate nREPL session, which
runs on a different thread, which has its own classloader. Here too there are
solutions. The first thing I tried was simply forcing Orchard (the library
backing CIDER) to use a specific classloader, and that works. Now I’m leaning
towards looping over all threads, and installing this priority-loader on every
thread that has a <code>DynamicClassLoader</code>.</p><h2 id="conclusion">Conclusion</h2><p>So what did we learn?</p><ul><li>There’s really no such thing as “the classpath”, but there’s a hierarchy of
class loaders, and you can kindly ask them to give you stuff.</li><li>Clojure has two places where it looks for a classloader, <code>Compiler/LOADER</code> and
the thread’s context classloader. Setting the first is easy, setting the
second requires some more care.</li><li>Adding something to “the classpath” is easy once you have a
<code>DynamicClassLoader</code>/<code>URLClassLoader</code>, but removing something that is part of
the classpath the application started with is impossible, and replacing
something only kind of works by using some heavy trickery to make the new
entry take precedence over the old.</li><li>Use the source! Clojure and Java are both open source, which is fantastic. I
would not have been able to get this far without having these sources
available.</li></ul><p>This is not the last word on these topics, these experiments are ongoing, we’ve
set up a repo under
<a shape="rect" href="https://github.com/lambdaisland/classpath">lambdaisland/classpath</a> where you
can find the current state of things. In particular it contains this cool helper:</p><pre><code class="language-clojure">(update-classpath!
'{:aliases [:dev :test :licp]
:extra {:deps {com.lambdaisland/webstuff {:local/root "/home/arne/github/lambdaisland/webstuff"}}}})
</code></pre><p>This will read <code>deps.edn</code>, use it to construct a “basis” with the given options,
and then add any new entries to the classpath. It’s as close to a <code>deps.edn</code>
reload as you’ll get, and you can add local overrides, as I’ve shown here.</p><p>We’re also working on other quality-of-life helpers in there, like
<code>git-pull-lib</code> to update the <code>:git/sha</code> of a library to the latest commit in a
branch.</p><p>If you want to discuss this post just head on over to <a shape="rect" href="https://clojureverse.org/t/the-classpath-is-a-lie/8093">ClojureVerse</a>!</p></<>>A Tale of Three Clojureshttps://lambdaisland.com/blog/2021-08-18-a-tale-of-three-clojures2021-08-19T23:07:38+00:00<<>><p><em>By Alys Brooks</em></p><p>Recently, I was helping a coworker debug an issue with loading a Clojure
dependency from a Git repository. (If you don’t know you can do this; it’s very
handy. <a shape="rect" href="https://clojure.org/guides/deps_and_cli#_using_git_libraries">Here’s</a> a
guide.) I realized that there were really two Clojures at play: the Clojure that
<code>clojure</code> was running to generate the classpath and the Clojure that was used by
the actual Clojure program.</p><p>Taking a step back, there are really three things we might mean when we say
“Clojure”:</p><ul><li>The language of Clojure, as in “Clojure has immutable data structures.”</li><li>The commandline tool, as in <code>clj</code></li><li>The dependency of Clojure, as in <code>[org.clojure/clojure "1.10.1"]</code></li></ul><p>Let’s disambiguate these three different Clojures and learn a little about how
Clojure works under the hood along the way.</p><h1 id="clojure-the-language">Clojure the language</h1><p>Usually when we’re talking about “Clojure,” we’re talking about the language,
like when we say things like “Clojure is a Lisp” or “Clojure has multiple
lock-free concurrency abstractions.”</p><p>But what about statements about stack traces, like “If I see another impenetrable Clojure
stack trace today, I’m going to give up programming for good and become a
lighthouse operator off the coast of Nova Scotia”?
Stack traces have a lot to do with the language—they include the namespace and
function for each line of the file in Clojure. However, they also have a lot to
do with the implementation. Stack traces of alternate implementations of
Clojure, like Clojurescript, reflect their host platform and the resulting
decisions the implementers made. For the JVM version of Clojure, the
implementation consists of classes (usually bundled as a JAR) specified on the classpath,
which we’ll discuss in the final section of this article.</p><p>For once, we’re not going to say much about Clojure the language. To actually
use Clojure, we need to run it. That’s usually accomplished with Leiningen,
Boot, or our next topic, the Clojure CLI tool.</p><h1 id="clojure-the-cli-tool">Clojure the CLI tool</h1><p>Officially, the <code>clojure</code> CLI tool’s purpose is to “declare
dependencies, assemble classpaths, and launch Clojure programs with data.”</p><p><code>clojure</code> serves a similar role as interpreters and runtimes like <code>python</code>,
<code>node</code>, and <code>java</code>, but as the “with data” above hinted, it does so in a
particularly Clojure-y way.</p><p>To help understand what it does, let’s <a shape="rect" href="https://sourceforge.net/projects/clojure/files/clojure/">grab the
JAR</a>, go back in time
and start Clojure manually, like <a shape="rect" href="https://web.archive.org/web/20080516070205/http://www.clojure.org/reference/getting_started.html">Clojure.org recommended in
2008</a>:</p><pre><code class="language-bash">
java -cp clojure.jar clojure.lang.Repl
</code></pre><p><a shape="rect" href="https://www.youtube.com/watch?v=r21CMDyPuGo">One does not simply</a> load a single JAR in the very latest versions because of the new
dependency on <code>clojure.spec.alpha</code>, but it does work for Clojure 1.8.0 and
earlier versions. If you have it locally, you can update the above command to:</p><pre><code class="language-bash">
java -cp ~/.m2/repository/org/clojure/clojure/1.8.0/clojure-1.8.0.jar clojure.lang.Repl
</code></pre><p>What does this do? Well, it starts up a JVM, telling it that we want to load a
JAR (basically a zip file of classes), and run the <code>clojure.lang.Repl</code> class
from it. All other classes it needs are either in the JAR or are included
with the JVM itself.</p><p>Say, what if you want to run an actual Clojure program? Simply call:</p><pre><code class="language-bash">
java -cp clojure.jar clojure.lang.Script script.clj
</code></pre><p>Easy! If we can start Clojure and run programs in compact one-liners, what’s the
point of the <code>clojure</code> CLI tool? Well, if you have any sort of dependencies
beyond Clojure itself, manually determining what goes after <code>-cp</code> quickly
becomes challenging.</p><p>Even before the <code>clojure</code> CLI tool, Leiningen or Boot, people used Java tools to help them manage the classpath.
When I asked on Clojurians slack, a few people commented that they used either
<code>ant</code> or a Maven plugin. According to Alex Miller, the Clojure Core team still
uses Maven plus <code>clojure-maven-plugin</code>. However, once introduced, Leiningen
swept the Clojure world. The <a shape="rect" href="https://cemerick.com/blog/2012/08/06/results-of-the-2012-state-of-clojure-survey.html">2012 State of Clojure
survey</a>,
the first that asked about build tooling, showed that <em>95 percent</em> of users used
Leiningen. Fast-forward to 2021, and Leiningen is now down to <a shape="rect" href="https://www.surveymonkey.com/results/SM-S2L8NR6K9/">75
percent</a>, Maven is down to 5
percent, and Ant isn’t even on the list anymore (although two people mentioned
it under Other).</p><p>Thus, the most important job of <code>clojure</code> (as well as Leiningen and Boot) is
generating the part of the Java invocation that comes right after <code>-cp</code>.
This illustrates a flaw in my comparison of <code>clojure</code> to <code>python</code> or <code>java</code>. While it is true that
a Clojure programmer uses <code>clojure</code> to start a Clojure program in the same way
she uses <code>python</code> to run a Python script or <code>java</code> to run a compiled Java program,
<code>clojure</code> the CLI tool is a launcher (and with <code>tools.build</code>, a <a shape="rect" href="https://clojure.org/guides/tools_build">build tool</a>)—it isn’t a runtime or interpreter in itself. (That’s <code>java</code>, plus the core language classes in that <code>clojure.jar</code> we saw above.)</p><p>How does it know what dependencies to add to the classpath?
Because it’s a Clojure tool, the dependencies are stored in the EDN format,
which is basically just the parts of Clojure syntax that are useful for
specifying data. (Because of the close correspondence between data and code in
Lisps, the “parts of Clojure syntax that are useful for specifying
data” turns out to be…most of it.) For example, <code>[1 2 3]</code> is valid EDN. <code>(vector 1 2 3)</code> is
<em>also</em> valid EDN, but it won’t be evaluated. In other words, it will be read as a list containing the symbol
<code>vector</code> followed by the numbers 1, 2, and 3, but the <code>vector</code> symbol won’t be looked up and called, so it will remain a list.
Of course, once read, you can pass the list to <code>eval</code> yourself (resulting in <code>[1 2 3]</code>) or do whatever else you want to with it.</p><p>While we’re on the topic, we can also look at the difference between <code>clojure</code>
and <code>clj</code>.
<code>clj</code> is actually a script rather than a binary, so we can look at it
directly in a text editor. Here’s what it contains on my system:</p><pre><code class="language-bash">#!/usr/bin/env bash
if type -p rlwrap >/dev/null 2>&1; then
exec rlwrap -r -q '\"' -b "(){}[],^%#@\";:'" clojure "$@"
else
echo "Please install rlwrap for command editing or use \"clojure\" instead."
exit 1
fi
</code></pre><p>If <code>rlwrap</code> exists, it runs it with a few Clojure-specific settings, like
indicating that parentheses are not part of words but separate them.
If <code>rlwrap</code> doesn’t exist, it <code>echo</code>es a message telling the user to install it.
In short, <code>clj</code> simply wraps <code>clojure</code> with <code>rlwrap</code>, which adds completion,
editing, and command history.</p><p>So, what’s in that <code>Clojure.jar</code> that <code>clojure</code> loads for us? That brings us to
the final of the three Clojures. Clojure the Dependency.</p><h1 id="clojure-the-dependency">Clojure the Dependency</h1><p>A lot of Clojure projects, whether they use the
Clojure CLI tool (sometimes referred to by its main library, <code>tools.deps</code>), Leiningen, or Boot, include a reference to Clojure itself as a
dependency.
This Clojure dependency is not the same as the Clojure standard library
(although it certainly includes it). More on that later.</p><p>However, all three of these tools are themselves written in Clojure, so how does
this work?</p><p>Here’s an abbreviated version of the process when the user runs <code>clj</code>:</p><ol><li>Runs the Clojure CLI tool once to generate the proper classpath. On this
run, the Clojure CLI tool’s classpath consists of a hard-coded JAR that contains the classes that make up
<code>clojure</code> as well as its dependencies.</li><li>Runs the Clojure CLI tool again with the new classpath in order to actually provide the REPL or run
the application.</li></ol><p>This means <code>java</code> actually gets started twice, unless it can use a cached
classpath and skip step 1.</p><p>I mentioned earlier that the Clojure dependency is more than the standard
library. Let’s open that early Clojure JAR from 2008 to see what’s inside. It
is an archive containing over 200 files, so we won’t look at them all.
Here are some broad categories of what’s inside:</p><ul><li><p>Actual machinery of the language. For example, <code>lang/LispReader.class</code> is the reader, which takes the characters and puts them into a data
structure—the lists Lisp is so famous for. (The reader is responsible for the Read part of
Read-Evaluate-Print-Loop (REPL).)</p></li><li>Persistent data structure implementations. For example, <code>lang/PersistentArrayMap.class</code> is one of the classes used for hash maps.</li><li><p>Various interfaces, like <code>clojure/lang/ISeq.class</code>, which is the
interface all seqs implement.</p></li><li>Core libraries, like <code>clojure/set/set.clj</code>, which contains the
<code>clojure.set</code> namespace.</li></ul><p>What does that mean in practice? One example is the addition of
<a shape="rect" href="https://clojure.org/guides/weird_characters#_and_namespace_map_syntax">namespace map
syntax</a>, which was added in Clojure 1.9.
In order for this to work, you need a copy of <code>lang/LispReader.class</code>
that supports this syntax. Running <code>clojure -Sdeps '{:deps {org.clojure/clojure
{:mvn/version "1.10.1"}}}' -e "(keys #::{:a 1, :b 2})"</code> succeeds, while
<code>clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.8.0"}}}' -e "(keys
#::{:a 1, :b 2})"</code> fails. Syntax changes like this are actually pretty rare because
it’s easy to extend Clojure without adding syntax.
More often, the Clojure core team adds
features to the core libraries. For example, Clojure 1.9 also added
<a shape="rect" href="https://github.com/clojure/clojure/blob/master/changes.md#13-new-predicates">a number of predicates</a>, like <code>boolean?</code> and <code>int?</code>, in order to support the
new spec library.</p><p>To tie it all together, in order to take advantage of the new namespace map
syntax added to the <em>language</em> or the new predicates added to the core library in Clojure 1.9, you need to specify the Clojure 1.9 <em>dependency</em>
to the <code>clojure</code> CLI <em>tool</em>.</p></<>>Launching the Lambda Island Redesignhttps://lambdaisland.com/blog/2021-07-23-launching-lambda-island-redesign2021-07-23T15:08:07+00:00<<>><p>It’s finally live! A gorgeous, in-depth redesign of the Lambda Island website.
After months of hard work we soft-launched earlier this week. Today we want to
tell you a little more about the project, the whys and the hows, and to invite
you to check it out. And when you’re done do come tell us what you think <a shape="rect" href="https://discord.gg/pCuZBDRruW">on our
Discord</a>.</p><p><img src="https://img.lambdaisland.com/de32254acec9d7dd8f380711286a46e4.png" alt="Redesigned front page" style="margin: 2rem 0" /></p><p>We already told you in <a shape="rect" href="https://lambdaisland.com/blog/2021-06-17-lambdaisland-is-changing">a previous
post</a> how
<a shape="rect" href="https://lambdaisland.com">Lambda Island</a> and <a shape="rect" href="https://gaiwan.co">Gaiwan</a> are
changing. In a short amount of time we went from a one man endeavor to a team of
six, drastically changing what we are able to take on and pull off.</p><p>Until last week the Lambda Island site still looked the way it did when we first
launched five years ago. It’s what I cobbled together in early 2016, and shipped
in a matter of months. It was functional, and had a certain DIY charm, but it
was not a professional design by any means. Since Lambda Island is how the
community knows us, and since the company is so much more than what it was back
then, we felt an upgrade was long overdue.</p><figure><p><img src="https://img.lambdaisland.com/2ff6a0b374731879c576ea39bd76d500.png" alt="" style="margin: 2rem 0" /></p><figcaption><p>What the site looked like <a shape="rect" href="https://web.archive.org/web/20170603005445/https://lambdaisland.com/">in 2017</a></p></figcaption></figure><p><a shape="rect" href="https://twitter.com/flpbar">Felipe Barros</a> took the lead on this, pitching
design ideas from soon after he joined in 2019. I held off the boat for a while,
but once we had reason to make new changes to the site we realized we could no
longer continue on the existing basis. It was time for a clean slate: branding,
design language, and implementation.</p><p>Felipe is a self-taught developer based in Recife, Brasil, and the first person
to join what would become our budding team. He had done web design work in the
past, creating static sites for local clients. These weren’t technically complex
projects, but what I saw were clean and appealing designs, making good use of
what the open web had to offer. I decided to bring Felipe on board.</p><p>To establish a fresh new brand identity we brought in <a shape="rect" href="https://lubovsoltan.com/">Lubov
Soltan</a>, a professional UI/UX designer and brand
specialist working out of Toronto, Canada. We worked with her in the past for
<a shape="rect" href="https://heartofclojure.eu/">Heart of Clojure</a> and we were all excited to work
together again. We settled on the lighthouse as a symbol, a guiding light to
help you navigate the stormy waters on your Clojure journey, and building on the
nautical theme of Lambda Island. Through several design iterations a beautiful
logo emerged, with day and nighttime versions. Lubov also helped us establish
branding guidelines, including colors and fonts.</p><p><img src="https://img.lambdaisland.com/70f71d5a00574a8d99fb01b2c0810a58.png" alt="The Lambda Island logos, a lighthouse in a round design" /></p><p>Felipe took it from there, creating the UI design, but Lubov would stay on as a
design consultant, meeting with Felipe every week to discuss his progress, and
provide her professional input. We thought this way of collaborating worked
really well. Rather than the designer throwing a Photoshop or Figma document
over the wall for the dev to merely implement, this allowed Felipe to bring his
own design sensibilities to bear, as well as his knowledge of the specific needs
of the platform, accessibility and responsiveness foremost. Meanwhile he got to
see how a trained designer reasons about these things. This helped Felipe to
grow tremendously as a frontend developer and a UI designer throughout this
project.</p><p>One source of inspiration were playing cards and collectible card games, and
this is perhaps the most distinguishing feature of this new look. Instead of a
long list, episodes are now smartly presented as cards in a grid. We also added
some simple controls for filtering the episodes—a small quality of life
improvement—making it easier to find what you are looking for.</p><p><a shape="rect" href="https://lambdaisland.com/episodes"><img src="https://img.lambdaisland.com/311ce20470cb5392d365131b7323343c.png" alt="The "cards" interface for episodes" style="margin: 2rem 0" /></a></p><p>The result is a design that feels contemporary, while avoiding the design
clichés of the day. We didn’t want to end up looking like every other website
out there. Pages are smaller and load faster, making the result feel snappy and
improving SEO. The site now has much improved keyboard navigation; an
accesibility improvement that not everyone will notice, but should be much
appreciated by the people who do.</p><p>On a technical level we also needed a reboot. The original site was styled with
Semantic UI and a bit of custom CSS, later changes and additions were done with
Tachyons. We had a potpourri of different design systems, and were no longer
excited about any of them.</p><p>Instead we adopted a less-is-more attitude, writing plain CSS using
<a shape="rect" href="https://github.com/noprompt/garden">Garden</a>. Browsers have come a long way, and
modern CSS lets you achieve a lot with little. Still, as the project grew we
started feeling the need for an organizational system to help us manage the
styling for the project. We looked at existing projects, and synthesized all the
good ideas we saw into a CSS-components-in-Clojure library which we dubbed
Ornament.</p><p>We are really happy how Ornament turned out, and hope to release it to the
public before long. It combines ideas from CSS-in-JS, BEM, utility classes,
design tokens; and builds on Garden and Girouette. We’ve since started using
Ornament on client projects and continue to improve and refine it.</p><p>As before the site is mainly rendered on the backend. This approach of course
doesn’t work for everyone, and there are times when using the web as an
application platform instead of a document platform makes sense, but in this
case sticking to a traditional HTML-on-the-server approach only presented
benefits. There’s a bit of ClojureScript in there, sprinkles with simple DOM
manipulation, but that’s it.</p><p>We’re not fully done yet, at the time of writing the main thing left in the old
design is this blog, and there are some lesser used pages like the 404 and 500
pages which still need an update. But sometimes you have to ship, and then keep
shipping. We’ll freshen up the blog before long, and start adding some more
improvements as we go along, like better author attribution.</p><p>The Gaiwan team is going on a summer break to recharge, so we wanted to get this
out before we sign off. We hope you like what you see! We’d love to hear what
you think, and answer any questions you may have <a shape="rect" href="https://discord.gg/pCuZBDRruW">on our
Discord</a>.</p></<>>Lambda Island Open Source Update July 2021https://lambdaisland.com/blog/2021-06-29-lioss-update-june-20212021-07-19T15:00:55+00:00<<>><p><em>By Alys Brooks</em></p><p>It’s been a while since our last update! Community contributors and Lambda
Island team members have been busy working on new projects and improving
existing ones.
We’re trying to strike a balance between venturing
into new territory while ensuring existing projects continue to improve.</p><p>If you want to join in, check out our <a shape="rect" href="https://lambdaisland.com/blog/2021-01-21-call-for-contributions-to-lioss">blog
post</a>
calling for contributions, or jump right in to our <a shape="rect" href="https://github.com/lambdaisland/kaocha/issues?q=is%3Aopen+is%3Aissue+label%3A%22first+issue+only%22">first
issues</a>
list.</p><h1 id="a-new-project-fetch">A New Project: Fetch</h1><p><a shape="rect" href="https://github.com/lambdaisland/fetch">Fetch</a> is a lightweight ClojureScript
library over the JavaScript fetch API. Besides saving you the trouble of
JavaScript interop, it returns a Ring-style response back from the Promise and
is already integrated into Transit for you. It’s still new, but we’re excited.
We’ve already started using it for our work with clients.</p><h1 id="in-beta-deja-fu">In Beta: Deja-Fu</h1><p>We’ve created a new library, <a shape="rect" href="https://github.com/lambdaisland/deja-fu">deja-fu</a>,
for managing local date times in JavaScript, intended as an alternative to
js-joda, <a shape="rect" href="https://github.com/juxt/tick">juxt/tick</a>, and other large libraries for
apps where time management isn’t a central concern. Besides being lightweight,
it boasts an ergonomic API, allowing you to manipulate datetimes as though they
were maps.</p><h1 id="friendlier-and-more-advanced-kaocha">Friendlier and More Advanced: Kaocha</h1><p><a shape="rect" href="https://github.com/lambdaisland/kaocha">Kaocha</a>, the most advanced Clojure test
runner, continues to get not only more advanced but also friendlier. Here are
some of the recent changes:</p><ul><li>Adds <code>.gitignore</code> support to <code>--watch</code>, allowing you to skip extra test runs.</li><li>Option to disable <code>deep-diff</code> in cases where you want to copy the output
into a source file or REPL.</li></ul><p>Many of the improvements have been adding error messages. A clear error message
that tells you exactly what you need to do can save as much time as a nifty new
feature or performance improvement.</p><p>We’ve also shored up the docs, including new docs for the Orchestra and Preload
plugins in September by Arne.</p><h1 id="consistent-cross-platform-logging-with-glgi">Consistent cross-platform logging with <code>glögi</code></h1><p>From the start, <a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a> has allowed you
to do <code>pedestal.log</code>-style logging in ClojureScript. However, this still meant
using a different library in Clojure and ClojureScript files. Now, the new
namespace <code>lambdaisland.glogc</code> is usable across <code>.clj</code>, <code>.cljs</code>, and <code>cljc</code>
files.</p><p>We also made fixes to maintain compatibility with newer versions of the Google
Closure library. I’m calling this out in part as a reminder to keep up with
changes to the library.</p><h1 id="smaller-changes">Smaller changes</h1><p>We’ve made smaller changes across our other projects:</p><ul><li><a shape="rect" href="https://github.com/lambdaisland/garden-watcher"><code>Garden-watcher</code></a> now has
support for <code>*.cljc</code> files.</li><li><a shape="rect" href="https://github.com/lambdaisland/regal"><code>Regal</code></a> now supports forms with a
fixed number of repeats, written as
<code>[:repeat form num]</code>.</li><li>We’ve expanded the areas we test, including Java 16 to our test matrices and
ensuring tests run correctly on Windows.</li></ul><h1 id="community-projects">Community Projects</h1><p>Kaocha has <a shape="rect" href="https://cljdoc.org/d/lambdaisland/kaocha/CURRENT/doc/readme">extensive support for
plugins</a>, so we’re
excited when people take advantage of it and scratch their own itch.</p><h2 id="try-try-again-andreacrottis-retry-plugin">Try, Try Again: AndreaCrotti’s Retry plugin</h2><p>Flaky tests are ones that do not pass or fail consistently, meaning that
rerunning the test can be enough for it to go from red to green or from green to
red. Usually you want to fix these so they are consistent, but some tests might
be necessarily flaky or you may want to save fixing them for a later date. In
the meantime, you can use AndreaCrotti’s <a shape="rect" href="https://github.com/AndreaCrotti/kaocha-retry">Retry
plugin</a> for Kaocha.</p><p>It reruns failed tests a configurable number of times, letting you know which
tests failed even with the retries and which ones succeed only after retrying
them.</p><h2 id="kaocha-in-vim-liquidzs-plugins">Kaocha in Vim: Liquidz’s Plugins</h2><p>While the <code>--watch</code> feature and REPL support make it pretty convenient to test
as you code, it can always be faster. If you code in Vim and use the Iced
plugin, Liquidz’s handy <a shape="rect" href="https://github.com/liquidz/vim-iced-kaocha">vim-iced-kaocha</a>
plugin lets you run tests right from Vim, adding to Vim-Iced’s useful toolbox of
shortcuts.</p><h1 id="dont-be-a-stranger">Don’t Be A Stranger</h1><p>We’d love to have your help on these projects. To reiterate, if you want to join in, check out our <a shape="rect" href="https://lambdaisland.com/blog/2021-01-21-call-for-contributions-to-lioss">blog
post</a>
calling for contributions, or jump right in to our <a shape="rect" href="https://github.com/lambdaisland/kaocha/issues?q=is%3Aopen+is%3Aissue+label%3A%22first+issue+only%22">first
issues</a>
list.</p><p>Even if you can’t or don’t want to help out, we’d love for you to
<a shape="rect" href="https://discord.gg/eXNrT6A6">join the new Lambda Island Discord</a>. We hope that
it will be an enjoyable and inclusive community for Clojurians of all experience
levels. (If the link expires, you can message Arne (Arne#3086), Mitesh
(ox#1485), or myself (alysbrooks#2195) for an invite.)</p></<>>Lambda Island is Changinghttps://lambdaisland.com/blog/2021-06-17-lambdaisland-is-changing2021-06-17T05:33:24+00:00<<>><p>Last month marked the five year anniversary of Lambda Island. Five years since I
quit my job, built a video platform, and figured out how to put together
tutorial videos. It’s been quite a ride. All this time I’ve been fortunate to be
part of the Clojure community, to watch it grow and evolve, and to help
individuals and companies to navigate these waters.</p><p>I learned some hard lessons along the way. Like how hard it is to bootstrap an
educational content business catering to a niche audience, or how lonely and
stressful it can be in business to go it alone.</p><p>And with these lessons came changes. I started doing more consulting work again.
To some extent it was a necessity, but it also provided me with an opportunity
to get involved with many Clojure projects out there in the wild. To work with
talented individuals, to learn what amazing things people were doing, and to
help shape these companies, products, and the greater narrative around Clojure
as a technology and community.</p><p>Out of this work grew many open source libraries and tools, like
<a shape="rect" href="https://github.com/lambdaisland/kaocha">Kaocha</a>,
<a shape="rect" href="https://github.com/lambdaisland/regal">Regal</a>,
<a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a> and
<a shape="rect" href="https://github.com/lambdaisland/corgi/">Corgi</a>, which became part of the Lambda
Island brand, and indispensible items in people’s toolbelts.</p><p>But I was still going it alone, and I didn’t like that, so in 2019,
<a shape="rect" href="https://gaiwan.co/">Gaiwan</a> was born, a bona fide Clojure shop, and slowly but
surely I started putting together a team. Fast forward to today. We are now a
team of 6, spread around the globe, working with amazing clients like
<a shape="rect" href="https://nextjournal.com/">Nextjournal</a>, <a shape="rect" href="https://www.runeleven.com/">Eleven</a>,
<a shape="rect" href="https://pitch.com/">Pitch</a>, <a shape="rect" href="https://itrevolution.com/">IT Revolution</a>, and
<a shape="rect" href="https://www.a-knowledge.eu/">A-Knowledge</a>.</p><p>We’ve been busy! And not just with client work. We’ve continued to grow and
improve our open source offerings. We’re still running
<a shape="rect" href="https://clojureverse.org/">ClojureVerse</a> and
<a shape="rect" href="https://clojurians-log.clojureverse.org/">Clojurians-log</a> for the
community. Our blog has gotten regular
updates. We’ve reached a lot of people on YouTube during the <a shape="rect" href="https://www.youtube.com/playlist?list=PLhYmIiHOMWoGIMCmCRwMSrWkHJg12vevR">Advent of
Code</a>,
and we even organized a conference, <a shape="rect" href="https://heartofclojure.eu/">Heart of
Clojure</a>.</p><p>What we haven’t been doing is posting new video content to Lambda
Island. I could say that with everything else
happening it has been nearly impossible for me to carve out the time to work on
new episodes. That’s not wrong, but the real issue is that I burned out on
making them. I would easily spend a full week on a 15 minute episode, slaving
over the script, agonizing over the details, and meticulously editing the final
result. Then it would go live and…crickets. Freebies would maybe get a few
comments on Reddit and a hundred views. Paywalled episodes would get a few
dozen. Neither would really bring in new revenue, and I’d be scrambling
afterwards to catch up with client commitments.</p><p>In hindsight it was a pretty toxic feedback loop. I suffered under my own
perfectionism, and with that continued to set the bar high for each subsequent
episode.</p><p>But this isn’t a story of doom and gloom. Instead it’s one of necessary change.
We’re a team now, and everyone on the team is excited to help plot the future of
Lambda Island. We don’t know exactly where we’ll end up, but we know the general
direction we’re setting out in.</p><p>One thing that is clear is that Lambda Island is going to start looking and
sounding a bit different. Before, Lambda Island was very much my voice,
literally and figuratively. One person’s style of writing, level of experience,
and view on technology. Going forward we are going to be a multitude of
voices. The first place you are going to notice this is on this blog, where both
Ariel and Alys have already started publishing <a shape="rect" href="https://lambdaisland.com/blog/2021-06-04-clojure-beginners-just-like-vegans-searching-for-good-cheese">some
articles</a> <a shape="rect" href="https://lambdaisland.com/blog/2021-01-21-call-for-contributions-to-lioss">of their
own</a>.</p><p>Meanwhile Felipe—with help from Ariel—has been hard at work behind the scenes to
give the site a major upgrade, the biggest overhaul the site has seen since we
launched in 2016. The result should look and feel much more fresh and modern, be
faster, more responsive and accessible, and allow for better attribution of
authors.</p><p><img src="https://img.lambdaisland.com/4aa9d3fa1c0865fbe35849911ea8ce4a.png" alt="Front page of the newly redesigned Lambda Island website" /></p><p>We also know that we want to put more effort into fostering a vibrant community
around Lambda Island. One that values curiosity, creativity, and exploration;
and where beginners can feel welcome and supported. We set up a Lambda Island
community on Discord and started inviting a few dozen individuals while we set
things up and get a feel for the place. We’ll soon be sending out more invites,
starting with Lambda Island subscribers and to our open source <a shape="rect" href="https://opencollective.com/lambda-island">supporters on
Open Collective</a>. The whole Gaiwan
team hangs out there as well, and so going forward this will be the best way to
get in touch with us and get involved in conversations. Don’t be a stranger!</p><p>As for video content… we don’t know yet. The most likely scenario is that you
are going to start seeing more uncut and unscripted content, like the Advent of
Code videos, instead of the highly condensed and polished episodes from before.
Unfortunately Pascal’s shorter letter applies here as well… I would have made
a shorter video, but I did not have the time.</p><p>It does seem a lot of people enjoy these uncut videos, bloopers included. You
get to really see how someone works in realtime, how they use their tools and
think through problems. For a language with a highly idiosyncratic and
interactive workflow like Clojure this is invaluable.</p><p>Lambda Island is changing. Change can be a little uncomfortable, but it’s a
constant of life. With this post I want to pave the way for the rest of the team, so that
they can be comfortable making their mark. We’ll talk in more details about some of
the things we are up to in upcoming posts. Meanwhile if you’d like to reach out
then you can contact me on Discord as <code>Arne#3086</code>. Don’t be a stranger!</p></<>>Why are Clojure beginners just like vegans searching for good cheese?https://lambdaisland.com/blog/2021-06-04-clojure-beginners-just-like-vegans-searching-for-good-cheese2021-06-08T06:34:01+00:00<<>><p>By Ariel Alexi</p><h2 id="have-you-ever-wondered-what-to-do-if-you-love-cheese-but-you-want-to-be-a-vegan-and-how-this-affects-you-when-you-learn-clojure">Have you ever wondered what to do if you love cheese but you want to be a vegan and how this affects you when you learn Clojure?</h2><p>This was my question too. I love cheese, but I wanted to find a good vegan cheese replacement for my breakfast sandwich.
This wasn’t a fun journey, all the vegan cheese substitutes weren’t like the real deal!
After a talk with one of my vegan friends, he told me that my point of view over cheese replacement in the sandwich was wrong.
Instead of searching for a good and tasty vegan cheese replacement, just replace it with an avocado.
Avocado is a good replacement for your morning sandwich without the compromising that you would do with tasteless vegan cheese substitutes.</p><p>In my opinion, there is a similarity between a Clojure beginner and a vegan trying to find a good replacement for cheese. Instead of enjoying what the vegan world has to offer, a lot of vegans are constantly on the lookout for the synthetic cheese substitute that is like real cheese. Between you and me, most of these substitutes are not tasty.
I wonder, why do those vegans keep looking for the things that are “similar to…” instead of looking for the things that are delicious in the vegan world?
Why not accept that in the vegan world there is no cheese or something as tasty as really good cheese, but you can just enjoy what you do have, like stir-fried vegetables and <a shape="rect" href="https://en.wikipedia.org/wiki/Hummus">hummus</a>?</p><h3 id="what-were-my-resources-and-what-happened-to-me">What were my resources and what happened to me?</h3><p>I started just like everyone else. I started to learn Clojure from the books (<a shape="rect" href="https://www.braveclojure.com/clojure-for-the-brave-and-true/">Clojure for the Brave and True</a> by Daniel Higginbotham and <a shape="rect" href="https://www.oreilly.com/library/view/clojure-programming/9781449310387/">Clojure Programming: Practical Lisp for the Java World</a> by Chas Emerick, Brian Carper, and Christophe Grand). I also practiced some coding questions from <a shape="rect" href="https://www.4clojure.com/">4Clojure</a>. While some other people found all these resources helpful, I felt like I was just learning a lot of words, instead of learning how to create sentences in this new language - Clojure.</p><p>When I wanted to write something, all the little examples I saw were nothing like what I wanted to do. They showed different algorithms such as changing strings or sorting lists, etc.
To solve coding problems in Clojure I was trying to force in concepts from other languages that I was familiar with, as it was in my journey hunting for new cheese. I was forcing myself to find a new kind of cheese.
Much like the lack of taste in the substitutes, I was missing something in those code examples and sources.</p><p>I never saw the use of maps, channels, or other things which are widely used in Clojure when you start to develop something like a server.</p><h3 id="what-eventually-helped-me-more-than-anything">What eventually helped me more than anything?</h3><p>The first time I was able to understand Clojure and write code myself was when I had to add new features to a program already written in Clojure and not write something from scratch. The very fact that I could change something in simple pre-existing code allowed me to see how things are written, and to continue through trial and error and debugging to understand the language more deeply.</p><div style="text-align: center;"><p><strong>This was my AVOCADO!</strong><br clear="none" /><img width="400" src="https://media.giphy.com/media/lKQ4k6JMgf1OoeIcVf/giphy.gif" /></p></div><h3 id="so-what-was-my-avocado-you-must-have-been-wondering">So, what was my avocado you must have been wondering.</h3><p>For me, I went back to the origins. I mean, I thought about how I have learned new coding languages at the university through coding projects.
That is why I have decided on a project that I will be excited about, and at the same time will be able to build in steps. This means that in each step everything must work and I can understand what the next step will be that I need to take.</p><p>I searched for a basic game I could work on. I created a 2D desktop game based on <a shape="rect" href="https://github.com/oakes/play-clj">Oakes/play.clj</a>, which I converted to a Pac-Man game. This was my avocado.</p><p>Creating a game allowed me not just to practice different Clojure syntax but also to run the game and check if the functionality works on the screen right away. Which means having a visual addition and not just evaluating functions on the REPL.</p><p>Seeing the whole game created in small steps and not evaluating algorithms is just as good as avocado in the vegan world which is more than just a synthetic cheese.</p><p>In addition, I think that without functional programming expertise it’s hard to start coding Clojure from scratch, but when you have a simple program with examples, and at each step, you know what you want to change, then you can see right away if it works or not. I found it much more helpful to start from working code, and then change it to what I want it to be, than to write it from scratch.</p><p>Bear in mind, like a work of art - it’s very difficult to start from scratch, but once you have an example it’s easier to go from there.
If you want to try this approach all you need to do is to think about what you want to learn - client or server-side or even maybe games, then find a sample code on the internet that is not more than 50 lines of code and start adding features.</p><p>If you ask me then my advice would be that when you start to learn Clojure, start from example code. If you want to learn Clojure and not just <code>Ctrl-F</code> or to Google all the time “how to… in Clojure”, you should be learning it by using other Clojure code itself.
Remember, if you want to develop in Clojure, you don’t need to be a vegan searching for the best cheese substitute, you just need to use what already exists and enjoy your avocado…</p><p>Good project hunting,
Good luck!</p></<>>The beginner's wayhttps://lambdaisland.com/blog/2021-04-05-things-i-have-learned-in-my-1st-month-of-clojure2021-04-05T18:12:16+00:00<<>><p>By Ariel Alexi</p><h2 id="an-oop-developer-finding-her-way-in-the-functional-world-what-could-go-wrong">An OOP developer finding her way in the functional world, what could go wrong?</h2><h2 id="so-why-clojure">So why Clojure?</h2><p>This was my main question to every Clojure developer that I have met, while I was wrestling with myself about the answer to that question. During my studies for a computer science degree, I had the chance to code in various languages and try different technologies.
Some of these languages were: OO languages (Python, Java, C++, and C#) and C. Although I am a junior OOP developer, developing into the functional world was always fascinating and interesting. That is why I had the question why from all the functional languages, I need to try and code in Clojure? While I was asking this question to different people, the answer was most often similar “I don’t know but..” and later on there was always the sentence of “…you have to try it and you will understand!”.</p><p>So, challenge accepted!</p><p>I am going to find out if Clojure makes it or breaks it for me. What could possibly go wrong? I am starting a new journey, a journey I think every junior developer must take - CODE AND TRY! Only by tapping away at the keyboard and trying to solve tasks, you learn what you want to do, and also what you never want to do again.</p><p>Bear in mind that I’m an Object-oriented programming (OOP) developer diving deep into the functional world, wish me luck!</p><div style="text-align: center;"><p><img width="400" src="https://img.lambdaisland.com/3150a148fe45f0cf84dade82fabbdd7b.gif" /></p></div><h2 id="what-is-clojure-and-why-should-we-care">What is Clojure and why should we care?</h2><p>To quote Clojure.org:</p><blockquote><p>Clojure is a dynamic, general-purpose programming language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language, yet remains completely dynamic – every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.</p></blockquote><h2 id="where-to-start">Where to start?</h2><p>Clojure was the first language that I had to learn outside of any course at the university and don’t forget it is a <a shape="rect" href="https://en.wikipedia.org/wiki/Functional_programming">functional language</a>, YAY.
It was me (with some help from my mentors) vs. Clojure.</p><h3 id="books">1) Books:</h3><p>There are different books I have tried reading during my study —</p><ul><li><a shape="rect" href="https://www.braveclojure.com/clojure-for-the-brave-and-true/">Clojure for the Brave and True</a> by Daniel Higginbotham</li><li><a shape="rect" href="https://www.oreilly.com/library/view/clojure-programming/9781449310387/">Clojure Programming: Practical Lisp for the Java World</a> by Chas Emerick, Brian Carper, and Christophe Grand</li></ul><p>I found that I enjoyed reading the first book. It was a funny and interesting book for a software self-learning book. Along with the book, you get some funny and interesting assignments and read stories to make the journey more interesting. This book is not for everyone, but give it a try. If it is not your cup of tea try the second one, you can always learn in the direct and straightforward (but boring in my opinion) way with the <a shape="rect" href="https://clojuredocs.org/">Clojure documentation</a>.
From my point of view, you should not learn a new language by its docs. I think you should use the language docs only when you are familiar with things and you are searching for some more small details.</p><h3 id="videos">2) Videos</h3><p>A quick intro for those who don’t know: Rich Hickey is a computer programmer and a speaker, known as the creator of the Clojure programming language.</p><p>As a part of my journey in discovering Clojure, I had an assignment to watch one of Rich Hickey’s videos every week. It may sound weird but give yourself an hour to watch them every week. Part of learning Clojure is to understand where and why we need and want it.
Rich Hickey’s talks will give you a different perspective about Clojure and coding.
In the beginning, I didn’t understand what everyone wanted from me, I had no opinion about them but as I continued watching more videos I started to consolidate my own opinion on the videos and the coding world as a programmer.</p><p>This is the list of Rich Hickey’s videos I watched, in the order I watched them:</p><ul><li><a shape="rect" href="https://www.youtube.com/watch?v=ScEPu1cs4l0">Are We There Yet? (2009)</a> — Rich asks whether we’re being well-served by object orientation as it is commonly embodied.</li><li><a shape="rect" href="https://www.youtube.com/watch?v=-I-VpPMzG7c">The Value of Values (2012)</a> — Rich compares place-oriented programming to programming with values.</li><li><a shape="rect" href="https://www.infoq.com/presentations/Simple-Made-Easy/">Simple Made Easy (2011)</a> — Rich discusses the definitional difference between simplicity and ease.</li></ul><p>I think the 3rd video is a must for every developer out there, no matter if you want to use Clojure or not.</p><p>You can find a complete list of <a shape="rect" href="https://github.com/tallesl/Rich-Hickey-fanclub#talks">here</a> as well as detailed <a shape="rect" href="https://github.com/matthiasn/talk-transcripts/tree/master/Hickey_Rich">transcripts</a> for most of them.</p><h2 id="how-was-it-to-learn-clojure">How was it to learn Clojure?</h2><p>As an OOP developer, my first few days with Clojure were something like Conan O’Brien crying</p><div style="text-align: center;"><p><img width="400" src="https://img.lambdaisland.com/fe9d9d977924734f3ee736270ffe0fc5.gif" /></p></div><p>News flash: learning a new language is hard, especially if you are a junior developer. It is difficult because learning a language that is so different from the concepts and fundamentals you already know is hard, so do give yourself some slack.</p><p>As an OOP developer, I needed to change my mindset about this new thing and its concept which means - in Clojure there are <strong>NO objects!</strong> (at least not that I recognized).</p><p>You only get data structures and functions, and it’s not an easy task for an OOP developer to throw objects out the window and act like it is all fine and easy.
On the bright side, Clojure has a small box of functions and data structures you need to get familiar with and you just need to know how to manipulate them.</p><p>But don’t you worry - my opinion about Clojure has changed. Now I mostly feel like this when I’m coding:</p><div style="text-align:center"><p><img width="400" src="https://img.lambdaisland.com/5709ed692053037b5d2fbcbea2cf2334.gif" /></p></div><p>You must have been wondering how I have changed my mind over my first month, but this I shall not reveal now. All that I wrote here, the books, and the videos are the first baby steps that usually Clojure developers take before they start to code and practice the examples in the books or the websites. Those are just the ordinary baby steps that people usually take, whereas I believe you should do it differently.</p><p>To learn more, you should read my next post.</p><p>See you there.</p></<>>Call for Contributions to Lambda Island Open Sourcehttps://lambdaisland.com/blog/2021-01-21-call-for-contributions-to-lioss2021-01-25T19:55:08+00:00<<>><p><em>By Alys Brooks</em></p><p>We’re always excited when someone new contributes a fix, test, documentation update, or even a major new feature to our open source projects.
If you’re new to programming, open source projects are a great way to get experience working on a larger project.
If you’re not so new, they can be an opportunity to try a new technology or work on a kind of software you usually don’t get a chance to.
Either way, it’s rewarding to help out your fellow users or developers—these issues are for Kaocha, one of the most popular and advanced Clojure test runners.</p><p>But open source can also be intimidating. If you haven’t been sure where to start, then this post is for you! We’ve outlined the process step by step, and we’re here to support you every step of the way.</p><p>Below are some issues we’ve specifically picked for newcomers that have roadmaps to guide your development.
If you get stuck, we’re happy to answer questions in the Clojurians Slack or on your PR. After you <a shape="rect" href="http://clojurians.net/">Sign up for Clojurians</a> you can find us in the <a shape="rect" href="https://app.slack.com/client/T03RZGPFR/C1DV21X9P">#lambdaisland</a> and <a shape="rect" href="https://app.slack.com/client/T03RZGPFR/CCY2V0U6A">#kaocha</a> channels.
We’re also available for a quick call to help orient you.</p><p>But who are we? Lambda Island provides educational materials on Clojure web
development, testing, and many other topics, plus we create open source projects like <a shape="rect" href="https://github.com/lambdaisland/kaocha">Kaocha</a> and <a shape="rect" href="https://github.com/lambdaisland/deep-diff2">deep-diff2</a>, which produces clear diffs for Clojure data structures.</p><p>Here’s how you get started:</p><ol><li>Pick an issue from the below list.</li><li>Confirm that it isn’t assigned and leave a comment saying you’re working on it.</li><li>Review the <a shape="rect" href="https://github.com/lambdaisland/kaocha#contributing">guidelines</a>.</li><li>Create a <a shape="rect" href="https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/fork-a-repo">fork</a>.</li><li><p>Clone the repository:</p><p>Go to your newly created fork on Github (this should look like
<code>https://github.com/[YOURUSERNAME]/kaocha</code>), and look for the big green
“Code” button. This will give you the URL that you can pass to <code>git clone</code>.</p><p>You have the option to use HTTPS or SSH, which you can read about <a shape="rect" href="https://docs.github.com/en/free-pro-team@latest/github/using-git/which-remote-url-should-i-use">in Github’s documentation</a>. Using SSH can be convenient but requires some configuration. When in doubt, just pick HTTPS.</p><pre><code class="language-sh">git clone https://github.com/[YOURUSERNAME]/kaocha.git
cd kaocha
</code></pre></li><li><p>Create a branch.</p><pre><code class="language-sh">git checkout origin/main -b <your-branch-name>
# Branch 'your-branch-name' set up to track remote branch 'main' from 'origin'.
# Switched to a new branch 'your-branch-name'
</code></pre><p>Pick a nice desriptive branch name like <code>fix-widget-interaction-issue</code> or <code>focus-command-line-narrows-suite-definition</code>.</p></li><li><p>Have a look around, and then make your changes.</p><p>Kaocha’s implementation lives under the <code>src</code> directory, while the <code>test</code> directory contains our test suite for testing Kaocha itself. The issue you picked should contain details on where to look to get started, but feel free to just open some files at random and have a look around. You won’t understand everything (or perhaps, anything), but that’s ok. Just get a feel for what is there, and try to imagine how some of it might fit together.</p><p>You can start by running <code>bin/kaocha</code>, which will cause Kaocha to test itself. This is a good starting point to make sure your environment is correctly set up. Next you can try to “jack in” with your editor so you are able to evaluate code. Once you are connected, you can start coding.</p></li><li><p>Test your changes. There are a couple of options:</p><ul><li>Using your REPL or interactive environment. This should be the first place
to start; try calling the functions you are working on directly to see if
they work. This isn’t always trivial since sometimes quite a bit of setup
is needed, but it’s the best way to get quick feedback. We’re happy to help
figure this part out. If you can get this far, then adding unit tests for
your changes will be easy too. (Or at least, easier!)</li><li>Use the <code>kaocha.repl</code> namespace to run some (or all) tests. This works on
two levels, this runs Kaocha, and so you might see your new code in action
that way, it also runs tests which test various parts of Kaocha.</li><li>You can also run <code>bin/kaocha</code> again from a terminal. This ensures you are
running from a clean slate, regardless of the state of your REPL. You may
want to use the <code>-c</code> option to specify a custom <code>tests.edn</code> file that you
can add any newly created options to or tweak the options to produce the
behavior you’re trying to fix. You can run <code>bin/kaocha unit</code> to just run
the unit tests, which is faster than running the entire suite including
integration tests.</li><li><p>Use your version of Kaocha on an Clojure project you already have, or on a
new “test” project you create. With <code>deps.edn</code> this is very
straightforward:</p><pre><code class="language-clojure">;; deps.edn in the new project
{:deps {lambdaisland/kaocha {:local/root "/path/to/where/you/cloned/kaocha"}}}
</code></pre><p>Follow the <a shape="rect" href="https://cljdoc.org/d/lambdaisland/kaocha/1.0.732/doc/2-installing">Kaocha documentation</a> to set up this test project with Kaocha (<code>tests.edn</code>, <code>bin/kaocha</code>, and <code>test/*.clj</code> files).</p><p>If you want to use Kaocha on a project that doesn’t use <code>deps.edn</code> but instead uses <code>project.clj</code> (Leiningen) or <code>build.boot</code> (Boot), then you need to build and install Kaocha locally first. Make sure you have Maven installed:</p><pre><code class="language-sh">$ mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.7, vendor: GraalVM Community, runtime: /home/arne/opt/graalvm-ce-java11-20.1.0
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.8.0-34-generic", arch: "amd64", family: "unix"
</code></pre><p>Then in the Kaocha directory run <code>mvn install</code>. This will install a local
copy of Kaocha, including your changes. Make note of the version, and use
exactly that version in the test project. Since this installs a copy into
your local Maven cache, you will have to rerun <code>mvn install</code> every time
you make changes to install a version with your changes.</p></li></ul></li><li><p>Add your changes to git:</p><p>When you’re ready, or you want to share your progress, then it’s time to create a git commit. First review your changes:</p><pre><code class="language-sh"># see which files changed
git status
# see the changes
git diff
# see a summary of the changes
git diff --stat
</code></pre><p>Now add your changes to the git “staging area,” so they are ready to be committed:</p><pre><code class="language-sh">git add <changed files>
</code></pre></li><li><p>Commit your changes:</p><pre><code>git commit
</code></pre><p>This will open your editor, so you can write a commit message. Start with a single line with short description, then leave a blank line, followed by a detailed description of the what and, more importantly, the why of your changes.</p></li><li><p>Push your changes to Github</p><pre><code>git push origin <your-branch-name>
</code></pre></li><li>Follow the link in the message that appears to create a new Pull Request from your changes.</li><li>Write up what you changed and create a pull request.</li></ol><p>If you get stuck at any of these steps, we’re happy to help!</p><p>To help you pick an issue that fits the amount of time you have, the list’s divided up into Quick(ish) Fixes and Features. Keep in mind that sometimes there are unexpected twists and turns in addressing an issue and that your first issue will take a little extra time as you aren’t (yet) familiar with the codebase.</p><h1 id="quickish-fixes">Quick(ish) Fixes</h1><h2 id="provide-a-warning-when-no-tests-are-found-34">Provide a warning when no tests are found #34</h2><p>It’s not always clear why no tests are found. If this happens, there should be a helpful error message to let the user know what might have gone wrong.</p><p>More details are in <a shape="rect" href="https://github.com/lambdaisland/kaocha/issues/34#issuecomment-742573031">this comment</a>.</p><h2 id="unhelpful-error-message-when-ns-patterns-is-a-string-124">Unhelpful error message when ns-patterns is a string #124</h2><p>It’s easy to miss the s in <code>:ns-patterns</code> and just specify a single pattern. If you make this mistake, you get an ambiguous error about class casting. If you add a helpful error pinpointing the configuration mistake, it could be cleared up.</p><p><a shape="rect" href="https://github.com/lambdaisland/kaocha/issues/124">This comment</a> has the details.</p><h1 id="features">Features</h1><h2 id="fix-the-is-trip-up-65">Fix the <code>is</code> trip-up (#65)</h2><p>The <code>is</code> macro is intuitively named. <code>(is (= "string1 "string2"))</code> is kind of like “string1 is equal to string2” or even “Is string1 equal to string?”. But in other programming languages (not to mention English), <code>is</code> by itself has a similar meaning to <code>=</code>, so it’s tempting to write <code>(is "string1" "string2")</code> . But in Clojure tests, writing <code>(is "string1" "string2")</code> won’t do what you want. When called with two arguments, the first argument is (still) the test and the second argument is the message. But values (besides <code>false</code>) are treated as equivalent to <code>true</code>, so you’ve created a test that can never fail!</p><p>We can smooth over this speed bump in Kaocha, since it’s looking at all your tests anyway.</p><p>In short, you’ll need to create a replacement for <code>clojure.test/assert-any</code> and use a technique called monkey patching to replace this library function with your new-and-improved one.</p><p><a shape="rect" href="https://github.com/lambdaisland/kaocha/issues/65#issuecomment-599216092">This comment</a> has the details.</p><h2 id="allow-configuring-skip-meta--focus-meta-at-the-top-level-47">Allow configuring skip-meta / focus-meta at the top level (#47)</h2><p>You can already tag tests with metadata, like <code>:Pending</code> or <code>:Critical</code> and configure tests suites to either skip them or focus on them (run only them). However, you can’t make this configuration global; you’d have to repeat it for each test. You can fill in this gap.</p><p>As a bonus, you could make <code>--skip-meta</code> add to what’s already skipped rather than replace what’s already being skipped.</p><p><a shape="rect" href="https://github.com/lambdaisland/kaocha/issues/47#issuecomment-742581321">This comment</a> has the details.</p></<>>Logging in Practice with Glögi and Pedestal.loghttps://lambdaisland.com/blog/2020-09-28-logging-in-practice-glogi-pedestal2020-09-28T16:03:51+00:00<<>><p>In an <a shape="rect" href="https://lambdaisland.com/blog/2020-06-12-logging-in-clojure-making-sense-of-the-mess">earlier post</a>
I tried to map out the Clojure/Java logging landscape a bit, so people can make
an informed choice about how to set up logging for their application. There are
several solutions out there with different trade-offs, and recently both <a shape="rect" href="https://github.com/ptaoussanis/timbre">Timbre</a>
and <a shape="rect" href="https://github.com/BrunoBonacci/mulog">mulog</a> have had some new releases. However until further notice the default
logging setup in the “Lambda Island Stack” remains pedestal.log on the backend,
and Glögi at the front.</p><p>This blog post is a bit of practical advice on how to make good use of logging,
with the focus on these two libraries, which both provide the same API, just for
a different platform.</p><p>This API consists of a number of macros corresponding with the different log
levels, which all take key-value pairs as arguments.</p><pre><code class="language-clojure">(ns foo.bar
(:require [#?(:clj io.pedestal.log :cljs lambdaisland.glogi) :as log])
(log/info :hello "world" :good "morning")
</code></pre><p>So it looks a lot like a call to <code>hash-map</code>, and generally that’s what it will look like in your logs</p><pre><code>17:51:17.743 [Thread-3] INFO foo.bar {:hello "world", :good "morning"}
</code></pre><p>What follows are a few dos and don’ts that I’ve figured out over time, they work
well for me, maybe there’s something in here that will work for you too. Feel
free to try them on for size.</p><h2 id="start-with-a-single-namespaced-keyword">Start with a single namespaced keyword</h2><p>While you can stick any number of keys at the top level, I often have only one, with a map as value.</p><pre><code class="language-clojure">(log/info :run-loop/starting {...})
</code></pre><p>Note that I don’t use the current namespace as the namespace for the keyword
(<code>::run-loop-starting</code>), since the namespace name already shows up in the logs
separately. Instead I use the namespace part of the keyword to point at some
context, and the name part to point at some event.</p><p>This “context” is often just the name of the function, but it could also be some
other name from your business domain. It needs to provide a person reading the
logs enough info to know what it is we’re talking about. It helps to group
related log messages, and when well chosen it gives you something to grep for.</p><p>The name part (the part after the slash) will usually be a verb. Note that
logging usually happens before or after performing side effects. Sending an
email, doing an API call, adding an event to a queue… Logging that you are
doing some pure computation usually doesn’t add much value.</p><p>That means that you are typically adding a log entry right before something
happens, or after something has finished. In the first case I will use a present
continuous tense (<code>:run-loop/starting</code>), in the latter a past tense
(<code>:run-loop/started</code>).</p><p>It’s fine to have both, in that case you probably want to decide which of the
two is more important to know about, and assign that one a higher log level.</p><p>This is usually the only top-level key I have in there, that way it also
functions as a kind of event identifier to identify specific log entries, making
it easy to filter them after the fact (logs are just data after all).</p><p>The only other top level keys I may add are <code>:exception</code>, since Pedestal.log and
Glögi both look for this key to log the exception, and <code>:message</code>, to include a
human readable message (a string).</p><h2 id="use-the-map-for-inputs-outputs-locals">Use the map for inputs, outputs, locals</h2><p>So what goes into that single map? That’s up to you of course, you want to put
enough relevant data in there so your logs tell you something, but you don’t
want to put huge data structures in there that will result in pages of output.</p><p>When logging before a side effect, try to include any arguments/parameters that
determine what happens. When logging after the effect, also include any return
or success/fail values.</p><p>Try to use the same names in your logging statement that you use in your code.
This makes it much easier to correlate the two later on, and it makes it easier
to copy some values from a log message into a REPL to reproduce some behavior.</p><pre><code class="language-clojure">(defn perform-action [{:keys [color size id]}]
(log/info :action/performing {:color color :size size :id id})
(let [result (do-action color size id)]
(log/trace :action/performed {:id id :result result})
result))
</code></pre><h2 id="layer-your-logging">Layer your logging</h2><p>When coaching people I often see them use logging the way you would use println
debugging. They add <code>log/error</code> statements anywhere where they want to inspect a
value, since that way they don’t need to think about log levels, and when
they’re done they take the logging out again.</p><p>Instead try to place meaningful logging statements that you can leave in. Here
log levels come into play. You want to think of logging of a certain function,
namespace, or subsystem as layers of information. Like layers in photoshop you
can show or hide them to see more or fewer details.</p><p>The bottom layer is stuff that you simply can not ignore, in other words
<code>log/error</code>. Right on top is things that you probably should pay attention to
because it might be a problem, this is <code>log/warn</code>.</p><p>The next layer is <code>log/config</code> (this one is only available in Glögi, not in
Pedestal, for pedestal use <code>log/info</code>). Here you see information about how
things are set up and configured. File system paths, port numbers, environment
variables, etc. Next up is <code>log/info</code>, informational messages about what the
system is doing under the hood. Major subsystems starting or stopping, messages
about current workload. This is usually a good default level.</p><p>Finally we get to the detailed layers, <code>log/debug</code> and <code>log/trace</code>. These you
don’t generally enable for the whole system because that would get too noisy,
but you turn them on for certain namespaces when you are working on them, or
figuring out an issue.</p><p>With <code>log/debug</code> you already want to give a fairly good idea of what’s actually
happening, logging individual requests, events, or whatever the unit of
processing in your system is.</p><p>Finally <code>log/trace</code> opens the floodgates, showing in fine details what is
happening. Here you no longer care much about parsimony, you care about
providing all the details that could be relevant to see exactly what’s going on.
When you do enable this log level it’s probably only for a short amount of time
for a very limited scope, so that you can capture all that information and then
analyze it.</p><p>I hope this advice resonates with people, if you have your own suggestions of
how to approach logging in practice then come find us in the
<a shape="rect" href="https://clojurians.slack.com/archives/C010TGGL02X">#observability</a> channel on
Clojurians Slack.</p></<>>Well Behaved Command Line Toolshttps://lambdaisland.com/blog/2020-07-28-well-behaved-command-line-tools2020-07-28T12:21:33+00:00<<>><p>Yesterday Vlaaad wrote a pretty cool blog post titled <a shape="rect" href="https://vlaaad.github.io/tools-cli-in-10-lines-of-code">Alternative to tools.cli
in 10 lines of code</a>.
It’s a great read and a very cool hack. It uses <code>read-string</code> to parse command
line arguments, resolves symbols, and invokes the first symbol/function, passing
in the rest of the parsed arguments. In other words: it’s the most
straightforward way to translate a command line invocation into a Clojure
function call.</p><p>The benefits are illustrated well in the post. It removes all possible friction.
Want to add a new subcommand? Just add a function. Adding CLI arguments and
options equals adding arguments and options to said function.</p><p>It’s actually not too different from what people do in shell scripts sometimes.</p><pre><code>#!/bin/sh
hello() { echo "Hi, $1"; }
world() { echo "It's big and it's blue"; }
# Expand all arguments to the script, so the first one is interpreted as a
# command
"$@"
</code></pre><p>What I like about Vlaad’s approach is that it establishes clear semantics. If
you’re a Clojure programmer you are familiar with how Clojure’s reader works,
and so your intuition transfers nicely. If you squint it almost looks like
you’re just invoking a function from the REPL.</p><p>So for tools that are either personal/internal, or that are really only meant to
be used by a Clojure audience I think this makes sense. That said I would like
to make a case for <a shape="rect" href="https://github.com/clojure/tools.cli">clojure.cli</a>. Don’t
dismiss it too easily.</p><p>Perhaps it seems a little large for what it does (how hard can command line
argument parsing be, right? … right?), but that’s because it takes care of a
lot of things for you. It makes sure your program follows established
conventions, minimizing surprises for your users.</p><p>With Babashka and GraalVM more and more Clojure friends are venturing into
writing command line utilities. <code>deps.edn</code> is also helping with this resurgance,
since a lot of tools that used to be Leiningen plugins are now simply libraries
with a <code>-main</code> namespace. They all need some form of command line argument
parsing, and many end up rolling their own ad-hoc version, loosely but
inconsistently based on existing UNIX conventions.</p><p>Now UNIX is famously inconsistent as well (looking at you <code>dd</code>). Just like in
Clojure there’s a big desire for retaining compatibility, so older tools tend to
have their idiosyncracies, but generally speaking things are converging towards
common conventions, which are described for instance in the <a shape="rect" href="https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html">GNU libc
manual</a>.</p><p>I should note that these are the GNU conventions, they are not universal, but
they are widely used. GNU in particular pioneered the double dash for long
option names which has become ubiquitous. POSIX (the standard that most UNIXes
adhere to) <a shape="rect" href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html">only mentions single-letter
options</a>.
If you want to read more about the overgrowth of command line styles this
<a shape="rect" href="https://web.archive.org/web/20090503075914/https://stackoverflow.com/questions/367309/which-command-line-commands-style-do-you-prefer">archived StackOverflow
question</a>
has you covered.</p><p>So what are those conventions? First of all a distinction is made between
“options” and “arguments”. Options start with <code>-</code> or <code>--</code>, the rest are
arguments.</p><pre><code>ls --color ./*
^------ argument
^-------------- option
</code></pre><p>Arguments may be positional, that’s up to how the program interprets them, but
options can be supplied in any order typically without changing the
semantics. They can be written before, after, or in between arguments, or
all of the above at the same time. Not dealing with this correctly is probably
the most common “violation” I see out there.</p><p>A high profile example is
<a shape="rect" href="https://clojurescript.org/reference/repl-and-main">cljs.main</a> I use this
example because a lot of people will be familiar with it, and because I think
the ClojureScript maintainers can take this bit of constructive criticism. Their
design choices are of course valid, and they have good reason to avoid adding more dependencies,
I’m merely using this as an illustration of
what it looks like when one deviates from these — admittedly fuzzy — “UNIX
conventions”, because this is actually a case that has tripped me up more than once.</p><p>When invoking <code>cljs.main</code> the <code>--main</code>, <code>--compile</code> or <code>--repl</code> options are
positional, they have to come last, after any “init options”.</p><p>Say you run a command and it’s not clear if it’s succeeded or not, so you want
to run it with <code>--verbose</code>. You press “up” in your terminal and tack on
<code>--verbose</code>. Oops, that doesn’t work. <code>--verbose</code> is an “init option”, it has to
come before the main/repl/compile option… Not very intuitive.</p><p>At this point it’s maybe a good time to say a few words about <code>-</code> vs <code>--</code>. GNU
tools tend to have “short options” that start with a single dash followed by a
single letter, and “long options” starting with a double dash. For example <code>ls
-a</code> and <code>ls --all</code>. The short ones are always a single letter because this
allows combining them in a single chunk, like <code>ls -adR</code> (the equivalent of <code>ls
--all --directory --recursive</code>).</p><p>Options can take associated values. The GNU conventions use <code>=</code> for this, a lot
of tools also just use a space. tools.cli (and a lot of GNU tools it seems) will
understand both.</p><pre><code>ls --format=commas
ls --format commas
</code></pre><p>tools.cli lets you pick both short and long options, but my advise is to start
with only long options. It’s more readable and explicit. If certain options are
used very commonly then look into providing some short versions. By that time
you should have an overview of the different options you’ll be offering, and can
make reasoned decisions about which letter to allocate to which option.</p><p>The only short options I tend to always add are <code>-h</code> / <code>--help</code> and
<code>-v</code>/<code>--verbose</code>. While you’re at it maybe also add a <code>--version</code>.</p><p><code>-</code> and <code>--</code> also have a specific meaning when used by themselves. <code>--</code> acts as
a terminator, anything after it is treated as arguments instead of options,
whether it starts with a dash or not. This is especially useful if your tool
ends up invoking some other tool, and you want to allow passing options through
verbatim. This is another thing that tools.cli gives you for free.</p><p><code>-</code> by itself can often be used instead of a file name, in which case STDIN will
be used as input file. It is also common for tools to read from STDIN if no
input file is specified, which makes it convenient to combine them with UNIX
pipes. But this convention you’ll have to implement yourself, tools.cli can’t
help you with that.</p><p>I hope this post will convince people to not dismiss tools.cli too quickly. For
general purpose tools, especially those with reach beyond the Clojure community
sticking to existing conventions is a Good Thing To Do.</p><p>To finish up here’s a little template to get you started. Don’t be daunted by
the hefty README, you don’t need that much to get started.</p><pre><code class="language-clojure"> (ns org.example.my.tool
(:gen-class)
(:require [clojure.tools.cli :as cli]))
(def VERSION "0.0.1")
(def cli-opts
[["-v" "--verbose" "Increase verbosity" :default 0 :update-fn inc]
[nil "--version" "Print version and exit"]
["-h" "--help" "Print this help information"]])
(defn main [args {:keys [] :as opts}]
;; main functionality goes here, you get a seq of positional arguments, and a
;; map of options.
)
(defn -main [& args]
(let [{:keys [errors options arguments summary]} (cli/parse-opts args cli-opts)]
(cond
(seq errors)
(do
(run! println errors)
(println summary)
(System/exit -1)) ;; non-zero exit code if something goes wrong
(:help options)
(do
(println summary)
(System/exit 0))
(:version options)
(do
(println VERSION)
(System/exit 0))
:else
(main arguments options))))
</code></pre><p><a shape="rect" href="https://www.reddit.com/r/Clojure/comments/hzd3t6/well_behaved_command_line_tools/">Discuss on Reddit</a></p></<>>Logging in Clojure: Making Sense of the Messhttps://lambdaisland.com/blog/2020-06-12-logging-in-clojure-making-sense-of-the-mess2020-06-12T16:55:10+00:00<<>><p><a shape="rect" href="https://img.lambdaisland.com/d2d44e2500e32e9d68ad8ca2e5a6880e.png"><img src="https://img.lambdaisland.com/d2d44e2500e32e9d68ad8ca2e5a6880e.png" alt="" /></a></p><p>You may also like <a shape="rect" href="https://lambdaisland.com/blog/2017-07-26-dates-in-clojure-making-sense-of-the-mess">Dates in Clojure: Making Sense of the
Mess</a></p><p>Logging seems like a simple enough concept. It’s basically <code>println</code> with a
couple of extra smarts. And yet it can be oh so confounding. Even when you
literally do not care a sigle bit about logging you may still be sent down a
rabbit hole because of warnings of things that are or aren’t on the classpath.
What even is a classpath?</p><h2 id="the-logging-wars">The Logging Wars</h2><p>Most of the mess that is logging we inherit from Java. During the first half of
the nillies logging libraries and frameworks were fighting for dominance in
what I will dramatically dub “The Logging Wars”. Around 2006 things started
stabilizing, but the scars from this period are still visible in our dependency
graphs today.</p><p>Here’s a typical example, snippets like this you find in nearly every production
Clojure project. To make sense of it we need to dive into a little bit of
history</p><pre><code class="language-clojure">{:deps
{,,,
org.slf4j/slf4j-api {:mvn/version "1.7.30"}
org.slf4j/jul-to-slf4j {:mvn/version "1.7.30"}
org.slf4j/jcl-over-slf4j {:mvn/version "1.7.30"}
org.slf4j/log4j-over-slf4j {:mvn/version "1.7.30"}
org.slf4j/osgi-over-slf4j {:mvn/version "1.7.30"}
ch.qos.logback/logback-classic {:mvn/version "1.2.3"}
}}
</code></pre><p>The first logging library to gain traction in the Java world was Log4j. Log4j
1.0 was released in January 2000, but its history goes back to 1996, when it was
developed as part of an EU research project called
<a shape="rect" href="http://www.semper.org/">SEMPER</a>.</p><p>Log4j is credited with being the first to introduce the hierarchical loggers, a
concept that would be adopted by nearly every logging library that came after
it, not just in Java.</p><p>Meanwhile Sun Microsystems was working on including logging functionality in the
JDK, <a shape="rect" href="https://www.jcp.org/en/jsr/detail?id=47">JSR-47</a> was initiated in 1999,
and the java.util.logging (JUL) package was included in Java 1.4 in February 2002. The
API has a striking similarity with Log4j.</p><h2 id="facadeoff">Facade/off</h2><p>Because it had taken Java so long to include an official logging API a bunch of
different libraries had sprung up to fill the void. This was kind of annoying,
because third party libraries would all depend on different logging
implementations, making it hard to get a single unified log of what was going on
in your application. Application developers also became more prudent about
coupling their code to a specific logging implementation, given how things were
shifting.</p><p>And thus the idea of a “Logging Facade” was born. A Logging Facade provides an
intermediate interface. Application or library code can use whatever logging
library code they prefer, the logging calls simply get forwarded to the facade.</p><p>The facade in turn will forward these to a specific logging implementation, so
you could swap out Log4j for java.util.logging for Avalon Logkit without having
to change a line of code.</p><p>The first library to popularize this idea was Jakarta Commons Logging (JCL),
which hit 1.0 in 2002, right around the time that Java 1.4 was released. JCL
would later be renamed to Apache Commons Logging.</p><p>Apparently JCL had some shortcomings, and so in 2005 SLF4J was conceived, the
Simple Logging Facade for Java. SLF4J, like JCL, is able to output to various
pre-existing libraries, including Log4j and java.util.logging, but the SLF4J
author also wrote a logging backend that is “native” to SLF4J, Logback.</p><p>SLF4J and Logback are both written by Ceki Gulcu who was already involved way
back with the original Log4j, so supposedly he’s had some time to mull over the
problem space, and you could say that SLF4J is the de facto standard now.</p><p>Given that SLF4J has become somewhat of a standard in the Java world, and that
you can pipe virtually every logging library out there into SLF4J, I would
recommend to center the logging of your application around SLF4J. Add the
necessary adapters to forward all other logging libraries to SLF4J. For the
backend I recommend Logback. That’s how you end up with the snippet at the top.</p><h2 id="logback-mountain">Logback Mountain</h2><p>SLF4J will automatically figure out which backend is available, and use that
one. This means that to make this work reliably you need to make sure there is
one and only one logging backend in your dependencies. A lot of libraries will
indirectly pull in the “nop” backend which is simply a placeholder that doesn’t
log anything. Use <code>lein deps :tree</code> or <code>clj -Stree</code> and add exclusions where
necessary. For example:</p><pre><code class="language-clojure">org.clojure/tools.deps.alpha {:mvn/version "0.6.480", :exclusions [org.slf4j/slf4j-nop]}
</code></pre><p>Some of the libraries that you want to pipe into SLF4J will do their own
auto-detection of where their logs will go, leading to more stuff to exclude.
You probably want to avoid having any of these in your list of dependencies,
directly or indirectly</p><ul><li>commons-logging</li><li>log4j</li><li>org.apache.logging.log4j/log4j</li><li>org.slf4j/simple</li><li>org.slf4j/slf4j-jcl</li><li>org.slf4j/slf4j-nop</li><li>org.slf4j/slf4j-log4j12</li><li>org.slf4j/slf4j-log4j13</li></ul><p>You can try this snippet to see if you need to add any extra exclusions.</p><pre><code class="language-shell">clj -Stree | egrep '^\s*(commons-logging|log4j|org\.apache\.logging\.log4j/log4j|org\.slf4j/simple|org\.slf4j/slf4j-jcl|org\.slf4j/slf4j-nop|org\.slf4j/slf4j-log4j12|org\.slf4j/slf4j-log4j13) '
</code></pre><p>or for Leiningen:</p><pre><code class="language-shell">lein deps :tree | egrep '\[(commons-logging|log4j|org\.apache\.logging\.log4j/log4j|org\.slf4j/simple|org\.slf4j/slf4j-jcl|org\.slf4j/slf4j-nop|org\.slf4j/slf4j-log4j12|org\.slf4j/slf4j-log4j13) '
</code></pre><p>If you are using Leiningen then it’s possible to declare exclusions at the top
level and have them apply to all dependencies, which is quite useful. Have a
look at Stuart Sierra’s <a shape="rect" href="https://github.com/stuartsierra/log.dev/blob/master/project.clj">log.dev</a>
example project for what that looks like.</p><p>To configure Logback all you need is a <code>logback.xml</code> file, which you typically
put under <code>resources/</code>. Do make sure <code>"resources"</code> is part of your <code>:paths</code>
(deps.edn). Leiningen adds it to its <code>:resource-paths</code> automatically.</p><p>This snippet is a good starting point:</p><pre><code class="language-xml"><configuration scan="true" scanPeriod="5 seconds">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
<logger name="org.apache.kafka" level="WARN" />
</configuration>
</code></pre><ul><li>It reloads <code>logback.xml</code> every five seconds, so it’s easy to change the
configuration without having to restart your REPL</li><li>It simply outputs to STDOUT, which is great during development, but also very
common for production app, since whatever cloud platform you are using can
aggregate the logs from there.</li><li>It sets the log level of the root logger to ‘INFO’, so by default you’ll see
errors, warnings, some information output about things like configuration, but
it won’t be too noisy.</li><li>Then you can bump the level up or down for specific namespaces/loggers.</li></ul><h2 id="theres-something-about-clojure">There’s Something About Clojure</h2><p>Note that so far I haven’t mentioned anything yet about Clojure libraries. But
given the above all you need is a pretty lightweight library that outputs to
SLF4J. Ideally said library would be able to log Clojure data structures
directly, so you can reap the benefits of structured logging (logs as data).
Which brings me to my favorite Clojure logging library
<a shape="rect" href="http://pedestal.io/api/pedestal.log/io.pedestal.log.html">io.pedestal.log</a>.</p><p>There are various other Clojure libraries, some of them have appealing
propositions, and you may or may not decide to adopt them in your projects. I
hope that with the background this article provides you can now more easily
assess where they fit in the ecosystem, and what they bring to the table.</p><p>You may also have come across
<a shape="rect" href="https://github.com/clojure/tools.logging">clojure.tools.logging</a>. This is an
attempt to provide a logging facade directly in Clojure. It has support for
various backends, including SLF4J. It provided an idiomatic API for logging from
Clojure early on (2011), without dictating a specific backend. Nowadays I think
there are better alternatives.</p><p><a shape="rect" href="https://github.com/pyr/unilog">Unilog</a> provides a more Clojure-esque
configuration interface to Logback, allowing you to use EDN files instead of
<code>logback.xml</code>.</p><p>Some library authors decided they had enough of this mess, and started over from
scratch, implementing an entire logging stack in Clojure. These include
<a shape="rect" href="https://github.com/ptaoussanis/timbre">Timbre</a> and
<a shape="rect" href="https://github.com/BrunoBonacci/mulog">μ/log</a>. Timbre is particularly
impressive in that it can still act as a backend for SLF4J.</p><p>μ/log pushes the idea of structured logging in a distributed context, where logs
are just another data stream akin to a Kafka topic, and indeed it can log to
Kafka directly, as well as ElasticSearch, Kinesis, or plain files or stdout.</p><h2 id="revenge-of-the-log4j">Revenge of the Log4j</h2><p>After a small decade of stability the Java folks decided to stir things up again
by coming up with Log4j 2. This is really a grand rewrite, and so it’s
unfortunate they didn’t just give it a new name.</p><p>Log4j 2 <a shape="rect" href="https://logging.apache.org/log4j/2.x/">tries to learn</a> from all that
came before it. It separates logging from implementation, is said to have better
performance in multi-threaded applications, has first class support for
data-based log messages (rather than strings), the list goes on.</p><p>I think Log4j 2 holds some interesting potential for Clojure, since it’s well
connected to the rest of the ecosystem (adapters to and from other logging
libraries and facades), and has support for <code>java.util.Map</code> based messages (an
interface implemented by Clojure’s maps). I’d be curious to see Clojure
libraries explore what they can do with this.</p><p><em>Update:</em> Sean Corfield pointed out that they are using clojure.tools.logging with Log4j 2 as the backend, which seems like a good way to get the benefits of Log4j 2 today. Certainly an option to consider!</p><h2 id="conclusion">Conclusion</h2><p>SLF4J + Logback has been my default for some time, and one that I’ve seen used
with success in real world projects, in particular in combination with
pedestal.log. It’s an approach that really leans in to Clojure’s philosophy of
making good use of its host instead of reinventing the wheel, and Logback’s
reload functionality make it a great fit for a REPL driven workflow.</p><p>Timbre however does form a strong contender, particularly for people who prefer
a pure Clojure approach, and with its SLF4J bridge it can still play the same
role as Logback, thus unifying application and library logging.</p><p>This post has been all about Clojure, luckily on ClojureScript things are a
little less messy. The Google Closure library contains <code>goog.log</code>, which offers
an API inspired by Log4j, and <a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a>
offers an idiomatic ClojureScript interface inspired by pedestal.log.</p><p>Happy logging!</p></<>>Lambda Island Open Source Update May 2020https://lambdaisland.com/blog/2020-06-05-open-source-update-may-20202020-06-05T13:46:30+00:00<<>><p><a shape="rect" href="https://twitter.com/flpbar">Felipe</a> and I have been busy again, read all about
what we’ve been up to in this month’s Lambda Island Open Source update.</p><p>We currently have two major projects we are working to get out: Regal and
Kaocha-cljs2. They are both still a work in progress, but we’ve again made some
good progress this month. Regal is not officially out yet but is at the stage
where people can safely start incorporating it into their projects.
Kaocha-cljs2 is a big undertaking, but we’re splitting this work into smaller
self-contained pieces, and two of those saw their first release this month: Chui
and Funnel.</p><h2 id="regal">Regal</h2><p><a shape="rect" href="https://github.com/lambdaisland/regal">Regal</a> is a regular expression Swiss
army knife, built around “regal syntax”, a way of representing regexes as
Clojure data structures, similar to what Hiccup syntax does for HTML.</p><p>For instance the regex <code>#"(\w+)=(\w+)"</code> would look like this:</p><pre><code class="language-clojure">[:cat
[:capture [:+ :word]]
"="
[:capture [:+ :word]]]
</code></pre><p>Regal can convert regal forms to regexes, but also parse regexes back to regal.
It can create generators for use with
<a shape="rect" href="https://github.com/clojure/test.check">test.check</a>, and integrates with
clojure.spec or <a shape="rect" href="github.com/metosin/malli">Malli</a>.</p><p>Regal also knows about different regex flavors, it currently supports
JavaScript, Java <= version 8, or >= version 9, and can parse and generate
regexes for each flavor. This means the semantics of a given regal form are
constant, it will always match the same strings, no matter the platform. It also
means you can do things like convert JavaScript regexes so you can use them in
Clojure.</p><p>All the major pieces of Regal of there, and we don’t expect the syntax of Regal
forms to change going forward. The tricky part is that we have multiple pieces
of code that encode the semantics of these regal forms. The regex generator, the
parser, the test.check generators, as well as specs for validating, and possibly
generating regal forms. These all need to match up.</p><p>Most of the recent changes have been about fine tuning these different
implementations. We might come across a regex that trips up the parser, or cases
where generating and then parsing a regex doesn’t yield the same result.</p><p>So far we’ve dealt with this by having data-driven
<a shape="rect" href="https://github.com/lambdaisland/regal/blob/master/resources/lambdaisland/regal/test_cases.edn">test-cases</a>
that cover all the regex features we support. These are used in multiple ways to
test the different parts of Regal, and we still have some more cases we need to
check for to make sure all our bases are covered.</p><p>But the best way to check that multiple implementations match is through
generative testing. We were unable to do this so far because we weren’t able to
reliably generate regal expressions. Their recursive nature means that
generators easily overflow the stack. However we’ve <a shape="rect" href="https://github.com/lambdaisland/regal/blob/master/src/lambdaisland/regal/spec_alpha.cljc#L12-L33">managed to work around
this</a>,
and so the road to more property-based testing is now open.</p><h2 id="kaocha-cljs2">Kaocha-cljs2</h2><p>Kaocha seeks to be a complete solution to all your Clojure testing needs, and of
course that includes ClojureScript. When we came out with Kaocha-cljs this
drastically lowered the bar for people to start writing tests in ClojureScript.
All you needed was a single line in your <code>tests.edn</code>, and things just worked …</p><pre><code class="language-clojure">#kaocha/v1
{:tests [{:type :kaocha.type/cljs}]}
</code></pre><p>… unless they didn’t. Dealing with ClojureScript is inherintly complex. You
can’t just look for tests on the filesystem and run them. The ClojureScript
compiler needs to be involved to compile your tests to JavaScript, and then a
JavaScript runtime is needed to execute them, and meanwhile Kaocha needs to stay
in control, deciding which tests to run, and collecting their results.</p><p>Kaocha-cljs managed to push that complexity away by building on top of
ClojureScript’s repl-env abstraction, in particular the REPL implementations for
the browser and for Node.js. These repl-envs take care of launching a JavaScript
runtime and communicating with it. They also handle the ClojureScript
compilation, so that we can just send snippets of code their way, and they will
compile them, send them on to the browser or node, and notify us of the result.</p><p>In a way this worked better than we had expected, and many people are quite
content with what kaocha-cljs currently offers, but we had some persistent
complaints.</p><p>People who used Shadow-cljs were out of luck, shadow has its own compilation
pipeline and in particular differs from standard ClojureScript in how it deals
with npm packages. That means that code written for shadow-cljs may not be
compatible with the built-in repl-envs.</p><p>There’s a wide spectrum of how people set up their ClojureScript compilation. We
did allow people to specify their own compiler settings, but they had to be
compatible with what the repl-envs expected. You can’t for instance use advanced
compilation. Given how many problems only show up when using <code>:advanced</code> this is
a legitimate ask, and something we were unable to support.</p><p>Finally the repl-envs internals are opaque and hard to debug. People routinely
had trouble getting them to work headless on CI, without any meaningful feedback
or error message to diagnose the issue.</p><p>So we went back to the drawing board. We started by creating a new ClojureScript
test runner, <a shape="rect" href="https://github.com/lambdaisland/chui">Chui</a>. Chui lets you run
specific tests on demand, dynamically at runtime. This is different from the
runner that ClojureScript ships with, which determines which tests to run at
compile time through macros.</p><p><code>chui-core</code> provides the test runner API. On top of that we built <code>chui-ui</code>, a
browser based UI for running tests and inspecting their results.</p><p>We also built <code>chui-remote</code>, a small components which allows controlling
<code>chui-core</code> over a websocket. This means that by including <code>chui-remote</code> into
your build it will be able to talk to Kaocha.</p><p>This gets us out of the business of invoking the ClojureScript compiler, or of
starting a JavaScript runtime. People can use whatever tool they like
(cljs.main, Figwheel, Shadow-cljs), whatever compiler settings they like, and
whatever runtime they like.</p><p>To facilitate the communication between Kaocha and Chui, and to make it easier
to do things like reuse existing browser tabs, we created
<a shape="rect" href="https://github.com/lambdaisland/funnel">Funnel</a>, you can read all about Funnel
in its <a shape="rect" href="https://clojureverse.org/t/announcing-the-first-release-of-funnel/6023">release announcement
(ClojureVerse)</a>.</p><p>This is all starting to come together. Kaocha-cljs2 is not ready yet for
primetime, but the main parts are implemented.</p><p>All of this does mean that using Kaocha-cljs2 will be a bit more involved in
terms of setup, at least in the beginning. Once the dust settles we may look
into providing conveniences for common scenarios. The big win however is that by
separating these different responsibilities we can cater for many more use
cases.</p><h2 id="glgi">Glögi</h2><p>Last but not least this month also saw a small release of
<a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a>, a ClojureScript logging library
based on Google Closure’s <code>goog.log</code>. Glögi does two things: it provides an
idiomatic logging API inspired by pedestal.log, and it implements a
<code>console.log</code>-based logging backend, which provides synatax highlighting for
ClojureScript data structures.</p><p>This latest releases makes Glögi more suitable suitable for use outside of a
browser, e.g. in Node.js or GraalVM, where CSS formatting for <code>console.log</code> is
not available.</p><p>Glögi is a great lightweight alternative to cljs-devtools for printing
ClojureScript data structures in the console, and contrary to cljs-devtools it
works in (nearly) all browsers.</p><h2 id="closing-words">Closing words</h2><p>That’s it for this months. If you like what we’re doing, or if your company
relies on our projects, then please consider pitching in through <a shape="rect" href="https://opencollective.com/lambda-island">Open
Collective</a>, or by <a shape="rect" href="https://lambdaisland.com/pricing">signing
up</a> for Lambda Island Education.</p><p>Our projects warmly welcome contributions. If you would like to get involved but
you are not sure where to start then get in touch, we are always happy to help.</p></<>>Lambda Island Open Source Update April 2020https://lambdaisland.com/blog/2020-05-04-open-source-update-april-20202020-05-04T07:33:26+00:00<<>><p>With people across the globe isolating themselves it feels sometimes like time
is standing still, but looking back it’s been a busy month again for the folks
here at Lambda Island. Besides all the maintenance releases we
shipped some new features for
<a shape="rect" href="https://github.com/lambdaisland/uri">lambdaisland.uri</a>, our JS logging library
<a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a>, and a new micro-library,
<a shape="rect" href="https://github.com/lambdaisland/edn-lines">edn-lines</a>. We also have some
exciting work in progress, including a brand new browser-based test runner for
ClojureScript, written with shadow-cljs in mind.</p><h2 id="funding">Funding</h2><p>A big shout goes out to two of our clients,
<a shape="rect" href="http://nextjournal.com/">Nextjournal</a> and <a shape="rect" href="https://pitch.com">Pitch</a>. They have
made a lot of this work possible either by funding projects directly, or by
dogfooding our tools and libraries and giving us an opportunity to improve them.
Thank you both for investing in this great ecosystem.</p><h2 id="contributions">Contributions</h2><p>We got again some great contributions this month!</p><p><a shape="rect" href="https://twitter.com/danielwithmusic">Daniel Compton</a> contributed two plugins to
<a shape="rect" href="https://github.com/lambdaisland/kaocha">kaocha</a>, an <code>:orchestra</code> plugin for
instrumenting functions with spec and orchestra, and a <code>:preloads</code> plugin,
useful for preloading files with specs.</p><p><a shape="rect" href="https://github.com/JarrodCTaylor">JarrodCTaylor</a> contributed a fix to
<a shape="rect" href="https://github.com/lambdaisland/deep-diff2">deep-diff2</a> to make
it usable in a browser context, and we’re already using this in Chui to show
nice diffs, similar to how Kaocha does it.</p><p><a shape="rect" href="https://twitter.com/jackrusher/">Jack Rusher</a> contributed a patch to
<a shape="rect" href="https://github.com/clojure-emacs/parseedn">parseedn</a>, our EDN parser/printer
for Emacs lisp, with support for converting dotted pairs to EDN, using similar
heuristics as Emacs’s JSON library.</p><p>Of course we also contribute to the projects we use. This month <a shape="rect" href="https://twitter.com/Helios">Davide
Taviani</a> contributed patches to clj-kondo.</p><p>We absolutely love getting contributions big and small, from first-timers or
Clojure veterans. We’ve started adding instructions about the contribution
process to all of our project READMEs to make it clear what to expect as a
contributor.</p><p>If you are not able to contribute code but you still want to support our
projects then consider pitching in to the <a shape="rect" href="http://opencollective.com/lambda-island">Open
Collective</a>.</p><h2 id="glgi">Glögi</h2><p>In Clojure our go-to logging solution is pedestal.log + logback. Having logs
that are just EDN data makes filtering and processing them trivial, and being
able to fine tune the log level per namespace or hierarchy is invaluable.</p><p>To provide the same experience on the ClojureScript side we created
<a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a>. Glögi detects if you are using
cljs-devtools, so you can navigate and inspect your data, while doing regular
console logging when cljs-devtools is not available.</p><p>We’ve now adding syntax highlighting to the console logging, so you get pretty
colorized log messages, no matter which browser you are using.</p><p><img src="https://img.lambdaisland.com/44082c318f1e53c08827a8961623385b.png" alt="Glogi screenshot" /></p><h2 id="uri">Uri</h2><p><a shape="rect" href="https://github.com/lambdaisland/uri">lambdaisland/uri</a> was the first open
source library we released. An idiomatic and cross-platform library for
manipulating URIs, that sticks closely to the RFC.</p><p>Uri has served us well over the years, but one bit that was still missing was
the manipulation of individual query parameters. According to the URI RFC a
query string can really be anything, so we never attempted to further parse it
into components, but the convention of having pairs separated by <code>&</code> and <code>=</code> has
become so ubiquitous that it made sense to support that directly.</p><pre><code class="language-clojure">(uri/assoc-query "//example.com"
:id 1
:name "Arne")
;;=> #lambdaisland/uri "//example.com?name=Arne&id=1"
(uri/query-map "?foo=%20%2B%26%3D")
;; => {:foo " +&="}
</code></pre><p>We’ve also improved the normalization of query parameters to make sure these
semantics are preserved.</p><h2 id="regal">Regal</h2><p>We’ve continued to work towards a first release of
<a shape="rect" href="https://github.com/lambdaisland/regal">Regal</a>, our regular expression library,
painstakingly going through all Regex features that are supported by (but
sometimes implemented differently on) Java and JavaScript, to make sure we
provide the same semantics on both platforms.</p><p>After talking to Malcolm Sparks from Juxt at ClojureD we realized it would be
valuable to decouple the regex flavor from the runtime platform, so you can for
instance parse or generate JavaScript-flavored regular expressions in Clojure.
The use case Malcolm pointed out is dealing with JSON Schema on the Clojure/JVM
side. What Regal now allows you to do is parse a JavaScript regex to a Regal
expression (Hiccup), then convert that to a Java regex (or vice versa).</p><p>We’ve also been fine tuning our generators and parser, using generative testing
to make sure all parts of the library match up in terms of regex semantics.</p><h2 id="chui">Chui</h2><p>A while ago we were approached by <a shape="rect" href="https://pitch.com">Pitch</a>, to see if we could
help improve their testing setup. Pitch writes a lot of ClojureScript, and uses
shadow-cljs as its main build tool. They were using shadow-cljs’s
<code>:browser-test</code> target together with <code>cljs-test-display</code> as UI, but the
experience was sub-par. With little affordances to focus in on specific tests
people were forced to re-run the whole test suite over and over, which was
actually discouraging people to write tests at all.</p><p>Test tooling for ClojureScript has always been somewhat of a challenge. Because
ClojureScript needs to be compiled up-front, and needs to be loaded in a
separate JavaScript runtime environment, it lacks the dynamic nature and runtime
inspection that Clojure has.</p><p>Most existing cljs test tooling takes the approach of building a single blob
containing both test runner and tests, and then running that as a unit.</p><p>After some hammock time we figured the way forward would be to capture as much
metadata about tests as possible during compilation, so that at runtime it
becomes possible to run tests on demand. The result is Chui. The core of Chui is
a new test runner API for ClojureScript (<code>lambdaisland.chui.runner</code>). On top of
this we then provide a browser-based GUI (<code>lambdaisland.chui.ui</code>), and a
shadow-specific <code>:runner-ns</code> which combines the two
(<code>lambdaisland.chui.shadowrun</code>).</p><p>Chui can in principle be used with any build tool, but it works particularly
well with shadow-cljs because of the functionality provided by <code>:browser-test</code>,
which handles finding tests on the file system and injecting them into the
build. Without this (or something similar) you will have to manually add
<code>:require</code> statements for all your test namespaces. Hopefully other tooling like
Figwheel-main will add similar functionality in the future, or perhaps we can
even get it into the ClojureScript compiler itself.</p><p>Chui is not released yet, but we are in the final stretch, and you can find
instructions on Github to already try it out yourself.</p><p>The Chui runner will also form the basis for the next generation of kaocha-cljs,
which we are working on with support from Nextjournal. I’m sure we’ll have more
to say about that next month.</p><p><img src="https://img.lambdaisland.com/2ec6384708d83d6116fd7a0df987cda7.png" alt="Chui screenshot" /></p><h2 id="kaocha">Kaocha</h2><p style="text-align: center"><img src="https://img.lambdaisland.com/368d754b43d22ad1b9e719b8c6055df3.png" alt="" /></p><p>Kaocha got a new logo! The talented <a shape="rect" href="https://twitter.com/moertel">Stefanie
Grunwald</a> has started work on a series of pixel-art
logos for our projects, the first one being this anvil for Kaocha. We can’t wait
to see this on a sticker!</p></<>>Coffee Grinders, part 2https://lambdaisland.com/blog/2020-03-29-coffee-grinders-22020-03-29T12:01:58+00:00<<>><p>Back in December I wrote about a coding pattern that I’ve been using more and more often in my work, which I dubbed “<a shape="rect" href="https://lambdaisland.com/blog/2019-12-16-advent-of-parens-16-coffee-grinders">Coffee Grinders</a>”. My thoughts around this were still gelling at the time, and so the result was a meandering semi-philosophical post that didn’t really get to the point, and that didn’t seem to resonate that much with people.</p><p>Some of the responses I got were “just use functions” or “sounds like a finite-state machine”, which makes it clear that I was not in fact making myself clear.</p><p>Having continued to encounter and apply this pattern I’d like to present a more concise, semi-formal definition of coffee grinders.</p><p>At this point I should note that interceptors as pioneered by Pedestal, and since implemented in several other places like Sieppari or re-frame are a specific type of Coffee Grinder. In other words, coffee grinders are a generalization of interceptors. This post will make a lot more sense if you are familiar with Interceptors, at least in passing.</p><p>I’m making the <a shape="rect" href="https://lambdaisland.com/episodes/interceptors-concepts">Lambda Island episode about Interceptors</a> free to watch for a limited time in case you need to brush up on that. There’s also a <a shape="rect" href="https://github.com/lambdaisland/ep47-interceptors/tree/master/src/lambdaisland">toy Interceptor implementation</a> available, which is written in a way that encourages getting familiar with the concept from a REPL.</p><p>So without further ado, a Coffee Grinder is a way to model and execute a step-by-step process. It has the following elements:</p><ul><li>A persistent (functional, immutable) data structure known as the <code>context</code>, which contains the complete current state of the process: inputs, outputs, intermediate state, and <code>work queue</code>.</li><li><code>Step functions</code>, which accept a <code>context</code>, and yield an updated <code>context</code>, possibly asynchronously</li><li>A <code>runner</code> component which takes a <code>context</code>, and repeatedly runs it through the next <code>step function</code> in the <code>work queue</code>, until the queue is empty or some other terminating condition is reached</li></ul><p>Some corollaries:</p><p>Step functions have access to the full <code>context</code>, including the <code>work queue</code>, which they are able to manipulate (by returning a <code>context</code> with an updated <code>work queue</code>). This allows queuing additional steps, but it also allows other things like prepending steps to the beginning of the queue, interleaving the steps in the queue with extra intermediate steps, or returning an empty queue, to terminate the process.</p><p>The complete state of the process is <a shape="rect" href="https://lambdaisland.com/blog/2019-12-01-advent-of-parens-1-clojure-vocab-reify">reified</a> in the <code>context</code>. You can pause, restart, or inspect the process at any time, including after an error occurs, as long as you have the right <code>context</code> value available.</p><p>Coffee Grinders distinguish themselves in the “rules” of how the runner picks up and executes work, how asynchrony is handled, and what extra features it supports, like error handling or terminating conditions. Let’s look at the most basic implementation of a coffee grinder.</p><pre><code class="language-clojure">(defn run [{:keys [queue] :as ctx}]
(if-let [[step & next-queue] (seq queue)]
(recur (step (assoc ctx :queue next-queue)))
ctx))
</code></pre><pre><code class="language-clojure">(run {:counter 0
:queue [#(update % :counter inc)
#(update % :counter inc)]})
;;=> {:counter 2 :queue nil}
</code></pre><p>Note again that we have the full context available, including the queue, so a single step can dynamically enqueue ten more steps.</p><pre><code class="language-clojure">(run {:counter 0
:queue [(fn [ctx]
(update ctx
:queue
concat
(for [i (range 10)]
#(update % :counter + i))))]})
;;=> {:counter 45 :queue nil}
</code></pre><p>A typical use case is HTTP routing. Remember that HTTP request handling was the inspiration for Interceptors in the first place. They replace the opaque “wrapping function” based middleware concept with a reified queue and stack.</p><p>Here a single <code>route</code> step is responsible for enqueuing the actual HTTP handler.</p><pre><code class="language-clojure">(defn handle-hello-world [req]
{:status 200
:body "Hiiii"})
(def routes
[[:get "/hello/world" handle-hello-world]])
(defn router [{:keys [request] :as ctx}]
(if-let [handler (some (fn [[m p h]]
(when (= [m p]
[(:request-method request)
(:path request)])
h))
routes)]
(update ctx
:queue
concat
[(fn [ctx]
(assoc ctx :response
(handler (:request ctx))))])
ctx))
(run {:request {:request-method :get
:path "/hello/world"}
:queue [router]})
;; => {:request {:request-method :get, :path "/hello/world"}
;; :queue nil
;; :response {:status 200, :body "Hiiii"}}
</code></pre><p>Another use case which I described in my original coffee grinders post is resource loading. If a step notices that a resource it needs to operate is not locally available, then it can prepend a resource-fetching step to the queue, and then re-enqueue itself.</p><p>Coffee grinders can implement asynchrony by allowing step functions to return not a new context, but a future value of a context, for instance a promise, core.async channel, or future.</p><pre><code class="language-clojure">(defn run [{:keys [queue] :as ctx}]
(if-let [[step & next-queue] (seq queue)]
(let [result (step (assoc ctx :queue next-queue))]
(recur (cond-> result
(instance? IDeref result)
deref)))
ctx))
</code></pre><p>The above code works fine in Clojure because the threaded nature of the JVM means you can always block until a result is available, no matter which flavor of asynchrony you are using. In evented contexts like JavaScript this is not possible, and you may need to fully embrace the asynchrony. In this code snippet we allow step functions to return JS promises. An extra <code>done</code> callback is used to deliver the final context back to the caller.</p><pre><code class="language-clojure">(defn run [{:keys [queue] :as ctx} done]
(if-let [[step & next-queue] (seq queue)]
(let [result (step (assoc ctx :queue next-queue))]
(if (promise? result)
(.then result #(run % done))
(recur result)))
(done ctx)))
</code></pre><p>So far we’ve reified process state, making what is otherwise spread across the stack and heap — in locals and instance fields — a single concrete value that we can pass around, inspect, manipulate, or serialize.</p><p>However we still have opaque function values in there, making the last option not viable. Instead we can further move towards pure data by enqueuing operations rather than functions, similar to how events and subscriptions are represented in re-frame.</p><pre><code class="language-clojure">(defmulti handle (fn [ctx [op _]] op))
(defmethod handle :add-to-counter [ctx [_ i]]
(update ctx :counter + i))
(defn run [{:keys [queue] :as ctx}]
(if-let [[op & next-queue] (seq queue)]
(recur (handle (assoc ctx :queue next-queue) op))
ctx))
(run {:counter 0
:queue [[:add-to-counter 31]
[:add-to-counter 11]]})
;;=> {:counter 42, :queue nil}
</code></pre><p>Now your context is pure data that can be sent over the wire or stored to disk. Maybe you want to snapshot intermediate states to pick them up later, capture the context in case of an error to accompany bug reports, or have multiple worker nodes in a network which pick up work in the form of in-progress contexts off a database or queue, execute the next step, and write back the result.</p><p>You can take this one step further, by also reifying your side effects, and letting the runner handle them. This has great benefits for testing, since all your step functions are now pure.</p><pre><code class="language-clojure">(defmulti handle-effect key)
(defmethod handle-effect :send-email [[_ {:keys [to subject body]}]]
,,,)
(defmethod handle :signup-user [ctx _]
(assoc ctx :fx {:create-user [,,,]
:send-email {:to ,,, :subject ,,,}}))
(defn run [{:keys [queue] :as ctx}]
(if-let [[op & next-queue] (seq queue)
next-ctx (handle (assoc ctx :queue next-queue) op)]
(do
(run! handle-effect (:fx next-ctx))
(recur (dissoc next-ctx :fx)))
ctx))
</code></pre><p>There are many more directions you can take this pattern. Just look at interceptors, which besides a <code>queue</code> also add a <code>stack</code> to the <code>context</code>, thus emulating HTTP middleware with its “on-the-way-in” and “on-the-way-out” operations.</p><p>I hope at this point I’ve at least piqued some people’s curiosity. Do let me know if you encounter a problem in your work that is elegantly solved with a coffee grinder.</p></<>>Launching an Open Collective for Lambda Island OSShttps://lambdaisland.com/blog/2020-02-03-open-collective-lambda-island-open-source2020-02-04T06:42:00+00:00<<>><h2 id="tldr">tl;dr</h2><p>We are launching an <a shape="rect" href="https://opencollective.com/lambda-island#section-contribute">Open Collective for Lambda Island Open Source</a>, to help support our Open Source and Community work. Please check it out, pass it (and this article) on to your boss, or consider contributing yourself.</p><h2 id="it-started-with-a-mission">It started with a mission</h2><p>The mission of Lambda Island has always been to help grow and improve the Clojure ecosystem, because compared to more mainstream technologies we see it as a materially better path to high quality, reliable, and maintainable software.</p><p>Achieving this mission has partly been educational, we want to help people get better at using this great language, and help them navigate the libraries, tools, and techniques that are on offer. So this we’ve done through our <a shape="rect" href="http://lambdaisland.com/episodes/">videos</a>, <a shape="rect" href="https://lambdaisland.com/blog/">blog posts</a>, and our <a shape="rect" href="https://lambdaisland.com/guides/clojure-repls/clojure-repls">Clojure REPL guide</a>.</p><p>But a comparatively niche ecosystem like Clojure also needs people to build and improve the libraries and tools so others can build on them. In today’s Open Source world we have come to expect a ready-to-consume library for every use case, but these libraries don’t write or maintain themselves.</p><h2 id="someone-needs-to-write-the-code">Someone needs to write the code</h2><p>Over the years we have contributed patches to dozens of Clojure projects. Often when writing a script we would run into snags, contribute a fix, and have it merged and released before the video came out. But we also ran into different kinds of snags, places where instead of a patch to an existing library, we felt that a new project was in order, either to fill a gap, or to improve upon the status quo.</p><p>The best known of these is the <a shape="rect" href="http://github.com/lambdaisland/kaocha">Kaocha</a> test runner. We realized that in terms of testing practices and tools the Clojure world was running miles behind other language communities, and so we set out to fix that, which resulted in a family of projects: extensions for running ClojureScript or Cucumber tests, and plugins for doing code coverage, CI integration, and more.</p><p>Thanks to funding by Clojurists Together we managed to push Kaocha further, which was extremely validating and allowed us to implement things we couldn’t have done otherwise.</p><p>Other projects include <a shape="rect" href="https://github.com/lambdaisland/deep-diff">deep-diff</a>, <a shape="rect" href="https://github.com/lambdaisland/uri">uri</a>, <a shape="rect" href="https://github.com/lambdaisland/glogi">Glögi</a>, <a shape="rect" href="https://github.com/lambdaisland/metabase-datomic">metabase-datomic</a>, <a shape="rect" href="https://github.com/lambdaisland/ansi">ansi</a>, and others, with more projects in the pipeline. Together these are now seeing over 50,000 monthly downloads on Clojars.</p><p>A final part we play in achieving our mission is running and maintaining community infrastructure, namely the <a shape="rect" href="https://clojureverse.org">ClojureVerse</a> discussion forum, and the <a shape="rect" href="https://clojurians-log.clojureverse.org">clojurians-log</a>, which captures what happens on Clojurians Slack, so these conversations are accessible to search engines, rather than being lost to history.</p><h2 id="stronger-together">Stronger together</h2><p>Up to a year ago the bulk of this work was done by a single person. Then last year Arne decided that running this as a one man show was no longer sustainable, and decided to found a company: <a shape="rect" href="https://gaiwan.co">Gaiwan.co</a>. Gaiwan is now responsible for all the above, as well as doing coaching, consulting, and development for clients directly. Gaiwan also ran the Heart of Clojure conference last summer, once again progressing the original mission.</p><p>Since then we’ve engaged a handful of other Clojurists, working on several successful client projects. Together we are figuring out what it means to be a team, and how to balance our community engagements with the need to make money.</p><p>Gaiwan is a remote-first company where people manage their own time. We expect people to spend at most 80% of their time on client work, how they spend the other 20% is up to them. They can spend it away from the screen, or work on their own projects, or they can work on Lambda Island Open Source. If it’s the latter, then that work should be paid. Similarly if people are in between client projects they should be able to work on open source and be compensated.</p><p><strong>We will partly fund this work ourselves as we’ve done in the past, but we’re also hoping the community sees enough value in what we put out there to pitch in. This is why today we’re launching the **</strong><a shape="rect" href="https://opencollective.com/lambda-island#section-contribute">Lambda Island Open Collective</a><strong>**.</strong></p><h2 id="becoming-a-collective">Becoming a Collective</h2><p>Open source funding drives are well established at this point, and certain funding models have become widely known and understood. Still we have a fairly specific idea of how we plan to use this Open Collective which might be a little different from what you’ve seen elsewhere, so allow me to explain.</p><p>Our collective is hosted by Open Collective Europe. They are our fiscal host, meaning that the money you donate, be it one time or monthly, goes into their account. This way a budget can accumulate in the collective over time.</p><p>To make it clear what we want to do with that money we will put forward pitches. (See Shape Up chapter 6: “<a shape="rect" href="https://basecamp.com/shapeup/1.5-chapter-06">Write the Pitch</a>”). We have three pitches so far, <a shape="rect" href="https://nextjournal.com/lambdaisland/kaocha-autotest">this one introducing a Kaocha autotest feature</a>, one that <a shape="rect" href="https://nextjournal.com/lambdaisland/pitch-clojurescript-support-for-deep-diff">ports our most popular library, deep-diff, to ClojureScript</a>, and a feature we’ve been wanting to bring to Clojure for a long time: <a shape="rect" href="https://nextjournal.com/lambdaisland/pitch-kaocha-doctests">Doctests!</a>. These form a fairly detailed plan of what we want to achieve, and what we think the development cost of this feature will be.</p><p>When the feature gets implemented and shipped, then we submit an expense to the collective, transferring said amount to Gaiwan where it becomes part of the resources we allocate for more open source work.</p><p>This models also provides us a way to invite community contributions. If you want to improve a Lambda Island project then start by writing a pitch (here’s a handy <a shape="rect" href="https://nextjournal.com/lambdaisland/pitch-template">template</a>), including the amount you’d like to receive for this contribution. If your pitch gets approved and you finish the work then you can submit an expense claim directly to the collective, and get compensated for your work.</p><p>What we want to avoid is being some abstract “good cause”, where it’s not really clear what your money is doing. Neither do we want to ask monet for something that we’ve already given to you for free, a common but flawed business model in Open Source.</p><p>Instead as a collective you are paying for something specific, which you will get at some point in the future. And if your company is particularly invested in seeing a certain improvement happen, then they can buy the development of that feature directly, at the price outlined in the pitch, or commission us to write the pitch in the first place.</p><p>We hope that this way of working will allow us to engage with the users of our projects on a new level, and that together we can continue to lift the Clojure ecosystem to new heights.</p><p>If you want to be part of this then please head over to the collective and pitch in. Are you using Kaocha or any of our other projects at work, then talk to your boss, and get them to pitch in.</p><p><strong><a shape="rect" href="https://opencollective.com/lambda-island#section-contribute">Help us make great things happen, become a backer today.</a></strong></p><p>A big thanks from the whole team!</p><p>Arne, Art, Davide, Enyert, Felipe, Sławek.</p></<>>Advent 2019 part 24, The Last Posthttps://lambdaisland.com/blog/2019-12-24-advent-of-parens-24-the-last-post2019-12-24T15:01:10+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Day 24, I made it! I did skip a day because I was sick and decided it was more
important to rest up, but I’m pretty happy with how far I got.</p><p>Let’s see how the others fared who took on the challenge. John Stevenson at
<a shape="rect" href="https://practicalli.github.io/blog/">Practicalli</a> got four posts out spread out
across the advent period, similar to lighting an extra candle every sunday of
the advent. Good job!</p><p><a shape="rect" href="https://porkostomus.gitlab.io/">Bobby Towers</a> mostly wrote about all the work
he’s done on his Mario-inspired media creation platform called MECCA, making it
to a respectable 6 day streak.</p><p><a shape="rect" href="http://slipset.github.io/">Erik Assum</a> aka slipset managed a three day streak,
and then wrote a few more posts, making it to day 8. Nice!</p><p><a shape="rect" href="https://alexanderoloo.com/">Alexander Oloo</a> did admit on Twitter that the odds
were small he would complete it. He did make it to day 9. I really appreciated
his more beginner-friendly articles. Good job Alex, keep writing!</p><p>Never afraid of a challenge Bozhidar Batsov also picked up the torch, writing on
his <a shape="rect" href="https://metaredux.com/">Metaredux</a> blog, running a flawless 24 day streak.
Respect! The posts are a mix of Clojure and Ruby, and all enjoyable.</p><p>What I really hoped to achieve with this challenge was to inspire at least a few
people in the Clojure world to blog more. These independent voices are a vital
part of a healthy community.</p><p>Some people asked if I wouldn’t run out of ideas, but honestly I still have a
dozen topics left without even trying. What people don’t realize is that things
don’t have to be revolutionary or ground breaking to make good posts. On the
contrary, people try too hard to be thought leaders, on blogs, in conference
talks, on Twitter. It gets really old!</p><p>Instead speak directly from your own experience, whatever that may be. It
doesn’t have to be new or advanced, it just has to be genuine. You don’t need to
wow, you just need to share. Every programming thing that anybody knows they
once had to learn. This means every small thing you do is worth writing about,
for the benefit of the next person.</p><p>And even more interesting then the technical details are the story, the context. Why did you do this? What were you working on? How did it turn out? My favorite blog posts and conference talks are basically this: “We had to solve this real-world problem under these constraints, we took this approach, and this worked out well (or not) for us”. My least favorite format is “Here’s an architecture/platform/methodology that is amazing and you should use it.”.</p><p>So write from your experience. Literally everything can be a topic, like a friend told me, it’s a state of
mind. Start noticing the little things you do, the bits of code you find
interesting, the things you figure out, the hacks and tricks you came up with or
learned from someone else. Start keeping that list, and then every so often take
some time, pick an item from the list, and write that blog post. Don’t dwell it
on it and polish it to perfection. Just get it out and move on!</p><p>I want to thank everyone who blogged about Clojure this year. You all rock! And I want to thank everyone who will blog about Clojure in 2020 in advance. To use Eric Normand’s creed: Rock on!</p><p>How about you, will you blog about Clojure in 2020? If you do then make sure to
add your blog to the <a shape="rect" href="http://planet.clojure.in/">Planet Clojure</a> blog
aggregator. I just <a shape="rect" href="https://github.com/ghoseb/planet.clojure/pull/297">addded a
PR</a> for Alexander Oloo, Erik
Assum, and Practicalli.</p><p>Other places to get your blog posts out are the <a shape="rect" href="https://clojureverse.org/c/learning-resources/blogs/42">Blogs Category on
ClojureVerse</a> or the
<code>#news-and-articles</code> on Clojurians slack,</p><p>You can also <a shape="rect" href="https://twitter.com/planetclojure">follow Planet Clojure on Twitter</a> if
you would like to see some more blogs on your timeline.</p><p>Enjoy your holidays, and see you all in 2020!</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 23, Full size SVG with Reagenthttps://lambdaisland.com/blog/2019-12-23-advent-of-parens-23-full-size-svg-with-reagent2019-12-24T13:43:45+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>I’ve been a big fan of SVG since the early 2000’s. To me it’s one of the great
victories of web standards. Mind you, browser support has taken a long time to catch up.
Back then you had to embed your SVG file with an <code><object></code> tag, and support for the
many cool features and modules was limited and inconsistent.</p><p>These days of course you can drop an <code><svg></code> tag straight into your HTML. What a
joy! And since the SVG can now go straight into the DOM, you can draw your SVG
with React/Reagent. Now there’s a killer combo.</p><p>I already mentioned
<a shape="rect" href="https://lambdaisland.com/blog/2019-12-10-advent-of-parens-10-hillchart-reagent-shadow-firebase">Hillcharts</a>
earlier in this series. Another SVG project that I’ve been tinkering on is a
game prototype called Hexagons (working title). It came about when I was on vacation the weeks
after Heart of Clojure. I had taken some time off to decompress, eat rösti, and
spend some time in nature. One evening my idle mind started wandering and came
up with an idea for a tabletop game. I built a first prototype of it the next
day from cardboard, playing cards, post-its and pebbles.</p><p>The prototype left to be desired, and I figured that if I wanted to experiment
more with the game mechanics then a browser-based version would actually be
pretty convenient. This would also allow me to try it out with some of my old
buddies back home whom I don’t get to see often, thanks to the magic of the internet.</p><p>The whole game is just a single big SVG, so my idea was to have this one <code><svg></code>
element simply span the full browser viewport. This wasn’t as easy as I thought
it would be. CSS was of little help, it was easy enough to have it span the full
width, but I couldn’t figure out a good way to have it span the full height.</p><p>So I came up with this.</p><pre><code class="language-clojure">(defn viewport-size []
{:width js/document.documentElement.clientWidth
:height js/document.documentElement.clientHeight})
</code></pre><p>This app is just a simple Reagent app, no re-frame or anything like that. Since
it’s pretty limited in scope I figured it would be fun to go back to basics, and
just use ratoms and reactions directly.</p><p>In typical reagent/re-frame style a single ratom contains the application state,
including the size of the browser viewport.</p><pre><code class="language-clojure">(defonce state (reagent/atom (viewport-size)))
</code></pre><p>And this is what the root of the render tree looks like. I’m using the SVG
<code>viewBox</code> property to shift the coordinate system so that <code>[0, 0]</code> is right in
the middle of the browser.</p><p>The game pieces are hexagonal tiles. You start with one tile, and then players
add more tiles one by one, so the game kind of grows out of the center where
that first tile is. That’s why I decided to lay the coordinates like this.</p><p>Other UI elements are drawn relative to the viewport edge. This turned out to
work well with the zoom functionality built into browsers. When you zoom out all
tiles and other elements get smaller, but they keep their position either
relative to the center of the viewport, or relative to the edge of the viewport.</p><pre><code class="language-clojure">(defn app-root []
(let [{:keys [width height]} @state
top-left-x (/ width -2)
top-left-y (/ height -2)]
[:svg {:width width
:height height
:viewBox (str/join " " [top-left-x top-left-y width height])}
;; UI goes in here
]))
</code></pre><p>You still need to handle browser resizing, that’s what this final snippet is
for.</p><pre><code class="language-clojure">(set! js/document.body.onresize (fn [_] (swap! state merge (viewport-size))))
</code></pre><p>Playing around with SVG, Reagent’s hiccup, and Figwheel or shadow-cljs for
reloading is a lot of fun. I can highly recommend it. If you’re not sure how to
start then just draw some things in Inkscape, convert the result to Hiccup, and
paste it into your code.</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 21, Project level Emacs config with .dir-locals.elhttps://lambdaisland.com/blog/2019-12-21-advent-of-parens-21-project-config-dir-locals2019-12-21T14:05:11+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>An extremely useful Emacs feature which I learned about much too late is the
<code>.dir-locals.el</code>. It allows you to define variables which will then be set
whenever you open a file in the directory where <code>.dir-locals.el</code> is located (or
any subdirectory thereof).</p><p>Here’s an example of a <code>.dir-locals.el</code> file of a project I was poking at today.</p><pre><code class="language-clojure">((nil . ((cider-clojure-cli-global-options . "-A:fig:dev")
(cider-preferred-build-tool . clojure-cli)
(cider-default-cljs-repl . figwheel-main)
(cider-figwheel-main-default-options . "dev")
(cider-repl-display-help-banner . nil))))
</code></pre><p>What this essentially does is set a bunch of CIDER variables for this specific
project. Mainly it stops CIDER asking annoying questions. It really bugs me when
it asks me five times a day what ClojureScript REPL I’m using. This is project
level stuff, so with <code>.dir-locals.el</code> I configure it at the project level and
CIDER never has to ask me again.</p><p>The syntax can throw people off though, so here’s a really quick primer on
association lists.</p><p>Emacs Lisp doesn’t have the handy hash-map syntax that we Clojurists enjoy. It
does have a kind of hash table but there’s no syntax literal for it, and in
practice it seems it’s rarely used. Instead, like most traditional Lisps, Elisp
primarily makes use of two-element tuples called cons cells.</p><p>You can create them with the <code>cons</code> function, or with a ‘dotted pair’ notation.
So these two expressions are identical.</p><pre><code class="language-clojure">(cons "val1" "val2")
'("val1" . "val2")
</code></pre><p>Need to have a hash-map like associative mapping from key to value? Well then
just stick a bunch of these cons cells in a list.</p><pre><code class="language-clojure">((:name . "Arne")
(:favorite-lisp . "not Elisp"))
</code></pre><p>Back to <code>.dir-locals.el</code>. It contains an association list mapping major modes to
variable mappings.</p><pre><code class="language-clojure">((clojure-mode . ((var-1 . "val 1")
(var-2 . "val 2")))
(ruby-mode . ((var-3 . "val 3"))))
</code></pre><p>This will set <code>var-1</code> and <code>var-2</code> as buffer local variables in Clojure buffers,
and it will set <code>var-3</code> in Ruby buffers.</p><p>Or you can not really care about the major mode and instead just put <code>nil</code>, that
way it will be used in all buffers regardless of their major mode. The per-major
mode stuff is mainly useful for general Emacs variables that you want to change
based on the language.</p><pre><code class="language-clojure">((js-mode . ((js-indent-level . 2)
(indent-tabs-mode . nil))))
</code></pre><p>And once you set up these variables for your project by all means do check the
<code>.dir-locals.el</code> into source control so others may benefit from it as well.</p><p>One caveat: when you update your <code>.dir-locals.el</code> the changes will only come
into effect when you (re-)open a file in that directory. This gets me all the
time. To quickly reload a file you already have open use <code>C-x C-f RET</code> in Emacs
or <code>SPC f A RET</code> in Spacemacs.</p><p>A final trick to be aware of is that you can use the special <code>eval</code> variable to
evaluate code when a file gets opened. This can be very useful for instance when
you are using a type of ClojureScript REPL that CIDER does not (yet) know about.</p><pre><code class="language-clojure">((nil . ((cider-default-cljs-repl . my-cljs-repl)
(eval . (cider-register-cljs-repl-type 'my-cljs-repl "(code-that-switches-to-the-cljs-repl)")))))
</code></pre><p>For the full lowdown you naturally only need to check the info pages, <code>(info "(emacs) Directory Variables")</code>.</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p></<>>Advent 2019 part 20, Life Hacks aka Emacs Ginger Teahttps://lambdaisland.com/blog/2019-12-20-advent-of-parens-20-life-hacks-emacs-ginger-tea2019-12-20T14:28:35+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Here’s a great life hack, to peal ginger don’t use a knife, use a spoon. I’m not
kidding. Just scrape off the peal with the tip of the spoon, it’s almost too
easy.</p><p>Here’s a Clojure + Emacs life hack:</p><pre><code class="language-emacs-lisp">(defun plexus-clojure-extras/cider-pprint-register (register)
(interactive (list (register-read-with-preview "Eval register: ")))
(cider--pprint-eval-form (get-register register)))
</code></pre><p>So what does this do? It takes the value of a specific register and evaluates
it as Clojure code with CIDER, showing the pretty-printed result in a separate buffer.</p><p>Registers are these places in Emacs where you can store a snippet of text (among other things) for
later use. They have one letter names, so for instance you
select some text, then press <code>C-x r s a</code> to put that text into register <code>a</code>.</p><table><thead><tr><th colspan="1" rowspan="1">Command</th><th colspan="1" rowspan="1">Keybinding</th><th colspan="1" rowspan="1">Description</th></tr></thead><tbody><tr><td colspan="1" rowspan="1">copy-to-register</td><td colspan="1" rowspan="1">C-x r s</td><td colspan="1" rowspan="1">Copy to region to a register.</td></tr><tr><td colspan="1" rowspan="1">view-register</td><td colspan="1" rowspan="1"> </td><td colspan="1" rowspan="1">See what’s in a register.</td></tr><tr><td colspan="1" rowspan="1">insert-register</td><td colspan="1" rowspan="1">C-x r i</td><td colspan="1" rowspan="1">Insert a given register into the buffer.</td></tr><tr><td colspan="1" rowspan="1">set-register</td><td colspan="1" rowspan="1"> </td><td colspan="1" rowspan="1">Set a register to a given value.</td></tr></tbody></table><p>Try <code>(info "(Emacs)Registers")</code> for more info on registers.</p><p>I have a number of these registers prepopulated when my Emacs starts, for instance:</p><pre><code class="language-emacs-lisp">(set-register ?k "(do (require 'kaocha.repl) (kaocha.repl/run))")
(set-register ?K "(do (require 'kaocha.repl) (kaocha.repl/run-all))")
(set-register ?r "(user/refresh)")
</code></pre><p>But often I’ll also bind them on the fly. Say I’m working on a complicated
function, and below the function in the buffer I have some code that calls said
function. I can jump back and forth between the two, each time updating the
function, then moving to the invocation and checking the result. Or I can copy
the invocation to a register, and invoke it from anywhere.</p><p>The last piece of the puzzle is binding this to some handy key combination. I’m
a Spacemacs user, and I’ve bound this to leader-key + <code>,</code>. My leader key is also
<code>,</code>, so I can now run the tests for a namespace with <code>,,k</code>. Pretty handy.</p><pre><code>(dolist (m '(clojure-mode
clojurec-mode
clojurescript-mode
clojurex-mode
cider-repl-mode
cider-clojure-interaction-mode))
(spacemacs/set-leader-keys-for-major-mode m
"," 'plexus-clojure-extras/cider-pprint-register))
</code></pre><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 19, Advent of Random Hackshttps://lambdaisland.com/blog/2019-12-19-advent-of-parens-19-advent-of-random-hacks2019-12-19T19:20:03+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Something I’m pretty good at is coming up with random hacks. The thing where
you’re like “hey how about we plug this thing into that thing” and everyone says
“why would you do that that’s a terrible idea” and I’m like (<em>mario voice</em>)
“let’s a go”.</p><p>And sometimes the result is not entirely useless. Like
<a shape="rect" href="https://github.com/borkdude/babashka/blob/5e10c913e0dc706c4b4de85e508779a67c6e51c0/README.md#convert-projectclj-to-depsedn">this little oneliner</a>
I came up with yesterday, using Babashka to “convert” a <code>project.clj</code> into a
<code>deps.edn</code>.</p><pre><code>cat project.clj | sed -e 's/#=//g' -e 's/~@//g' -e 's/~//g' | bb '(let [{:keys [dependencies source-paths resource-paths]} (apply hash-map (drop 3 *in*))] {:paths (into source-paths resource-paths) :deps (into {} (for [[d v] dependencies] [d {:mvn/version v}]))} ) ' | jet --pretty > deps.edn
</code></pre><p>You could do this properly, write a hundred lines or more to deal with profiles,
aliases, etc, or you can cook this up in ten minutes, and do the rest by hand.</p><p>Reminds me of that time I wrote a <a shape="rect" href="https://twitter.com/plexus/status/308362006711394304">chat server that fits in a
tweet</a>, (the old 140 character ones), on the bus on the
way back from a conference. Code golf was a lot easier when I still did Ruby :D</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 18, Exploring Bootleghttps://lambdaisland.com/blog/2019-12-18-advent-of-parens-18-exploring-bootleg2019-12-18T09:43:36+00:00<p><a shape="rect" href="https://nextjournal.com/plexus/advent-2019-part-18-exploring-bootleg">Today’s post is published on Nextjournal</a>. Enjoy!</p>Advent 2019 part 17, trace! and untrace!https://lambdaisland.com/blog/2019-12-17-advent-of-parens-17-trace-untrace2019-12-17T10:02:59+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Here’s a little REPL helper that you may like.</p><pre><code class="language-clojure">(defn trace! [v]
(let [m (meta v)
n (symbol (str (ns-name (:ns m))) (str (:name m)))
orig (:trace/orig m @v)]
(alter-var-root v (constantly (fn [& args]
(prn (cons n args))
(apply orig args))))
(alter-meta! v assoc :trace/orig orig)))
(defn untrace! [v]
(when-let [orig (:trace/orig (meta v))]
(alter-var-root v (constantly orig))
(alter-meta! v dissoc :trace/orig)))
</code></pre><p>Pass <code>trace!</code> a function var and it will print out the invocation each time the function gets called.</p><p>Drop it in <code>user.clj</code> and <code>trace!</code> to your heart’s content!</p><pre><code class="language-clojure">user> (trace! #'foo.bar/baz)
user> (foo.bar/baz 123)
(foo.bar/baz 123)
nil
user>
</code></pre><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 16, Coffee Grindershttps://lambdaisland.com/blog/2019-12-16-advent-of-parens-16-coffee-grinders2019-12-16T10:48:08+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Over the last year or so I’ve found myself using some variations on a
certain pattern when modelling processes in Clojure. It’s kind of like a
event loop, but adapted to the functional, immutable nature of
Clojure. For lack of a better name I’m calling these coffee grinders. (The
analogy doesn’t even really work but the kid needs to have a name.)</p><p>Since I saw <a shape="rect" href="https://www.youtube.com/watch?v=UJnsXUVsr7w">Avdi Grimm’s OOPS Keynote</a>
at Keep Ruby Weird last year I’ve been thinking a lot about the transaction vs process
dichotomy. Avdi talks about the “Transactional Fallacy” from around 15:25. From his slides:</p><p>A transaction:</p><ul><li>Has a goal and is synonymous with its goal.</li><li>Is synchronous (blocking).</li><li>Can’t be put aside for later.</li><li>Completes (or fails) in essentially zero time.</li><li>Not supposed to be able to hang.</li><li>Either completes or aborts.</li><li>Self-destructs on error. (there is no concept of resuming)</li></ul><p>The transactional fallacy is when you model something as a transaction while it
really is a process. In Clojure we are perhaps even more susceptible to this
because we tend to model things simply as functions. A web service call, a
database import, a HTTP handler which in turn consults several microservices,
they all just become functions. When you call such a function all you can do is
wait for it to finish or blow up. What if you want to inspect where in the
process it is at, maybe show a progress bar or some stats to the user? What if
you want to be able to resume a failed process?</p><p>Then you need to model these things <em>as a process</em>. Luckily in Clojure we
actually have great tools to do this, because in Clojure time is first class. We
model time as successive values of immutable data.</p><p>A process has a notion of internal state. Inspired by
<a shape="rect" href="http://pedestal.io/reference/interceptors">interceptors</a> I tend to use a map
for this called <code>context</code> or <code>ctx</code>.</p><p>A process happens step by step, so it has a notion of what to do next. I tend to
model this as some kind of queue of simple EDN events, maps which have an event
<code>:type</code> property.</p><p>I’ll describe two concrete use cases, with some clear differences but also
clearly some commonalities. The first is the message queue the
<a shape="rect" href="https://github.com/lambdaisland/kaocha-cljs">kaocha-cljs</a> uses to handle
messages coming from the ClojureScript pREPL. The second is code I wrote for
<a shape="rect" href="https://nextjournal.com">Nextjournal</a> to reify part of their data model.</p><p>To run ClojureScript tests Kaocha-cljs starts up a ClojureScript pREPL. A pREPL
is a Process REPL, it returns data structures rather than just a stream of text,
so it is more suitable for tooling.</p><p>The first thing Kaocha-cljs will do after starting this pREPL is open a
websocket connection back to Kaocha, so it can send testing events back while
tests are running, like <code>:fail</code>, <code>:pass</code>, <code>:begin-test-var</code>, etc.</p><p>Kaocha sends forms to the pREPL like <code>(require 'foo.bar-test)</code> and
<code>(foo.bar-test/some-test)</code>. It then asynchronously starts receiving messages,
either from the pREPL whenever something gets printed, when an exception occurs,
when the function returns, or from websocket (test started, test failed, etc).</p><p>Based on these messages it needs to keep track of where it is in the process
(has the pREPL booted? has the websocket connection been established? has the
namespace loaded? has the test finished?), tally up test results, assign them to
the right test in the result data, and decide to continue with the process,
moving on to the next test or namespace, or time out.</p><p>To do this it puts all these messages on a
<code>java.util.concurrent.LinkedBlockingQueue</code>. Then at each point where we need to
wait for the ClojureScript side to do stuff it invokes this <code>queue-consumer</code>.</p><pre><code class="language-clojure">(defn queue-consumer [{:keys [queue timeout handlers result]}]
(let [poll #(.poll queue timeout TimeUnit/MILLISECONDS)
handlers (merge default-queue-handlers handlers)]
(loop [message (poll)
context {}]
(if (nil? message)
(if-let [timeout-handler (get handlers :timeout)]
(timeout-handler context)
:timeout)
(if-let [handler (get handlers (:type message))]
(let [context (handler message context)]
(if-let [result (result context)]
result
(recur (poll) context)))
(recur (poll) context))))))
</code></pre><p>This takes messages of the queue one by one, and dispatches them to a dispatch
function based on the message type. Each dispatch function receives the message
and the context map, and returns a potentially updated context map.</p><p>A special <code>result</code> handler is called after each message is handled, if it
returns truthy then the loop finishes and Kaocha can continue.</p><p>Here’s an example of how <code>queue-consumer</code> is used</p><pre><code class="language-clojure">(queue-consumer {:queue queue
:timeout timeout
:handlers {:cljs/ret
(fn [{:keys [val] :as event} context]
(when (map? val)
(throw (ex-info "ClojureScript Error while loading Kaocha websocket client" event)))
(cond-> context
(= (str done) val)
(assoc :eval-done? true)))
:cljs/out
(fn [{:keys [val]} context]
(if (= "NODE_WS_NOT_FOUND\n" val)
(throw (ex-info "Nodejs: require('ws') failed, make sure to 'npm install ws'."
(merge limited-testable context)))
(println val))
context)
:timeout
(fn [{:keys [eval-done?] :as context}]
(if eval-done?
(throw (ex-info "Failed initializing ClojureScript runtime" (merge limited-testable context)))
(throw (ex-info "Kaocha ClojureScript client failed connecting back." (merge limited-testable context)))))}
:result (fn [context]
(and (:eval-done? context)
(:ws-client/ack? context)))})
</code></pre><p>This has turned out to be a surprisingly stable and reliable approach, that not
only does what it says on the tin, but also manages to fail better, since Kaocha
has ample context for error reporting.</p><p>The second use case was for Nextjournal. Nextjournal uses Datomic, but most of
the nodes that make up an article are not directly represented as Datomic
entities. Instead we store Transit encoded blobs, either of the full article
document state, or of a change event that updates a previous version.</p><p>This has worked quite well, since the application usually requires a complete
article document to work with, so building it up from individual entity nodes
each time would be a lot of extra work, but the downside is that we can’t query
for article nodes directly. So we decided to <a shape="rect" href="https://lambdaisland.com/blog/2019-12-01-advent-of-parens-1-clojure-vocab-reify">reify</a>
some of these, starting with “environments” (think: Docker images + a reference to the article that created them).</p><p>An article may have exported multiple versions of multiple environments over
time, so we need to loop over the complete article’s history. These in turn may
be based on environments from other articles, so to correctly link them to their
parent we need to reify those environments first. But when running this locally
I may not have all the articles that are referenced imported yet, so if any are
missing they need to be imported from production.</p><p>So I came up with another coffee grinder approach. This time the queue is part
of the context, similar to how it is in interceptors. I also split out side
effects here, so action handlers return <code>:tx-data</code>, which is then transacted
outside of the handler.</p><p>If I try to reify an environment but its parent is missing, then it will put the
reification of the parent on the queue, and then enqueue itself again
afterwards.</p><pre><code class="language-clojure">
(defn- run-queue!
"Naive task runner. Takes a context map consisting of a `queue` of actions, and
a datomic connection and initial db. The `handler` executes one action at a
time, returning an updated context map.
This action handler can manipulate the queue, i.e. prepend (lifo) or
append (fifo) actions. If the returned `ctx` contains `:tx-data` then this
will be transacted, and subsequent actions will receive the new database
value."
[ctx handler]
(loop [{:keys [conn db queue] :as ctx} ctx]
(let [[action & queue] queue]
(if action
(do
(log/debug (first action) (second action))
(let [ctx (handler action (assoc ctx :queue queue))
db (if-let [tx-data (:tx-data ctx)]
(do
(log/trace :tx-data tx-data)
(:db-after @(datomic/transact conn tx-data)))
db)]
(recur (dissoc (assoc ctx :db db) :tx-data))))
ctx))))
</code></pre><p>The handler in this case is not a map of functions but a multimethod.</p><pre><code class="language-clojure">(defmethod handle-action :import-article [[_ article-uuid] ctx]
(let [import-data (article/article-imports [article-uuid]
{:host "https://nextjournal.com"})]
(-> ctx
(add-tx-data import-data)
(queue-prepend [[:reify-article-environments [:nextjournal/id article-uuid]]]))))
</code></pre><p>And invoking the whole thing</p><pre><code class="language-clojure">(defn reify-article-environments
"Given a sequence of article ids (:db/id), go over the history of each article,
reifying all environments that are exported by runtimes, imported by
docker-environment nodes, or that are referenced by transclusion. For
non-production environments this will also import any article that is
referenced by transclusion.
See namespace level docs for some overarching considerations."
[conn db-ids]
(run-queue! {:conn conn
:db (datomic/db conn)
:queue (for [article-id db-ids]
[:reify-article-environments article-id])}
handle-action))
</code></pre><p>What I like about these approaches is the intermediate state that is built up
progresses in clear increments. In this way it’s very different from keeping
state in a mutable object with a bunch of fields. There’s a single point where
the state progresses, so it is always consistent, and you have access to the
previous and next values, so you can store them, diff them, save them to retry
etc.</p><p>I also like how in this last example both enqueuing new actions, and transacting
data into the database are accomplished by simple adding stuff to the context.
When functions need to query Datomic they also find the database value in the
context, so even though in a way these implementations do a lot of work, at
their core they are pure functions, making them a joy to test. (with the
exception of the article import).</p><p>Today has been a longer brain dump, but I’m looking forward to hear what people
think, and maybe inspire a few of you to try out similar patterns when you find
that what you are modeling is really more a process than a transaction. I’m
opening <a shape="rect" href="https://clojureverse.org/t/coffee-grinders/5218">a thread on ClojureVerse</a> for comments and discussion.</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/coffee-grinders/5218">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 15, jcmd and jstackhttps://lambdaisland.com/blog/2019-12-15-advent-of-parens-15-jcmd-and-jstack2019-12-15T19:12:50+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Two shell commands anyone using JVM languages should be familiar with are <code>jcmd</code>
and <code>jstack</code>. They are probably already available on your system, as they come
bundled with the JDK. Try it out, run <code>jcmd</code> in a terminal.</p><p>This is what the result might look like</p><pre><code>$ jcmd
4508 sun.tools.jcmd.JCmd
23330 clojure.main -m nrepl.cmdline --middleware ["refactor-nrepl.middleware/wrap-refactor", "cider.nrepl/cider-middleware", "cider.piggieback/wrap-cljs-repl"]
23053 clojure.main -m nrepl.cmdline --middleware ["refactor-nrepl.middleware/wrap-refactor", "cider.nrepl/cider-middleware", "cider.piggieback/wrap-cljs-repl"]
</code></pre><p>What it does is list all running JVM processes, with their process id (pid), the
class whose <code>main</code> function is being executed, and the arguments to said <code>main</code>
entrypoint. In this case I have two REPLs started from CIDER, and you also see
the jcmd process itself.</p><p>This output tends to be a but more useful than <code>ps ax | grep clojure</code>, as it
hides the often noisy JVM options like classpath, properties, and heap size.</p><p>Once you have found the pid with <code>jcmd</code>, you can invoke <code>jstack</code>.</p><pre><code>$ jstack 2330
Full thread dump OpenJDK 64-Bit GraalVM CE 19.1.0 (25.212-b03-jvmci-20-b04 mixed mode):
"main" #1 prio=5 os_prio=0 tid=0x00007f7828010800 nid=0x5b2f waiting on condition [0x00007f782d8ec000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at nrepl.cmdline$dispatch_commands.invokeStatic(cmdline.clj:441)
at nrepl.cmdline$dispatch_commands.invoke(cmdline.clj:425)
at nrepl.cmdline$_main.invokeStatic(cmdline.clj:448)
at nrepl.cmdline$_main.doInvoke(cmdline.clj:443)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
...
</code></pre><p>What you get is the full stack trace of every thread of the given process. This
can be a lot of output so you might want to pipe it to <code>less</code>.</p><p>This can be extremely useful when a process seems to be stuck and you want to
know what on earth it’s actually doing. For each thread it also tells you if its
active (<code>RUNNABLE</code>), or sleeping/blocked (<code>TIMED_WAITING</code>/<code>WAITING</code>).</p><p>Since it’s a lot of output you might want to filter it a bit. Most of the output
will not relate to your program, it’s either Java/JVM stuff like the GC thread,
or parts of the stack that are inside Clojure or in third party libraries,
whereas you usually want to know where in your code the program is executing.</p><p>Simply grepping can already be quite helpful here. Presumably you have a common
namespace prefix for your project, so you can grep for that.</p><pre><code>jcmd 2330 | grep com.nextjournal
</code></pre><p>But that will mash all threads together. To still keep each thread name and
state visible you could do something like this (this is copied verbatim from my
shell history).</p><pre><code>jstack 25235 | egrep '(^"|nextjournal|State)' | less
</code></pre><p><code>egrep</code> is a shorthand for <code>grep -E</code> (TIL <code>egrep</code> is considered deprecated, so
try <code>grep -E</code> if <code>egrep</code> doesn’t work). It switches <code>grep</code> from “basic” to
“extended” regular expressions, which only means that things like
<code>(foo|bar|baz)</code> groups work as you would expect. In basic grep you need to add
backslaches to a bunch of the special characaters, and I can never forget which
ones do need it and which ones don’t. (Is it <code>\(foo\|bar\|baz\)</code> or
<code>\(foo|bar|baz\)</code>?)</p><p>The thread name starts with a double quote as the first character on a line
(<code>^"</code>), my own namespaces contain in this case <code>nextjournal</code>, and the state of
the thread contains the word <code>State</code>.</p><p>You can now combine this with <code>watch</code> to keep an eye on your process as its
running. (Example again taken from my shell history directly).</p><pre><code>watch -n 0.2 'jstack 31758 | egrep "(environm|migrate|graph)"'
</code></pre><p>These are pretty basic tools, so YMMV, but I like how simple and lightweight
they are. When a process is already pinning all of my CPU cores I also don’t
have the luxury to spin up <code>visualvm</code> to capture profiling data, it just needs
too many resources. In this case <code>jcmd</code> and <code>jstack</code> can provide some early
insights cheaply.</p><p>(BTW <code>visualvm</code> is another great tool for another. Props to Alexander Yakushev
for introducing me to all three of these and several more in <a shape="rect" href="https://www.youtube.com/watch?v=HNp5EaRs3KI">his Dutch Clojure
Days talk</a> from a few years back.)</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 14, Why did the Clojurist drop out of school?https://lambdaisland.com/blog/2019-12-14-advent-of-parens-14-why-did-the-clojurist-drop-out-of-school2020-02-05T06:10:19+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Because they didn’t like classes.</p><p>Almost missed today’s post, so I’ll keep it short.</p><p>It’s fairly well known that you can put <code>set -e</code> at the top of a shell script to
exit the script when a command returns a non-zero exit status. What isn’t as
widely known is that <code>set -x</code> will cause every command to be printed out, so you
get a trace of what the script is doing.</p><p>And what is not nearly as widely known as it should be is that you can also
execute <code>bash -x some_script</code> to run that script with tracing on. Curious what
the <code>clj</code> / <code>clojure</code> scripts do exactly? Then try this:</p><pre><code>bash -x `which clojure`
</code></pre><p>What’s great about this is that any environment variables get expanded, so you really get the exact commands that are executed, and you can copy and tweak and re-run them as you like.</p><p>That’s it, see you tomorrow!</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p></<>>Advent 2019 part 13, Datomic Test Factorieshttps://lambdaisland.com/blog/2019-12-13-advent-of-parens-13-datomic-test-factories2019-12-13T09:13:23+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>When I started consulting for <a shape="rect" href="https://nextjournal.com/">Nextjournal</a> I helped
them out a lot with tooling and testing. Their data model is fairly complex,
which made it hard to do setup in tests. I created a factory based approach for
them, which has served the team well ever since.</p><p>First some preliminaries. At Nextjournal we’re big fans of Datomic, and so
naturally we have a Datomic connection as part of the Integrant system map.</p><pre><code class="language-clojure">{:com.nextjournal.journal.system/datomic-conn
#profile
{:default {:uri #or[#env JOURNAL_DATOMIC_URI "datomic:mem://nextjournal_dev"]}
:prod {:uri #or[#env JOURNAL_DATOMIC_URI #secret "journal/datomic/uri"]
:com.nextjournal.journal.system.vault/service #integrant/ref[:com.nextjournal.journal.system.vault/service]}
:test {:uri "datomic:mem://nextjournal_test"
:delete-db? true}}}
</code></pre><p>For testing we use a dynamic var <code>*test-system*</code>. You could just set up a system
and pass it around, but in this case a dynamic var is really convenient, since
this allows you to set it up in a fixture, and test helper functions can access
the system directly.</p><pre><code class="language-clojure">(ns ns com.nextjournal.journal.test-system)
(def ^:dynamic *test-system*)
;; REPL use only
(defn start! [] ,,,)
(defn stop! [] ,,,)
(defn wrap-system [test]
(binding [*test-system* (ig/init! config)]
(try
(test)
(finally
(ig/halt! *test-system*)))))
;; in a test namespace:
;; (use-fixtures :each test-system/wrap-system)
</code></pre><p>With that we can make helpers to quickly get the Datomic connection, get a
database value, or do a query. This already takes a lot of clutter out of tests.</p><pre><code class="language-clojure">(defn conn []
(:com.nextjournal.journal.system/datomic-conn *test-system*))
(defn db []
(datomic/db (conn)))
(defn q [query & args]
(apply datomic/q query (db) args))
</code></pre><p>To get test data into the database we have this handy <code>transact!</code> utility. It
transacts whatever transaction data you give it, and returns a map with resolved
temp-ids. If it finds any temp-ids that end in <code>"-id"</code>, then it will also return
a <code>(datomic/entity ,,,)</code> for that id.</p><pre><code>(defn transact*!
"Like `transact!` but takes the connection explicitly"
[conn & tx-datas]
(let [tx-data (vec (mapcat #(if (map? %) [%] %) tx-datas))
report @(datomic/transact conn tx-data)
db-after (:db-after report)
tempids (:tempids report)
ids (into {} (keep (fn [{id :db/id}]
(when (string? id)
[(keyword id)
(datomic/resolve-tempid db-after tempids id)]))
tx-data))]
(reduce-kv (fn [ids kid id]
(let [kidn (name kid)]
(if (str/ends-with? kidn "-id")
(let [k (keyword (str/replace kidn #"-id$" ""))
e (datomic/entity db-after id)]
(assoc ids k e))
ids)))
ids
ids)))
(defn transact!
"Transact data and resolve temp-ids.
Calls datomic/transact on the current connection in *system*, and :db/id in
the input that are strings are resolved as tempids, keywordified, and returned
as a map. On top of that any keys ending in \"-id\" are also resolved as
entities, dropping the -id suffix.
Each argument can be either a map or a vector of maps.
See also: defactory.
"
[& tx-datas]
(apply transact*! (conn) tx-datas))
</code></pre><p>So for example say I’m creating an article and a person entity for use in my test, this will look something like this.</p><pre><code class="language-clojure">(let [{:keys [article article-id
person person-id]}
(transact! {:db/id "article-id" :article/change 1234}
{:db/id "person-id" :person/email "foo@bar.com"})]
article ;;=> {:db/id 158953 ,,,}
article-id ;;=> 158953
person ;;=> {:db/id 158954 ,,,}
article-id ;;=> 158954
)
</code></pre><p>I can destructure whatever I need out of that. Most of the time it’s most
convenient to work with the entity maps, but if you need the ids instead to pass
on to another function then you can access those directly.</p><p>With all of that set up it’s time for the grand finale: <code>defactory</code>!</p><pre><code class="language-clojure">(defmacro defactory
"Define a datomic factory helper. This should look like a function which takes a
single argument (an options map), and returns datomic transaction data, either
a single map or a vector of maps.
Use strings for temp-ids if possible.
This will create a regular function that just returns that data, but also a
method ending in a bang which will transact the tx-data against the
*test-system*, and return a map of resolved temp-ids."
[name & [a1 a2 & rst :as args]]
(let [[docstring arglist body]
(if (string? a1)
[a1 a2 rst]
["Factory function" a1 (next args)])]
`(do
(defn ~name ~docstring [& ~arglist]
(let [res# ~@body]
(if (map? res#)
[(dissoc res# :suffix)]
(mapv #(dissoc % :suffix) res#))))
(defn ~(symbol (str name "!")) ~docstring
[& [opts#]]
(let [conn# (:datomic/conn opts# (conn))
opts# (dissoc opts# :datomic/conn)]
(transact*! conn# (~name opts#)))))))
</code></pre><p>This is already a hairy macro, I guess you don’t always want to see how the
sausage gets made, but in usage it’s really nice.</p><pre><code class="language-clojure">(ns com.nextjournal.journal.test-factories)
(defactory profile
"Creates a profile entity with random handle. Options are merged in directly.
Provides a \"profile-id\" tempid."
[opts]
(merge #:profile
{:db/id "profile-id"
:handle (rand-username)
:name "Jonny Zimmerman"}
opts))
(defactory person
"Creates a person entity. Options are merged in directly. The
`:person/password` key is treated special, it is converted to a password
digest and stored as such. Expects a \"profile-id\" temp-id in the same
transaction.
Provides a \"person-id\" tempid."
[opts]
(merge #:person
{:db/id "person-id"
:profile "profile-id"
:email (rand-email)}
(cond-> opts
(:person/password opts)
(-> (dissoc :person/password)
(assoc :person/password-digest
(password/encrypt (:person/password opts)))))))
</code></pre><p>This has created four functions: <code>profile</code>, <code>profile!</code>, <code>person</code>, and <code>person!</code>.
The versions without a bang just return transaction data.</p><pre><code class="language-clojure">(profile)
;;=> {:db/id "person-id", :profile/handle "jonny7", :profile/name "Jonny Zimmerman"}
(profile {:profile/name "Arne Brasseur" :profile/website "https://lambdaisland.com"})
;;=> {:db/id "person-id", :profile/handle "jonny7", :profile/name "Arne Brasseur" :profile/website "https://lambdaisland.com"}
(let [{:keys [person]} (transact! (person) (profile))]
(-> person :person/profile :profile/handle) ;;=> "marcus123"
)
</code></pre><p>The version with a bang immediately call <code>transact!</code>.</p><pre><code class="language-clojure">(let [{:keys [profile]} (profile!)]
(profile :profile/handle) ;;=> "marcus123"
)
</code></pre><p>We have a bit more logic in the factories to make it easy to create multiple
entities of the same type in a single transaction, in that case they will get
temp-ids like <code>article-1-id</code>, so you can destructure them as <code>{:keys [article-1
article-2]}</code>.</p><p>Is this API perfect? Far from it, there are definitely some things I would
reconsider if I did a second iteration of this. It’s also not the kind of code I
would write for regular, non-test use. It already starts to smell a little of
“magic”, the kind that as a recovering Rubyist I have learned to avoid. Pulling
up database connections from behind the covers, checking for the type of
arguments to have a more flexible interface, transacting things
“automatically”… these are things Clojure has taught me to avoid, generally.</p><p>But in this case the magic does serve a higher purpose, it is taking the
friction out of writing unit tests, and anything that helps and encourages
people to test their code is a win in my book. And because it takes a lot of
clutter and boilerplate out of the tests they start to better convey intention.
Besides being there to test stuff tests also act as a form of documentation.
They show examples of how APIs can be used, and in the case of a failure they
point at the scenario that is causing trouble. So all in all we think these
factories are a win.</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p></<>>Advent 2019 part 12, Pairing in the Cloud with Tmuxhttps://lambdaisland.com/blog/2019-12-12-advent-of-parens-12-pairing-cloud-tmux2019-12-12T11:40:51+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>I’m a strong believer in pair programming. It can be intense and exhausting, and
its a skill you need to learn and get good at, but it’s extremely valuable. It
improves knowledge sharing, prevents mistakes, and helps people to stay on track
to make sure they are building the right thing, which is arguably one of the
hardest aspects of our job.</p><p>But <a shape="rect" href="http://gaiwan.co">Gaiwan</a> is a remote-first company. We are spread out
across Germany, Brazil, Italy, and work with clients as far away as Singapore
and Hong Kong, so we need good ways to pair remotely. For this we need a tool
that is</p><ul><li>cross platform, at least across Linux and mac, this rules out most of the
newer tools that try to solve this problem</li><li>symetrical, the experience should be the same for both participants, this
rules out simple screen sharing, as well as most desktop sharing or VNC style
tools</li><li>responsive, nothing is worse than a laggy editor</li></ul><p>This may seem like an impossible set of requirements, but sometimes all you need
to do is change your perspective, and instead of looking at the shiniest new
developments, look at the classics. Unix once again comes to our rescue!</p><p>What we do nowadays is spin up a linux machine in the cloud, both participants
ssh into that, and then we use <code>tmux</code> to share the screen. This means you have
to use a terminal based editor. For this we use Spacemacs in hybrid mode. This
allows quickly switching between Emacs-style and vim-style keybindings, which is
a good compromise.</p><p>We have most of the setup automated, including pulling SSH keys off of Github so
people can immediately log in, and setting up the tools and dependencies we use.
Finally we use <code>ngrok</code> to set up a HTTP tunnel so you can still access the app
you’re working on in your own browser.</p><p>This you then combine with whatever audio/video solution you prefer. We tend to
use <a shape="rect" href="https://whereby.com">Whereby</a> (previously Appear.in).</p><p>This works really well because it’s literally just a bunch of characters going
over the wire, rather than complete video frames. It also means you can still
copy/paste into and out of your terminal/editor, which is a nice plus.</p><p>At Clojure/Conj this year I talked to some folks from
<a shape="rect" href="https://www.viasat.com/">Viasat</a> about this approach, and turns out they do
almost the exact same thing, which was very validating. One nice touch is that
they equip everyone with a hardware SIP speaker phone, to get the lowest latency
and highest fidelity audio. That’s definitely something we will consider as
well.</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about
application architecture, development process, tooling and testing. I
collaborate with other talented people under the banner
<a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about how we could help
you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 11, Integrant in Practicehttps://lambdaisland.com/blog/2019-12-11-advent-of-parens-11-integrant-in-practice2020-07-01T08:16:56+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>I’ve been a fan of <a shape="rect" href="https://github.com/weavejester/integrant">Integrant</a> pretty
much ever since it came out. For me there is still nothing that can rival it.</p><p>The recently released <a shape="rect" href="https://github.com/juxt/clip">clip</a> by the folks from
Juxt does deserve an honorable mention. It has an interesting alternative
approach which some may prefer, but it does not resonate with me. I prefer my
system configuration to be just data, rather than code wrapped in data.</p><p>However there’s another Juxt library that I do use on pretty much any project
these days, and that combines wonderfully with Integrant:
<a shape="rect" href="https://github.com/juxt/aero">Aero</a>. Aero is basically just an EDN reader, but
it provides half a dozen reader tags that make it ideal for system
configuration, and so it pairs with Integrant like camembert with Bordeaux.</p><p>Aero provides things like <code>#env MY_ENV_VAR</code>, <code>#include "my_other_file.edn"</code>, or
<code>#profile {:prod 80 :dev 8080}</code>.</p><p>So the first thing I’ll do on a project is usually to set up <code>integrant</code>,
<code>integrant-repl</code>, and <code>aero</code>. Let’s see what that looks like.</p><p>The starting point for using Integrant is your configuration map, which goes in
a file named <code>resources/app_name/system.edn</code>. So if your main namespace is
<code>feralberry.core</code>, then this goes into <code>resources/feralberry/system.edn</code>. This
way it can be found on the classpath, and bundled into a jar, without
potentially clashing with other files on the classpath.</p><p>For instance:</p><pre><code class="language-clojure">{:feralberry.http/router {:routes [["/" {:get {:interceptors :index}}]]}
:feralberry.http/server {:port 2533
:router #ig/ref :feralberry.http/router}
:feralberry.storage/crux {:crux.node/topology :crux.kafka/topology
:crux.node/kv-store crux.kv.rocksdb/kv
:crux.kv/db-dir "data/db-dir-1"
:crux.kafka/bootstrap-servers "localhost:9092"}}
</code></pre><p>Next we need a way to read this configuration:</p><pre><code class="language-clojure">(ns feralberry.system
(:require [aero.core :as aero]
[clojure.java.io :as io]
[integrant.core :as ig]))
(defmethod aero/reader 'ig/ref
[_ tag value]
(ig/ref value))
(defn config [profile]
(aero/read-config (io/resource "feralberry/system.edn") {:profile profile}))
(defn prep [profile]
(let [config (config profile)]
(ig/load-namespaces config)
config))
</code></pre><p>Here I define an extra reader tag for Integrant references, and a helper to read
in the configuration for a given profile (like <code>:dev</code>, <code>:prod</code>, <code>:test</code>).
Finally I add a helper function <code>prep</code> which tries to load any namespaces
referenced in the configuration map, and then returns the configuration.</p><p>Next we need a way to start the system from the REPL, this is done in the <code>user</code>
namespace, since this gets loaded automatically during development, so these
functions are immediately available after booting up.</p><pre><code class="language-clojure">(ns user)
(defmacro jit
"Just in time loading of dependencies."
[sym]
`(requiring-resolve '~sym))
(defn set-prep! []
((jit integrant.repl/set-prep!) #((jit feralberry.system/prep) :dev)))
(defn go []
(set-prep!)
((jit integrant.repl/go)))
(defn reset []
(set-prep!)
((jit integrant.repl/reset)))
(defn system []
@(jit integrant.repl.state/system))
(defn config []
@(jit integrant.repl.state/config))
</code></pre><p>This code looks a bit weird because of the <code>jit</code> thing. The reason I do this is
I don’t like having any <code>:require</code> in my <code>user</code> ns, since it slows down the TTR
(Time To REPL), so I only load stuff on demand. I also added two helpers to
easily inspect the system and the current configuration from the REPL.</p><p>So now I have a nice reloaded workflow. Just <code>(go)</code> after booting the REPL, or
<code>(reset)</code> to reload namespaces and restart.</p><p>With that the boilerplate is out of the way and I can start implementing my
system components, for instance:</p><pre><code class="language-clojure">(ns feralberry.storage
(:require [integrant.core :as ig]
[crux.api :as crux]))
(defmethod ig/init-key ::crux [_ config]
(crux/start-node config))
(defmethod ig/halt-key! ::crux [_ crux]
(.close crux))
</code></pre><p>Note that I don’t need to explicitly require this namespace, it will get loaded
by integrant based on the configuration key <code>:feralberry.storage/crux</code>.
Alternatively I could have put these methods in a <code>feralberry.storage.crux</code>
namespace, that would have worked as well.</p><p>Since Integrant is simply a library it isn’t always clear to people how to use
it in practice. I hope this post was helpful in showing at least one
pattern/convention for setting things up.</p><p><em>Hi, my name is Arne (aka @plexus) and I consult companies and teams about application
architecture, development process, tooling and testing. I collaborate with other
talented people under the banner <a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have
a chat about how we could help you with your project then please <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 10, Hillcharts with Firebase and Shadow-cljshttps://lambdaisland.com/blog/2019-12-10-advent-of-parens-10-hillchart-reagent-shadow-firebase2019-12-10T10:45:32+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Recently I led a workshop for a client to help them improve their development
process, and we talked a lot about <a shape="rect" href="https://basecamp.com/shapeup">Shape Up</a>, a
book released by Basecamp earlier this year that talks about <em>their</em> process.
You can read it for free on-line, and I can very much recommend doing so. It’s
not a long read and there are a ton of good ideas in there.</p><p>One of these ideas has also become a feature in Basecamp, namely hill charts.
These provide a great way to communicate what stage a piece of work is in. Are
you still going uphill, figuring things out and discovering new work, or are you
going downhill, where it’s mostly clear what things will look like, and you’re
just executing what you discovered?</p><p><a shape="rect" href="https://hillchart-91fcf.firebaseapp.com/#a3e77804-4aee-44c2-808c-70ff37d803a6"><img src="https://hillchart-91fcf.firebaseapp.com/a3e77804-4aee-44c2-808c-70ff37d803a6.svg" alt="Hillchart" height="600px" width="1000px" /></a></p><p>The team really liked this idea, but they’re not planning to switch to Basecamp,
since they’re quite heavily invested in another tool. So during breaks and on
the way home I wrote up a little app that does just that. Click on the chart
above and you can see it in action. Simply remove the uuid from the URL to get a
new chart.</p><p>The UX could still be vastly improved, but with some clicking around hopefully
it’s easy enough to figure out how it works. Click on the big plus to add a dot,
click on a dot to start moving it around, click again to put it down. The “M”
button on the right gives you a markdown snippet that you can paste for instance
in Github or Gitlab, and the button below that lets you edit the captions. The
code can be found on Github: <a shape="rect" href="https://github.com/plexus/hillchart">plexus/hillchart</a>.</p><p>I used Firebase for this project which was a first for me, and I have to say the
experience was pretty great. Firebase basically does three things: it serves the
app (HTML/CSS/JS), it acts as a database, and using a Firebase Cloud Function
(akin to an AWS Lambda), it can render a chart to SVG, which is then used by the
Markdown snippet to embed the chart in other platforms.</p><p>It’s really neatly self-contained that way, and by using the Firestore storage
you get synchronization basically for free. Try opening the same chart in two
tabs and edit it and you’ll see what I mean.</p><p>Matt’s article on the Applied Science blog <a shape="rect" href="http://www.appliedscience.studio/articles/cljs-firebase-functions.html">ClojureScript on Firebase Cloud
Functions</a>
was very helpful to set things up. I initially started the project with
Figwheel, but switched over to Shadow-cljs as it seemed to make using the
Firebase npm package a bit easier. That was another first for me and I have to
say it was a very positive experience.</p><p>Go ahead and browse around the code. It’s really just a Reagent app. There are
several bits that would be interesting to write about, I’ll just show how the
Firestore integration works. (slightly redacted)</p><pre><code class="language-clojure">(ns hillchart.main
(:require [reagent.core :as r]))
;; Reagent app state
(defonce state (r/atom {:dots []}))
;; Each chart is identified by a random UUID, if there is no UUID yet in the URL
;; then we'll generate one, i.e. create a new chart
(when (= "" js/document.location.hash)
(set! js/document.location.hash (random-uuid)))
;; Grab the current ID, this will stay fixed
(def chart-id (subs js/document.location.hash 1))
;; Initialize firestore, grab the charts collection, and grab the document with
;; the current chart-id.
(def ^js db (js/firebase.firestore))
(def ^js fs-charts (.collection db "charts"))
(def ^js fs-doc (.doc fs-charts chart-id))
;; Helper that merges the state coming from the db into the reagent state
(defn fs->state! [^js doc]
(swap! state merge (js->clj (.data doc) :keywordize-keys true)))
;; Grab the data the first time, the `defonce` is just so this doesn't happen
;; again when hot reloading.
(defonce fetch-doc
(.then (.get fs-doc) fs->state!))
;; Listen for changes
(defonce setup-listener
(.onSnapshot fs-doc fs->state!))
;; Whenever a change happens I'll call this to push changes to the db. I used a
;; watch on the atom first, but since there are things in there like the current
;; viewport size that don't need to be stored I figured this worked better.
(defn save-doc! []
(.set fs-doc (clj->js (select-keys @state [:dots]))))
</code></pre><p>And this is the code that powers the cloud function, so an SVG version can be
rendered on the server-side. Data access is a little different here, but it
comes down to the same thing.</p><pre><code>(ns hillchart.firebase
(:require ["firebase-functions" :as functions]
["firebase-admin" :as admin]
[hillchart.main :as main]
[reagent.dom.server]
[clojure.string :as str]))
(.initializeApp admin)
(defn render-chart-svg [^js req, ^js res]
(let [doc-id (str/replace (str/replace (.-path req) ".svg" "") "/" "")
^js db (.firestore admin)
^js fs-charts (.collection db "charts")
^js fs-doc (.doc fs-charts doc-id)]
(.then (.get fs-doc)
(fn [doc]
(.type res ".svg")
(.send res
(reagent.dom.server/render-to-static-markup
[main/Chart (assoc (js->clj (.data doc) :keywordize-keys true)
:screen/width 500
:screen/height 300
:static? true)])))))) ;;<- hide the buttons
(def cloud-functions
#js {:renderChartSVG (.onRequest functions/https render-chart-svg)})
</code></pre><p><em>Hi, my name is Arne and I consult companies and teams about application
architecture, development process, tooling and testing. I collaborate with other
talented people under the banner <a shape="rect" href="http://gaiwan.co">Gaiwan</a>. If you like to have a chat about any of these things just <a shape="rect" href="/p/contact">get in touch!</a></em></p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 9, Dynamic Vars in ClojureScripthttps://lambdaisland.com/blog/2019-12-09-advent-of-parens-9-dynamic-vars-clojurescript2019-12-11T09:13:09+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Clojure has this great feature called Dynamic Vars, it lets you create variables
which can be dynamically bound, rather than lexically. Lexical (from Ancient
Greek λέξις (léxis) <em>word</em>) in this case means “according to how it is written”.
<code>let</code> bindings for instance are lexical.</p><pre><code class="language-clojure">(defn hello [x]
(str "hello " x))
(defn greetings []
(str "greetings" foo)) ;; *error*
(let [foo 123]
(hello foo)
(greetings))
</code></pre><p>The <code>let</code> introduces a <em>lexical</em> binding for <code>foo</code>, it can be accessed in
<code>(hello foo)</code> because lexically (textually) it’s within the <code>let</code>. <code>greetings</code>
is outside of the let so it can’t “see” <code>foo</code>.</p><p><em>Dynamic</em> bindings are introduced with <code>binding</code>, and are visible within the
same call stack. Any function that is called on the same thread can “see” the
binding. For this to work you need to add metadata marking the var as
<code>:dynamic</code>, and you need to give the name earmuffs (that’s what those asterisks
are called. Cute, right?)</p><pre><code class="language-clojure">(def ^:dynamic *foo*)
(defn greetings []
(str "greetings" *foo*)
(binding [*foo* 123]
(greetings))
</code></pre><p>Ok so this is great and handy (although don’t overuse it, but that’s a topic for
another time). But how does this work in ClojureScript. For one ClojureScript
does not have vars, they compile to plain JavaScript globals. JavaScript (and
thus ClojureScript) also doesn’t have threads, so the mechanisms provided by
vars like root binding and thread bindings are not relevant. But dynamic vars do
exist in ClojureScript… curious! I’ve been wondering for some time now how
they work, and today I finally investigated.</p><p>Here’s how <code>binding</code> is implemented. Curious indeed! It calls the analyzer to
check if the vars are actually defined as dynamic, and if that doesn’t throw a
compile-time error it will just call <code>with-redefs</code>.</p><pre><code class="language-clojure">(core/defmacro binding
"binding => var-symbol init-expr
Creates new bindings for the (already-existing) vars, with the
supplied initial values, executes the exprs in an implicit do, then
re-establishes the bindings that existed before. The new bindings
are made in parallel (unlike let); all init-exprs are evaluated
before the vars are bound to their new values."
[bindings & body]
(core/let [names (take-nth 2 bindings)]
(cljs.analyzer/confirm-bindings &env names)
`(with-redefs ~bindings ~@body)))
</code></pre><p><code>with-redefs</code> is even more curious.</p><pre><code class="language-clojure">(core/defmacro with-redefs
"binding => var-symbol temp-value-expr
Temporarily redefines vars while executing the body. The
temp-value-exprs will be evaluated and each resulting value will
replace in parallel the root value of its var. After the body is
executed, the root values of all the vars will be set back to their
old values. Useful for mocking out functions during testing."
[bindings & body]
(core/let [names (take-nth 2 bindings)
vals (take-nth 2 (drop 1 bindings))
orig-val-syms (map (comp gensym #(core/str % "-orig-val__") name) names)
temp-val-syms (map (comp gensym #(core/str % "-temp-val__") name) names)
binds (map core/vector names temp-val-syms)
resets (reverse (map core/vector names orig-val-syms))
bind-value (core/fn [[k v]] (core/list 'set! k v))]
`(let [~@(interleave orig-val-syms names)
~@(interleave temp-val-syms vals)]
~@(map bind-value binds)
(try
~@body
(finally
~@(map bind-value resets))))))
</code></pre><p>This one takes a bit more effor to read, but the clue is in this line:
<code>bind-value (core/fn [[k v]] (core/list 'set! k v))</code>. Basically what the example
from above translates to is:</p><pre><code class="language-clojure">(set! *foo* 123)
(try
(greetings)
(finally
(set! *foo* nil)))
</code></pre><p>That’s it, that’s all there is. Somewhat anti-climactic I must admit. It just
sets the value, calls your code, and sets it back afterwards.</p><p>(<a shape="rect" href="https://github.com/clojure/clojurescript/blob/23cedecbf4f704f9fee672e395bbfa1e3fe3ee1a/src/main/clojure/cljs/core.cljc#L2232-L2267">with-redefs and binding in context</a>)</p><p>So do you need <code>:dynamic</code> in ClojureScript? Or can you just use <code>with-redefs</code>,
or even just <code>set!</code>? It looks like it, although there are of course good reasons
(readability, showing intent) to still use them if you want to rebind something
later on.</p><p>So case closed? Not really. I had this nagging suspicion that there might still
be more going on, so I went through the compiler and analyzer to see what they
do with things marked as <code>:dynamic</code>, and sure enough in <code>cljs.compiler</code> we find
this:</p><pre><code class="language-clojure">(defmethod emit* :invoke
[{f :fn :keys [args env] :as expr}]
(let [info (:info f)
fn? (and ana/*cljs-static-fns*
(not (:dynamic info))
(:fn-var info))
,,,] ,,,))
</code></pre><p>So what are we looking at? This is the code that emits (prints out JavaScript)
function invocations. Usually this just emits things like <code>foo.call(123)</code>, but
there’s an optimization you can enable that makes calling multi-arity functions
a bit faster. When you have a function like</p><pre><code class="language-clojure">(defn foo
([arg1] ,,,)
([arg1 arg2] ,,,))
</code></pre><p>Then ClojureScript will emit three functions: one that takes one argument, one
that takes two, and one that dispatches to the right one based on
<code>arguments.length</code>.</p><p>When you enable the <code>:static-fns</code> compiler option then ClojureScript will try to
avoid calling the dispatch function, calling the specific arity versions
directly. But… this would break when you rebind the dispatch function, so if
<code>:static-fns</code> is enabled, and the function you’re calling is dynamic, then emit
a regular <code>foo.call(...)</code>.</p><p>(Note that there are some caveats with <code>:static-fns</code>. At the ClojureScript
Internals workshop that David Nolen gave in Berlin earlier this year he
mentioned it may actually perform worse than when you leave it off. I don’t
remember the exact details, it might have been because it prevents some Google
Closure Compiler optimizations.)</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 8, Everything is (not) a pipehttps://lambdaisland.com/blog/2019-12-08-advent-of-parens-8-everything-is-a-pipe2019-12-09T14:55:37+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>I’ve always been a big UNIX fan. I can hold my own in a shell script, and I
really like the philosophy of simple tools working on a uniform IO abstraction.
Uniform abstractions are a huge enabler in heterogenous systems. Just think of
Uniform Resource Locators and Identifier (URLs/URIs), one of the
<a shape="rect" href="https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm">cornerstones</a> of
the web as we know it.</p><p>Unfortunately since coming to Clojure I feel like I’ve lost of some of that
power. I’m usually developing against a Clojure process running inside (or at
least connected to) my trusty editor, and the terminal plays second fiddle. How
do I pipe things into or out of that?</p><p>On top of that consider that most UNIX tools are line based (grep, awk, sed,
cut, …), how do I marry that with Clojure’s EDN view of the world (another
fabulous uniform abstraction).</p><p>I already covered <code>nc</code>, <code>rep</code>, and <code>jet</code> in a previous post. Each of these is
just a single focused tool, but together they start to form a kind of Swiss army
knife. Today I’ll look at a few more of these: HTTPie, Pup, kafkacat and websocat.</p><p><a shape="rect" href="https://httpie.org/">HTTPie</a> is a tasty alternative to <code>curl</code>. It’s a bit more
intuitive when it comes to passing along custom headers or parameters, and it
can pretty print and format JSON. I have to thank Tommi from Metosin for
indirectly introducing me to this one, I came across it in one of the Metosin
READMEs (can’t remember which one).</p><p>After adoping HTTPie (the command is simply <code>http</code>) life was great, until I
tried to use it to connect to a websocket. Websockets are kind of HTTP so I
thought it might work, but they’re also kind of their own thing so I don’t blame
the devs for leaving this out. Focused tools, right? Fortunately there’s
<a shape="rect" href="https://github.com/vi/websocat">websocat</a> to do just that.</p><p>See how there’s a common theme here? We have various mechanisms for shuttling
data from one place to another, either as a stream, or a single shot transfer,
and we’re simply hooking these to UNIX pipes. (Macbook owners can think of
dongles as an apt metaphor).</p><p>The final tool I want to mention is <code>kafkacat</code>, you can probably guess by now
what it does. I’ve been poking around a bit with Kafka lately, and having a
simple command line client for both sending and receiving was super helpful to
debug and better understand things.</p><p>So this is your set of dongles:</p><ul><li>TCP streams: <code>nc</code></li><li>Websocket streams: <code>websocat</code></li><li>HTTP requests: <code>http</code></li><li>kafka topics: <code>kafkacat</code></li><li>nREPL connections: <code>rep</code></li></ul><p>But just plugging things together isn’t always enough, they also need to speak
the same format. In the Clojure world we like things like Transit and EDN, but
the rest of the world speaks JSON, or XML/HTML, or CSV, and UNIX tools like to
have simple lines of text, so we need to convert between these formats.</p><p>I already mentioned <code>jet</code>, and <code>jq</code> is helpful for extracting things from JSON.
The final one I want to introduce in this series is <code>pup</code>, which is like <code>jq</code>
for HTML.</p><p>For instance:</p><pre><code class="language-sh">get-latest-datomic-version () {
http https://my.datomic.com/downloads/free | pup .latest attr{href} | sed 's^/downloads/free/^^'
}
</code></pre><p>Or say so you are putting Transit on a Kafka topic, but you want to see that as
pretty printed EDN to inspect what’s going over the wire? easy peasy.</p><pre><code class="language-sh">kafkacat -b 127.0.0.1:9092 -t topic-1 | jet --from transit --pretty
</code></pre><ul><li><code>jet</code> convert/query EDN/Transit/JSON ; pretty-print EDN</li><li><code>jq</code> convert/query/pretty-print JSON</li><li><code>pup</code> extract from HTML/XML</li></ul><p>And of course all the classics are still there, <code>sed</code>, <code>grep</code>, <code>awk</code>, <code>cut</code>,
<code>head</code>, <code>tail</code>, <code>tee</code> and many more.</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 7, Do that dotohttps://lambdaisland.com/blog/2019-12-07-advent-of-parens-7-do-that-doto2019-12-07T19:43:58+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p><code>doto</code> is a bit of an oddball in the Clojure repertoire, because Clojure is a
functional language that emphasizes pure functions and immutabilty, and <code>doto</code>
only makes sense when dealing with side effects.</p><p>To recap, <code>doto</code> takes a value and a number of function or method call forms. It
executes each form, passing the value in as the first argument. At the end of
the ride it returns the original value.</p><p>The typical example of using <code>doto</code> is for dealing with mutable Java objects</p><pre><code class="language-clojure">(doto (java.util.HashMap.)
(.put "hello" "world")
(.put "hi" "earth"))
;; => {"hi" "earth", "hello" "world"}
</code></pre><p>And it really cleans up invoking the Builder pattern.</p><pre><code class="language-clojure">(.build
(doto (FooBuilder.)
(.setBar 123)
(.toggleQux)
(.setName "hello")))
</code></pre><p>But what I want to show is that there are a few places where <code>doto</code> is useful
when you’re not doing Java interop.</p><p>The first is with <code>prn</code> or <code>println</code>. These functions return <code>nil</code>, so if you
want to quickly add a <code>prn</code> to check what a function is returning you have to
work around that, for instance by adding a let.</p><pre><code class="language-clojure">;; say you have something like this
(foo (bar (baz x)))
;; what's coming back from `baz`?
;; this does not work:
(foo (prn (bar (baz x))))
;; you could do this, but that's a mouthful
(let [x (bar (baz x))]
(prn x)
(foo x))
;; instead throw in a `(doto ... prn)`
(foo (doto (bar (baz x)) prn))
</code></pre><p>The cool thing is this also works in thread-first forms (not in thread-last
though), just throw that <code>(doto prn)</code> in between any two forms.</p><pre><code class="language-clojure">(-> x
baz
(doto prn)
bar
foo)
</code></pre><p>Or with some markers to make the output a bit more obvious:</p><pre><code class="language-clojure">(-> x
baz
(doto (prn '<-baz))
bar
(doto (prn '<-bar))
foo)
</code></pre><p>You can also throw in a <code>(doto ... assert)</code> if you’re getting paranoid that
there’s a <code>nil</code> somewhere where you don’t expect it.</p><p>And if you’re using Datomic from a REPL or from tests, and you want to be able
to quickly recreate an in-memory database, you can do this.</p><pre><code class="language-clojure">(let [uri (doto "datomic:mem:test1" d/delete-database d/create-database)
conn (d/connect uri)]
,,,)
</code></pre><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 6, A small idiomhttps://lambdaisland.com/blog/2019-12-06-advent-of-parens-6-a-small-idiom2019-12-06T12:09:03+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>As an avid tea drinker I’ve been poring (pouring?) over this catalog of teas.</p><pre><code class="language-clojure">(def teas [{:name "Dongding"
:type :oolong}
{:name "Longjing"
:type :green}
{:name "Baozhong"
:type :oolong}
{:name "Taiwan no. 18"
:type :black}
{:name "Dayuling"
:type :oolong}
{:name "Biluochun"
:type :green}])
</code></pre><p>I like all kinds of teas, but my favorites are definitely oolongs, so I want to
filter this list. Here’s a common way to do that.</p><pre><code class="language-clojure">(filter #(= :oolong (:type %)) teas)
;; => ({:name "Dongding", :type :oolong}
;; {:name "Baozhong", :type :oolong}
;; {:name "Dayuling", :type :oolong})
</code></pre><p>I’m sure many of you have written anonymous functions just like this one, I
certainly have written my fair share, but nowadays I would write this
differently.</p><pre><code class="language-clojure">(filter (comp #{:oolong} :type) teas)
</code></pre><p>This may or may not look strange, depending on how used you are to using sets as
functions, and composing with <code>comp</code>, but it just looks so much cleaner to me.
It also doesn’t create a closure. Anonymous functions close over all local
variables, so as long as there’s a live reference to it none of those local
variables can be garbage collected.</p><p>The real kicker however is that this is so nicely and naturally extensible.</p><pre><code class="language-clojure">;; get all black or oolong teas
(filter (comp #{:black :oolong} :type) teas)
;; => ({:name "Dongding", :type :oolong}
;; {:name "Baozhong", :type :oolong}
;; {:name "Taiwan no. 18", :type :black}
;; {:name "Dayuling", :type :oolong})
;; get all non-oolong teas
(filter (comp (complement #{:oolong}) :type) teas)
;; => ({:name "Longjing", :type :green}
;; {:name "Taiwan no. 18", :type :black}
;; {:name "Biluochun", :type :green})
</code></pre><p>If <code>(some #{:foo} coll)</code> can be mainstream, then surely <code>(comp #{:foo} :bar)</code>
can be too, so go forth and <code>comp</code>!</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 5, Clojure in the shellhttps://lambdaisland.com/blog/2019-12-05-advent-of-parens-5-clojure-in-the-shell2019-12-05T16:57:16+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>I already showed you netcat, and how it combines perfectly with socket REPLs.
But what if all you have is an nREPL connection? Then you use
<a shape="rect" href="https://github.com/eraserhd/rep">rep</a></p><pre><code>$ rep '(clojure.tools.namespace.repl/refresh)'
:reloading ()
:ok
</code></pre><p><code>rep</code> will find a <code>.nrepl-port</code> file, or you can specify the port directly.</p><pre><code>rep -p 4829 "(require 'foo.bar :reload)"
</code></pre><p>Unfortunately <code>rep</code> is not able to read code from STDIN, making it the odd one
out in my exposition of command line tools, but there’s a pull request open for
that.</p><p>Rep is part of a new breed of Clojure tools that use GraalVM to compile to
native binaries, giving them super fast start-up times.</p><p>Finally using Clojure from the shell has become tractable, which must have been
what Michiel Borkent (a.k.a. borkdude) was thinking when he created
<a shape="rect" href="https://github.com/borkdude/babashka">Babashka</a>, or <code>bb</code> for friends.</p><pre><code class="language-shell">$ bb -e '(+ 1 1)'
2
</code></pre><p>My favorite part of <code>bb</code> are the <code>-I</code> and <code>-O</code> options. The first will read and
parse consecutive edn forms from STDIN, and make them available as a lazy
sequence bound to <code>*in*</code>. The second will print the return value of your program
as newline-delimited EDN forms.</p><p>Here’s a little loop to get the download numbers for Kaocha for the past ten days</p><pre><code class="language-clojure">for i in {1..10} ; do curl -s https://clojars.org/stats/downloads-`date -d "now - $i days" "+%Y%m%d"`.edn; done |
bb -I -O '(->> *in* (map #(get % ["lambdaisland" "kaocha"])) (map vals) (map (partial apply +)))'
</code></pre><p>But what if instead of EDN you have some other format, like <a shape="rect" href="http://jsonlines.org/">newline-delimited
JSON</a>? Then you use another one of Michiel’s projects,
and a perfect complement to Babashka: <a shape="rect" href="https://github.com/borkdude/jet">Jet</a>!</p><pre><code class="language-clojure">cat log.json | jet --from json --to edn | bb -I '(map ,,, *in*)'
</code></pre><p>It can convert back and forth between JSON, EDN, and Transit, and it can pretty
print. So say you have an big old ugly Transit blob and you want to actually
look at what’s in there, you could do this.</p><pre><code>jet --from transit --pretty < blob.transit | less
</code></pre><p>Pretty neat. It also has some querying capabilities similar to <code>jq</code>, and an
interactive mode, although for complex querying I would probably just use
Babashka, since that way I can just write Clojure.</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 4, A useful idiomhttps://lambdaisland.com/blog/2019-12-04-advent-of-parens-4-a-useful-idiom2019-12-04T15:49:43+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Here’s a little Clojure idiom that never fails to bring me joy.</p><pre><code class="language-clojure">(into {} (map (juxt key val)) m)
</code></pre><p>The keen observer may notice this does exactly nothing. It’s a complicated way
of taking a map and returning that same map. However by changing parts of that
expression you can do so many different things, because what this really does is
pull a map into its individual bits: map entries, and keys, and values, and then
putting it back together, so by plugging into this you can do almost any
map-to-map transformation imaginable, as well as many other collection-to-collection transformations.</p><p>Let’s start with a simple <code>map-vals</code> as you might find in utility libraries like
Medley.</p><pre><code class="language-clojure">(defn map-vals
"Maps a function over the values of an associative collection."
[f m]
(into {} (map (juxt key (comp f val))) m))
(map-vals inc {:x 1 :y 2})
;; => {:x 2, :y 3}
</code></pre><p>Or how about <code>keywordize-keys</code>?</p><pre><code class="language-clojure">(defn keywordize-keys
[m]
(into {} (map (juxt (comp keyword key) val)) m))
(keywordize-keys {"x" 1 "y" 2})
;; => {:x 1, :y 2}
</code></pre><p>Or imagine you have a sequence of maps, and you want to turn it into a map using
the ids as keys, so you can quickly look them up by id.</p><pre><code class="language-clojure">(let [coll [{:id 456 :x "hello"}
{:id 641 :x "world"}
{:id 941 :x "wide"}]]
(into {} (map (juxt :id identity)) coll))
;; => {456 {:id 456, :x "hello"}
;; 641 {:id 641, :x "world"}
;; 941 {:id 941, :x "wide"}}
</code></pre><p>We’ve only scratched the surface here. You could take this in any direction you
like.</p><p>I called this pattern an idiom at the top of the post. In natural language
idioms are things like “I’ll show you the ropes” or “I’m about to hit the sack”.
Their meaning is not self-evident, but for native speakers they convey a lot of
meaning and connotation in a short phrase. Even though they seem to obscure
meaning, they actually require less parsing and processing because of the
familiarity of the phrase, making them a very useful tool for communication.</p><p>If however you want to be understood by a wide and diverse audience then you
probably want to go easy on the idioms. In programming you have to make a
similar trade-off.</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 3, `every-pred` and `some-fn`https://lambdaisland.com/blog/2019-12-03-advent-of-parens-3-some-fn-every-pred2019-12-03T15:07:22+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Ah <code>clojure.core</code>, it’s like an all you can eat hot-pot. Just when you think
you’ve scooped up all it has to offer, you discover another small but delicious
delicacy floating in the spicy broth.</p><p>In exactly the same way I recently became aware of two functions that until now
had only existed on the periphery of my awareness. I’ve since enjoyed using them
on several occasions, and keep finding uses for them.</p><p>These are <code>every-pred</code>, and <code>some-fn</code>. I’ll demonstrate them with an example.
Say you have a discussion forum. People can be admins or moderators, and besides
their username they can optionally set a nickname.</p><pre><code class="language-clojure">(def people [{:name "Elsie"
:admin? true
:mod? true}
{:name "Lin Shiwen"
:nickname "Rocky"
:mod? true}])
</code></pre><p>If you want to grab people’s nickname if they have one, or fall back to their
regular name otherwise, then this is pretty elegant with <code>some-fn</code>.</p><pre><code class="language-clojure">(map (some-fn :nickname :name) people)
;; => ("Elsie" "Rocky")
</code></pre><p>If on the other hand you want to find everyone who’s both an admin and a mod,
then you can do so easily with <code>every-pred</code>.</p><pre><code class="language-clojure">(filter (every-pred :admin? :mod?) people)
;; => ({:name "Elsie", :admin? true, :mod? true})
</code></pre><p>Clojure has a bunch of functions with <code>every</code> and <code>some</code> in their names, which
can be a bit confusing. In <code>some?</code> and <code>some-></code> it refers to a value being
something other than <code>nil</code>, whereas in <code>some</code> and <code>some-fn</code> it means “the first
one that applies”.</p><p>I think this overloading is part of the reason why these two have eluded me so
long. Only recently did I have the aha insight that while their names are very
different they are really each other’s complement. One combines functions with a
logical <code>or</code>, the other does so with a logical <code>and</code>.</p><pre><code class="language-clojure">(map #(or (:nickname %) (:name %)) people)
;; => ("Elsie" "Rocky")
(filter #(and (:admin? %) (:mod? %)) people)
;; => ({:name "Elsie", :admin? true, :mod? true})
</code></pre><p>So next time you find yourself writing an anonymous function like this, consider
reaching for <code>some-fn</code> or <code>every-pred</code> instead.</p><p>The only difference is that <code>every-pred</code> coerces its result to a boolean, making
it a little less general purpose. Say we had an <code>every-fn</code> instead, you could
combine it with <code>some</code> to get for instance the name of the first admin in the
list.</p><pre><code class="language-clojure">(some (every-fn :admin? :name) people)
</code></pre><p>The implementation of <code>every-fn</code> is left as an exercise to the reader. (hint:
it’s just like <code>every-pred</code>, but without the calls to <code>boolean</code>).</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 2, Piping hot network sockets with Netcathttps://lambdaisland.com/blog/2019-12-02-advent-of-parens-2-netcat2019-12-02T15:11:45+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>Part of what I want to do in this series is simply point at some of the useful
tools and libraries I discovered in the past year. I’ve adopted a few tools for
doing network stuff on the command line which I’ll show you in another post.
First though we’ll look at a classic: netcat!</p><p>I’ve been using netcat for years, it’s such a great tool. It simply sets up a
TCP connection and connects it to STDIN/STDOUT. Pretty straightforward. I’ve
been using it more and more though because of Clojure’s socket REPL.</p><p>You might have to poke around a bit to find out how to install it on your system
or distribution. Debian/Ubuntu packages it as <code>netcat-openbsd</code>, RHEL has it as
<code>nmap-ncat</code>. Mac users can <code>brew install netcat</code>. The executable is sometimes
called <code>ncat</code>, but usually you find it as just <code>nc</code>.</p><p>First start a socket REPL by setting the <code>clojure.server.repl</code> Java property.
(<code>-J</code> passes an option straight through to <code>java</code>, and <code>-D</code> is the java flag to
set a property).</p><pre><code class="language-clojure">clj -J-Dclojure.server.repl='{:port,5555,:accept,clojure.core.server/repl}'
</code></pre><p>(Note the “commas are whitespace” trick, to prevent the shell from splitting
this into multiple arguments. Not really necessary here, but it has served me
well for instance in Dockerfiles. Life is too short to mentally process shell
quoting rules.)</p><p>Now connect to it with <code>nc</code> and you got yourself a REPL!</p><pre><code class="language-clojure">nc localhost 5555
</code></pre><p>Add <code>rlwrap</code> for some readline-based command line editing:</p><pre><code class="language-clojure">rlwrap nc localhost 5555
</code></pre><p>I now by default install <code>rlwrap</code> and <code>nc</code> on servers that I administer, and
always start my Clojure processes with a socket REPL on, just in case I need to
log in and debug stuff. It’s also neat for sending Clojure commands to a process
from scripts, or from cron. In these cases you should add <code>-N</code> so it closes the
connection upon EOF.</p><pre><code>echo '(foo.bar/baz)' | nc -N localhost 5555
</code></pre><p>Other stuff you can do with netcat: wait until a socket is ready to accept
connections. Great in shell scripts if you first start a service, and then want
to use that service later on.</p><pre><code class="language-sh">while ! nc -z localhost 1234; do sleep 0.5; done
</code></pre><p>I’ve even used <code>nc</code> with a NULL modem (an ethernet cable with the send/receive
wires twisted, so you can connect two network interfaces directly), as a cheapo
way of transferring files. This also works when you’re on the same local
network.</p><pre><code># on the receiving end, -l for "listen"
# ifconfig eth0 10.0.0.1
nc -l 10.0.0.1 9999 > file.txt
# on the sending end
# ifconfig eth0 10.0.0.2
nc 10.0.0.1 9999 < file.txt
</code></pre><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Comment on ClojureVerse</a></strong></p></<>>Advent 2019 part 1, Clojure Vocab: to Reifyhttps://lambdaisland.com/blog/2019-12-01-advent-of-parens-1-clojure-vocab-reify2019-12-01T22:26:32+00:00<<>><p><em>This post is part of <a shape="rect" href="https://lambdaisland.com/blog/2019-11-25-advent-of-parens">Advent of Parens
2019</a>, my attempt to
publish one blog post a day during the 24 days of the advent.</em></p><p>An interesting aspect of the Clojure community, for better or for worse, is that
it forms a kind of linguistic bubble. We use certain words that aren’t
particularly common in daily speech, like “accretion”, or use innocuous little
words to refer to something very specific. Even a simple word like “simple” is
no longer that simple.</p><p>We can thank Rich Hickey for this. He seems to care a great deal about language,
and is very careful in picking the words he uses in his code, documentation, and
in his talks.</p><p>The word I want to look at today is “reify”. Pronunciation tip: it has three
syllables, re-i-fy.</p><p>To reify is to “make something a thing”, it comes from the Latin “res”, meaning
“thing”. When you reify you have a <em>reification</em> on your hands. It can mean to
make something abstract concrete. This is what Clojure’s <code>reify</code> macro does, it
takes a Java Interface, and turns it into a concrete object.</p><p>Take for instance the <code>reify</code> macro in Clojure. You pass it an interface name
and method definitions, and what you get back is a concrete object that
implements this interface.</p><pre><code class="language-clojure">(let [listener (reify java.awt.event.ActionListener
(actionPerformed [this action-event]
,,,))]
(.addActionListener button listener))
</code></pre><p>Reify can also mean to make something “first class”, i.e. to take something
implicit, and represent it in an explicit way so it can be referred to and
manipulated like other parts of the system.</p><p>Most languages have some version of global variables, but Clojure vars are a
little different because they are <em>reified</em>. You can refer to a var itself, pass
it as an argument or return it from a function.</p><pre><code class="language-clojure">(def xxx "foo" 123)
(defn set-doc! [v doc]
(alter-meta! v assoc :doc doc))
(set-doc! #'xxx "An important number")
(:doc (meta #'xxx))
</code></pre><p>In Datomic transactions are reified. They are represented as entities, you can
query them, refer to them, etc.</p><p>Reifying in this sense can be a powerful way to allow for a higher order of
programming. The interceptor queue in Pedestal is a reified execution stack,
allowing you to inspect and change that stack at runtime.</p><p>And of course when you decomplect you’ll often find that there’s a concept
waiting to be reified.</p><p><strong><a shape="rect" href="https://clojureverse.org/t/advent-of-parens-2019/5159">Post your comments on ClojureVerse</a></strong></p></<>>Advent of Parens 2019https://lambdaisland.com/blog/2019-11-25-advent-of-parens2019-12-24T13:46:34+00:00<<>><p>Ah, the advent, the four weeks leading up to Christmas. That period of glühwein
and office year-end parties.</p><p>The last couple of years I’ve taken part in the <a shape="rect" href="https://adventofcode.com/">Advent of Code</a>, a series of
programming puzzles posted daily. They’re generally fun to do and wrapped in a
nice narrative. They also as the days progress start taking up way too much of
my time, so this year I won’t be partaking in Advent of Code, instead I’m trying
something new.</p><p>From the first to the 24th of December I challenge myself to write a single
small blog post every day. If my friend Sarah Mirk can do a <a shape="rect" href="https://www.mirkwork.com/year-of-zines">daily zine for a
whole year</a>, surely I can muster a few daily paragraphs for four weeks.</p><p>I’ve been keeping some notes of things I want to mention. It’ll be a hodge podge
of topics, sharing tips and tricks about Clojure, Emacs, UNIX; pointing at cool
tools and libraries I came across in the past year, and perhaps some end-of-year
rumination and reflection, should the mood strike.</p><p>These posts will be a little different stylistically from the usual Lambda
Island material. I usually like to provide ample context and cover topics in
depth. This time I’ll be brief and get to the point. Another way to look at it
is that these will be the kind of extra things that you pick up when watching
Lambda Island videos, a handy Emacs binding, a shell command you hadn’t seen
before, or an elegant Clojure idiom.</p><p>If you feel inspired to do your own advent of blog posts then let me know, I’ll
be happy to add a link. I hope it will be adventageous ;)</p><h3 id="other-people-doing-advent-of-parens-2019">Other people doing Advent of Parens 2019:</h3><ul><li><a shape="rect" href="https://alexanderoloo.com/">Alexander Oloo</a></li><li><a shape="rect" href="https://porkostomus.gitlab.io/">Bobby Towers</a></li><li><a shape="rect" href="http://slipset.github.io/">Erik Assum</a> aka slipset</li><li>John Stevenson at <a shape="rect" href="https://practicalli.github.io/blog/">Practicalli</a></li><li>Bozhidar Batsov at <a shape="rect" href="https://metaredux.com/posts/2019/12/01/meta-advent-2019.html">Metaredux</a></li></ul></<>>Lambda Island Streaming Live this Thursday and Fridayhttps://lambdaisland.com/blog/2019-09-03-streaming-sessions-thursday-friday2019-09-03T14:35:18+00:00<<>><p>We are definitely back from holidays, and to demonstrate that we’re not just doing one but two live stream events!</p><h2 id="felipe-and-arne-pairing">Felipe and Arne pairing</h2><p><strong>Thursday 5 September, 13:00 to 15:00 UTC</strong></p><p>I will be pairing with Felipe Barros, working on a reimplementation of the <a shape="rect" href="http://activities.heartofclojure.eu/">activities app</a> that we used for Heart of Clojure. We only had a single session so far, and we’re starting from the ground up, with nothing but an empty directory, so this is a great way to see a Clojure app slowly unfold.</p><p>Unfortunately we didn’t record the first session, but we can briefly recap what we have so far. Last time we added Integrant and http-kit, next up is adding in reitit, and making some more decisions about our stack. There will also be plenty of Spacemacs involved!</p><h2 id="lambda-island-office-hours">Lambda Island Office Hours</h2><p><strong>Friday 6 September, 14:00 to 15:00 UTC</strong></p><p>This is a new things we’re trying out, people can send their questions to <a shape="rect" href="mailto:officehours@lambdaisland.com">officehours@lambdaisland.com</a>, and I’ll do my best to answer them.</p><p>We announced this in our mailing list several weeks ago, but so far no questions have come in. There’s still time though. I’ll start preparing on Friday, so you have until Thursday evening to send in questions. Preferably about Lambda Island episode content, or about Lambda Island open source projects, but I’ll also consider regular Clojure questions.</p><p>If nothing comes in I’ll demonstrate some advanced uses of Kaocha.</p><p>Both events will be streamed on our <a shape="rect" href="https://www.youtube.com/channel/UCGg8tJU0JxHS0xmKVA7t66g">YouTube channel</a>, I’ll update this post with links to the stream before we start.</p></<>>Fork This Conferencehttps://lambdaisland.com/blog/2019-08-09-fork-this-conference2019-08-09T12:03:10+00:00<<>><p>Last weekend Heart of Clojure took place in Leuven, Belgium. As one of the core
organizers it was extremely gratifying to see this event come to life. We
started with a vision of a particular type of event we wanted to create, and I
feel like we delivered on all fronts.</p><p>For an impression of what it was like you can check out <a shape="rect" href="https://twitter.com/malweene/status/1157931033477701632">Malwine’s comic summary</a>, or <a shape="rect" href="https://manuel-uberti.github.io/clojure/2019/08/04/clojure-heart/">Manuel’s blog post</a>.</p><p>It seems people had a good time, and a lot of people are already asking about
the next edition. However we don’t intend to make this a yearly recurring
conference. We might be back in two years, maybe with another Heart of Clojure,
maybe with something else. We need to think about that.</p><p>But I do hope the legacy of Heart of Clojure can live on. One of the things we
optimized for was socializing, making connections. We brought people from 31 (!)
countries together, and enabled countless new connections. This will without a
doubt have an impact on the Clojure community.</p><p>One of my goals with Heart of Clojure was also to provide people with new ideas.
We had a very wide range of talks, many of them not directly related to Clojure,
but all of them inspiring and sparking interesting conversation.</p><p>Finally I wanted to inspire other conference organizers. I have visited
countless technical conferences over the years, and have developed strong
opinions about what I think makes an event great. Some of these ideas are
already wide spread, some of them are well accepted in other communities, but
not yet prevalent in Clojure. In particular I was hoping to introduce some ideas
from Ruby conferences like Eurucamp and Isle of Ruby into the Clojure community.</p><p>So that’s what this post is about, a quick recap of the specific things we did,
that we would love to see others copy. Maybe the existing conferences will pick
up a thing or two, or maybe someone else will feel inspired to create a new
event. Seems there’s a market for a European summer conference, maybe you want
to fill that gap?</p><h2 id="a-two-phase-cfp">A Two Phase CFP</h2><p>This is already well known by now, but still worth mentioning. Our selection
process occured in two steps. First we had a group of reviewers rate talks
anonymously. They did not see the speaker’s name or other identifiable details,
they just had the title and the abstract. This way you get a largely unbiased
rating of the proposal itself.</p><p>Once all proposals are rated the program committee made a final selection. This
time they did look at who the speakers were, to assure we had a balanced and
diverse program, including first time speakers, and speakers from different
places geographically, with a strong European representation, since we wanted to
also showcase our local talent.</p><p>Finally also worth mentioning that after this final selection we paired all
speakers up with a mentor. Some people worked together with their mentor
extensively, others only checked in once or twice, but the offer was there, to
make sure people are supported in delivering the best talk they could.</p><p>This is not the final word on finding a good CFP process, I heard that for ClojuTRE Metosin is experimenting with rating talks on multiple criteria, and then making a shotgun spread so you get talks across the spectrum. Hopefully they will write about their experience, and maybe even open source the app they developed for this.</p><p>To run our CFP we forked the <a shape="rect" href="https://github.com/heartofclojure/cfp-app">CFP App developed for RubyConf</a>.</p><h2 id="long-breaks">Long breaks</h2><p>We had 16 sessions spread over two days. That’s not that many, for comparison,
the upcoming <a shape="rect" href="https://clojutre.org/2019/#program">ClojuTRE</a> has well over 20, which I’d say is fairly typical.</p><p>However we made the choice to keep the program fairly light. The sessions are what bring the people
together, they provide inspiration and a shared experience, but in my mind a
conference is really about meeting people, so we wanted to optimize for that.</p><p>We also wanted to prevent (to some extent) the mental exhaustion that easily
sets in after a day of listening to talks. So we had plenty of breaks, typically
half an hour each, with a one and a half hour lunch break the first day, and an
extended lunch+siesta break on the second day.</p><p>This siesta break concept was pioneered by Eurucamp, and it took a little bit of
convincing to get everyone on the team on board with it, but it’s so great, and
people loved it. Basically on Saturday from noon to four people could do
whatever. We offered lunch of course, and there were some activities to opt-in
to (more on that later), but really it was “space intentionally left blank”. And
so people grab a drink, sit on the grass, have long extended conversations. They
go back to their hotel to freshen up or take a nap, they head into town, visit a
museum, or pull out their laptops to share what they are working on.</p><p>The result: a much more interactive event, with deeper, more genuine
connections. Trust me, four o’clock is there before you know it.</p><h2 id="activities">Activities</h2><p>All these people travel to a country and city they’ve never been to, that has so
much to offer, only to sit in a hall and stare at a screen until they’re
exhausted. Isn’t that a shame? We wanted people to interact, to explore the
city, to do things in small groups, which are so much more amenable to getting
to know people.</p><p>So we organized Activities. We had guided city tours the day before, we did a
creative code jam during the siesta break, there was a screen printing company
on site where you could go to get our logo printed or create your own designs.</p><p>And other people could create activities as well. Juxt organized an impromptu
workshop about Crux, people went bouldering, we had competing beer and wine
tasting events, and over a dozen people learned from our sketchnoter Malwine how
to sketchnote.</p><p>On Friday evening one of our speakers and Leuven native Maarten Truyens
organized an “adventurous dinner”. Everyone who signed up got split into twelve
groups of six to eight people each, and we made twelve different restaurant
reservations, so people could get a dinner with a small group, and meet some of
the other attendees, sponsors, and speakers.</p><p>Compare this with what usually happens: everyone stands around at the end of the
day, clustering into ever bigger groups, and by the time you start moving you
realize you need to on the spot find a place that will fit 20 people without
prior notice.</p><p>To make this all possible we used the <a shape="rect" href="https://github.com/heartofclojure/activities">activities
app</a> first developed for Eurucamp, we just had to style it a bit, and promote it.</p><h2 id="lightning-talks">Lightning talks</h2><p>This has always been one of my favorite parts of Ruby conferences. To qualify as
a lightning talk a few requirements need to be met. The talks are exactly five
minutes each, after five minutes the audience starts clapping and its over, and
people sign up for them on the spot, by simply writing their name and talk title
down on a big sheet that’s hung up in the venue.</p><p>If you like to have everything under control this may seem scary. There’s no
telling what people will bring to the stage, but that’s what makes them so
great. You get really spontaneous, diverse, and entertaining talks most of the
time. And if it’s not, then in five minutes there’s already the next one.</p><p>This format allows people to show something they hacked together the day before,
or to respond to a talk that they saw earlier at the conference. It allows
people who feel a bit insecure to make a last minute call. It’s just a really
great format, and I’ve rarely seen it gone wrong. Just make sure everyone is
well aware of the code of conduct!</p><p>Also worth mentioning is that we did not end the day with lightning talks, which is fairly common. Instead we kept a closing keynote to close off the day and the event. This way you don’t end on the erratic, chaotic energy of the lightning talk session.</p><p>We also made sure this closing keynote wasn’t a deep technical talk. I’ve seen a few Clojure conferences do this, and I find it really hard to still engage with at the end of the day, my brain is just too fried. Instead we had a talk about community, focusing on the human aspects, which I thought was a great takeaway to end the day.</p><h2 id="start-the-second-day-a-little-later">Start the second day a little later</h2><p>Conferences bring far away friends together, and so what do they do after a day
of focusing on talks? They go and relax, get a bite, grab a drink, and before
you know it it’s two in the morning and you’re calculating how much sleep you
can still get and if you’ll skip the first talk or not.</p><p>This isn’t necessarily what we recommend people to do, we did mention in a few
places to be careful with Belgian beers, and to try and be fresh and bright in
the morning (or to at least keep the partying until the end of the second day),
but this is still the reality for a good part of the attendees.</p><p>So we start a little later the second day (and don’t start too soon in any
case). The first day we had doors at 9:00, and started the sessions at 9:30, on
Saturday doors opened at 9:45, and sessions started at 10:15. I’ve been to
conferences where the first speaker on the second day is talking to just a
handful of people, that’s not a fun place to be.</p><h2 id="conclusion">Conclusion</h2><p>These weren’t the only things that made Heart of Clojure great, some things are
harder to copy, and sometimes you have to get a little lucky. I think our
biggest strength was the awesome venue, and we were lucky to find it, but it’s also our biggest weakness. If
it had been a heat wave like the weak before, or if we had had serious
thunderstorms, we would have been in trouble. We were also very close on capacity, there is no room to grow for us at this location.</p><p>We were also really lucky with the
accessibility, being close to the train station and to Brussels airport, and to be in a city that’s compact and walkable. These are all circumstances that might be hard to replicate elsewhere.</p><p>And a final caveat: these things I listed here may not be the right thing for
your event. You need to be the judge of that. Different people like different things, and different organizers set different priorities. But hopefully one or two of these ideas
can trickle into some of the other great conferences we already have.</p><p>And if you do feel inspired to organize a new Clojure conference, once or
repeating, then get in touch. We would be really happy to help. Maybe next
summer?</p></<>>Advice to My Younger Selfhttps://lambdaisland.com/blog/2019-08-07-advice-to-younger-self2019-08-07T14:21:30+00:00<<>><p>When I was 16 I was visited by a man who said he had come from the future. He
had traveled twenty years back to 1999 to sit down with me and have a chat.</p><p>We talked for an hour or so, and in the end he gave me a few pieces of advice. I
have lived by these and they have served me well, and now I dispense this advice
to you.</p><h2 id="become-allergic-to-the-churn">Become allergic to The Churn</h2><p>First he warned me about the churn, and I have since learned to recognize it.</p><p>The Churn is losing a day debugging because a transitive dependency changed a
function signature. The Churn is spending a week just to get a project you wrote
a year ago to even run. The Churn is rewriting your front-end because a shiny
new thing came around.</p><p>Many people are blind to The Churn. They waste days, weeks, months of their
lives without blinking, or worse, they get a perverse sense of achievement
because the thing which was working fine works <em>again</em>.</p><p>The Churn is a resource sapping disease, and the first step to recovering from
The Churn is to recognize The Churn, the second step is realizing it doesn’t
have to be this way.</p><p>There are a plethora of techniques available to combat The Churn that have been
around for decades, but our industry is slave to a hype cycle driven by
popularity contests, with little regard for upgrade paths, backward
compatibility, and long term support.</p><p>Luckily many people do see The Churn for what it is, and once your doors of
perception are cleansed you will learn to recognize these people and the work
they do. You will discover vast ecosystems of languages and tools that will go
to great lengths not to break what wasn’t broken.</p><p>These are the olives of technology. Olives aren’t candy, and tasting them the
first time isn’t always pleasant, but soon you will develop a taste for them.
They have been here since ancient times, and they will remain for centuries to
come. They are <em>good</em> for you, solid, reliable, nutritious. Eat less candy and
more olives.</p><p><strong>Examples</strong>: Clojure, Common Lisp, HTML, Make</p><p><strong>Counter-examples</strong>: JavaScript, Ruby, React-Preact-Vue-Angular-…</p><h2 id="study-the-great-traditions">Study the Great Traditions</h2><p>He then went on to speak about The Great Traditions. These are technologies that
have been around for so long they form their own worlds. They are like great
philosophical traditions, with foundational documents, and generations of
commentaries and derived works.</p><p>From the outside looking in they are like alien civilizations, complete with
their own languages and cultures, perhaps unlike anything you’ve come across
before.</p><p>The combined effort and experience that has gone into creating each of these
traditions is measured in millenia. These people <em>know</em> things. They have <em>seen</em>
things. And when you scratch away the hipster veneer of your technology stack
you will find they are <em>everywhere</em>.</p><p>So how do you go about approaching something this vast? First off, don’t rush
it, time is on your side. Maybe you are Young and have More Important Things To
Do First. That’s fine. These things were here before you, and they will
patiently wait until you’re ready.</p><p>If possible find a mentor, someone to be your shifu and teach you these ancient
arts. Stay humble, and let your curiosity guide you.</p><p>Some of the things you learn will be eminently practical in your day to day
work, but beyond that you will develop an appreciation for technologies that
stand the test of time, and the design, the culture, the principles they are
born from.</p><p><strong>Examples</strong>: UNIX, LISP, The Web, Emacs, TeX</p><p>We spoke about many more things in the end, and maybe one day I’ll write about
those too. And then he left, mysteriously as he had come.</p><p>Now twenty years later, prepping my time machine, I know that person was me.</p></<>>ClojureScript logging with goog.loghttps://lambdaisland.com/blog/2019-06-10-goog-log2019-07-02T07:12:48+00:00<<>><p><em>This post explores <code>goog.log</code>, and builds an <a shape="rect" href="https://github.com/lambdaisland/glogi">idiomatic ClojureScript
wrapper</a>, with support for cljs-devtools,
cross-platform logging (by being API-compatible with Pedestal Log), and logging
in production.</em></p><p><em>This deep dive into GCL’s logging functionality was inspired by work done with <a shape="rect" href="https://nextjournal.com/">Nextjournal</a>, whose support greatly helped in putting this library together.</em></p><p>Clojure’s standard library isn’t as “batteries included” as, say, Python. This
is because Clojure and ClojureScript are <em>hosted languages</em>. They rely on a host
platform to provide the lower level runtime functionality, which also allows
them to tap into the host language’s standard library and ecosystem. That’s your
batteries right there.</p><p>On the JVM that means you get to use the thousands of classes from Java SE. In
ClojureScript you can access browser or Node.js APIs, or you can make use of the
many gems hidden in the Google Closure Library.</p><h2 id="the-google-closure-what">The Google Closure What?</h2><p>Ok this one is always a little awkward to explain. These three things are
different but related:</p><ol><li>The Clo<b>j</b>ureScript Compiler</li><li>The Google Clo<b>s</b>ure Compiler (GCC)</li><li>The Google Clo<b>s</b>ure Library (GCL)</li></ol><p>The ClojureScript Compiler takes ClojureScript sources and turns them into
JavaScript. Easy enough.</p><p>The Closure Compiler is a optimizing JavaScript-to-JavaScript compiler from
Google. It’s <em>really good</em> at what it does, but in order to do its job your
JavaScript has to be structured in a certain way, so that it’s amenable to
<a shape="rect" href="https://en.wikipedia.org/wiki/Static_program_analysis">static analysis</a>.</p><p>Coincidentally the ClojureScript compiler generates JavaScript in the format
that the Google Closure Compiler expects, and all ClojureScript tooling can
invoke the Google Closure Compiler for you. You just need to tell it which
optimization level you want (whitespace, simple, or advanced).</p><p>Finally the Google Closure Library is nothing but a big JavaScript library (or a
collection of libraries, if you will), again written to be easily optimizable by
the GCC. It’s the batteries! The nice thing is that ClojureScript already makes
use of the GCL, so you can count on it being available.</p><p>GCL was originally written when developing GMail, and it powers a lot of Google
products. Here’s how Google puts it:</p><blockquote><p>The Closure Library is a broad, well-tested, modular, and cross-browser
JavaScript library. You can pull just what you need from a large set of
reusable UI widgets and controls, and from lower-level utilities for DOM
manipulation, server communication, animation, data structures, unit testing,
rich-text editing, and more.</p><p>The Closure Library is server-agnostic, and is intended for use with the
Closure Compiler.</p></blockquote><h2 id="a-treasure-cave">A Treasure Cave</h2><p>There’s a lot of really usefull stuff in there, but unfortunately it’s not
terribly discoverable or well documented. The best resource you’ll find on-line
is the <a shape="rect" href="https://google.github.io/closure-library/api/">generated API
documentation</a>, which rarely
explains the why or how. There’s an <a shape="rect" href="http://shop.oreilly.com/product/0636920001416.do">O’Reilly book from
2010</a> that covers both the GCC
and the GCL. If you really want comprehensive documentation that is really your
only option at the moment.</p><p>Which is unfortunate because there really is a ton of gold in there, but at
least it’s open source so we can stumble through the sources trying to find the
things we were looking for. That’s what I recently did for <code>goog.log</code>, the
logging functionality provided by the GCL. In this blog post I’ll walk you
through the various things you can do with <code>goog.log</code>, and create an idiomatic
ClojureScript wrapper while I’m at it.</p><h2 id="getting-started">Getting started</h2><p>Giving your library a Finnish name is all the rage these days, so I’m going with
<a shape="rect" href="https://en.wikipedia.org/wiki/Gl%C3%B6gi">Glögi</a> (am I doing this right?).
There will be macros involved, which is always a bit mind-wrenching in
ClojureScript, but we’ll use the <a shape="rect" href="https://blog.fikesfarm.com/posts/2016-03-01-clojurescript-macro-sugar.html">macro sugar
trick</a>,
so that consumers of the library don’t have to care.</p><p>For the main API I will mimic <code>io.pedestal.log</code>, with <code>trace</code>, <code>info</code>, <code>debug</code>,
<code>warn</code>, and <code>error</code> macros, which take a sequence of key-value pairs.</p><p>Macros go in a <code>.clj</code> files, since macroexpansion happen during compilation,
which is executed by Clojure (the ClojureScript compiler is written in Clojure).</p><pre><code class="language-clojure">;; src/lambdaisland/glogi.clj
(ns lambdaisland.glogi)
(defmacro trace [& _])
(defmacro info [& _])
(defmacro debug [& _])
(defmacro warn [& _])
(defmacro error [& _])
</code></pre><p>ClojureScript functions go into <code>glogi.cljs</code>. This is the actual ClojureScript
library that we’ll require from other ClojureScript namespaces. By doing a
<code>:require-macros</code> with the same namespace name the macros we defined in
<code>glogi.clj</code> will become available as well.</p><pre><code class="language-clojure">;; src/lambdaisland/glogi.cljs
(ns lambdaisland.glogi
(:require-macros [lambdaisland.glogi]))
(defn logger [name] ,,,)
</code></pre><p>Now you can require this like any other namespaces, and access both functions
and macros.</p><pre><code class="language-clojure">;; src/lambdaisland/glogi/demo.cljs
(ns lambdaisland.glogi.demo
(:require [lambdaisland.glogi :as log]))
(log/warn :oh "no")
(log/logger "acme.widget.factory")
</code></pre><h2 id="our-first-logger">Our first logger</h2><p>Alright, let’s see if we can get this thing to log something. Load the
<code>goog.log</code> and <code>goog.debug.Logger</code> namespaces, create a <code>Logger</code> object, and
call the <code>warning</code> function, passing it the logger and a message.</p><pre><code class="language-clojure">(ns lambdaisland.glogi ;;.cljs
(:require [goog.log :as glog]
[goog.debug.Logger :as Logger])
(:require-macros [lambdaisland.glogi]))
(glog/warning (glog/getLogger "lambdaisland.glogi") "oh no!")
</code></pre><p>Now you may or may not have seen this log message pop up in your browser
console:</p><pre><code>[976.053s] [lambdaisland.glogi] oh no!
</code></pre><p>If you didn’t that’s probably because there are no “log handlers” installed yet.
Like a lot of logging systems <code>goog.log</code> separates log producers and consumers.
We may be producing log messages, but no one is consuming them.</p><p><code>goog.log</code> comes with several possible consumers out of the box, the most
straightforward one is the <code>goog.debug.Console</code> which logs, you guessed it, to
the browser console. If you’re using Figwheel (which of course I recommend),
then Figwheel will have enabled this console logger already, but otherwise you
need to do it yourself, by instantiating it, and setting <code>.setCapturing</code> to
<code>true</code>.</p><pre><code class="language-clojure">(ns lambdaisland.glogi
(:require [goog.log :as glog]
[goog.debug.Logger :as Logger]
[goog.debug.Logger.Level :as Level]
[goog.debug.Console :as Console])
(:import [goog.debug Console FancyWindow DivConsole])
(:require-macros [lambdaisland.glogi]))
(.setCapturing (Console.) true)
</code></pre><p>Alternatively you can use <code>Console/autoInstall</code>. This function will check for
the presence of the string <code>"Debug=true"</code> in the URL (mind the case), so you can
enable logging on demand.</p><pre><code class="language-clojure">(Console/autoInstall)
</code></pre><p>There are a few other options, you can log to a DOM element with <code>DivConsole</code>,
or to a separate browser window with <code>FancyWindow</code> (make sure to allow pop-ups).</p><p>With present day browser consoles these aren’t that useful perhaps, although the
<code>FancyWindow</code> one has some cool options to change the log level for each logger
individually.</p><pre><code class="language-clojure">(.setCapturing
(DivConsole.
(js/document.getElementById "app"))
true)
(.setEnabled (FancyWindow.) true)
</code></pre><p>I’ve wrapped all these options in helper functions, so a simple call to
<code>glogi/enable-console-logging!</code> should be all you need.</p><h2 id="logger-hierarchy">Logger hierarchy</h2><p>You may have noticed I gave the logger the same name as the namespace. This is a
good practice to follow, because loggers are hierarchical. They inherit their
log level from their parent logger, which is obtained by dropping the last
segment from the dotted name. So the parent of the <code>acme.widget.factory</code> logger
is the <code>acme.widget</code> logger.</p><pre><code class="language-clojure">(defn logger
"Get a logger by name, and optionally set its level."
([name]
(glog/getLogger name))
([name level]
(glog/getLogger name level)))
(def child1-logger (logger "acme.widget.factory"))
(def child2-logger (logger "acme.widget.sales"))
(def parent-logger (logger "acme.widget" Level/FINE))
(def other-logger (logger "acme.trinket" Level/WARNING))
(glog/info child1-logger "this gets logged")
(glog/info child2-logger "this too")
(glog/info other-logger "this doesn't")
;; [190607 16:50:02.49] [1175.643s] [acme.widget.factory] this gets logged
;; [190607 16:50:02.50] [1175.650s] [acme.widget.sales] this too
</code></pre><p>If a logger doesn’t have a log level set, and neither does any of its ancestors,
then the log level of the “root logger” is used. You can access the root logger
by using an empty string for the logger name.</p><pre><code class="language-clojure">(def root-logger (logger ""))
</code></pre><p>The log level is represented by a simple JavaScript object, which has a name and
a value</p><pre><code class="language-clojure">(.-name Level/FINEST)
;; => "FINEST"
(.-value Level/FINEST)
;; => 300
</code></pre><p>There’s not much reason to create your own <code>Level</code> objects, instead there are a
bunch of predefined constants you can use. For our own API it would be nice if
we could simply reference these with a keyword, so I’ll set up a mapping for
that.</p><p>Now the <code>log</code> and <code>set-level</code> functions are starting to feel a bit more like a
ClojureScript API.</p><pre><code class="language-clojure">(def level
{:off Level/OFF
:shout Level/SHOUT
:severe Level/SEVERE
:warning Level/WARNING
:info Level/INFO
:config Level/CONFIG
:fine Level/FINE
:finer Level/FINER
:finest Level/FINEST
:all Level/ALL
;; pedestal style
:trace Level/FINE
:debug Level/CONFIG
:warn Level/WARNING
:error Level/SEVERE})
(defn log
"Output a log message to the given logger, optionally with an exception to be
logged."
([name lvl message]
(log name lvl message nil))
([name lvl message exception]
(glog/log (logger name) (level lvl) message exception)))
(defn set-level
"Set the level (a keyword) of the given logger, identified by name."
[name lvl]
(assert (contains? level lvl))
(.setLevel (logger name) (level lvl)))
(set-level "acme.widgets" :debug)
(log "acme.widgets" :debug "There's a problem with the widgets.")
;; [190610 16:55:03.99] [674.672s] [acme.widgets] There's a problem with the widgets.
</code></pre><h2 id="macro-magic">Macro Magic</h2><p>Now we can finish our library by adding the macros which all call down to this
<code>log</code> function. As mentioned this API is largely inspired by <code>io.pedestal.log</code>.
The idea here is that log messages are just events, and so they should be
machine readable as well as human readable.</p><p>These logging macros all take a sequence of key value pairs, which get logged in
the form as a map.</p><pre><code class="language-clojure">;; glogi.clj
(defn- log-expr [form level keyvals]
(let [keyvals-map (apply array-map keyvals)
formatter (::formatter keyvals-map 'pr-str)]
`(log ~(::logger keyvals-map (str *ns*))
~level
(~formatter
~(-> keyvals-map
(dissoc ::logger)
(assoc :line (:line (meta form)))))
~(:exception keyvals-map))))
(defmacro trace [& keyvals]
(log-expr &form :trace keyvals))
(defmacro debug [& keyvals]
(log-expr &form :debug keyvals))
(defmacro info [& keyvals]
(log-expr &form :info keyvals))
(defmacro warn [& keyvals]
(log-expr &form :warn keyvals))
(defmacro error [& keyvals]
(log-expr &form :error keyvals))
</code></pre><p>If you’re not used to macros then this probably looks pretty dense, with the
backtick and all the tildes (in this context called “unquote”). The cool thing
is that by using macros we can extract the current namespace and line number,
using <code>*ns*</code> and <code>(:line (meta &form))</code> respectively. So now log messages
automatically use a logger based on the namespace. This way loggers respect
namespace hierarchy, and so you can enable or disable logging for a subtree of
namespaces by changing the log level of a single shared parent logger.</p><pre><code class="language-clojure">(ns lambdaisland.glogi.demo
(:require [lambdaisland.glogi :as glogi]))
(glogi/enable-console-logging!)
(glogi/warn :msg "oh no!")
;; [190610 17:24:50.61] [2461.288s] [lambdaisland.glogi.demo] {:msg "oh no!", :line 6}
</code></pre><p>I’ve incorporated a few other features borrowed from Pedestal.log. You can pass
an exception in with the <code>:exception</code> key, and it will be passed down to
<code>goog.log</code> with the optional exception argument, so that you get a proper stack
trace in the console.</p><p>You can pass in a custom formatter or use a different logger name as well with
<code>:lambdaisland.glogi/formatter</code> (defaults to <code>identity</code>) and
<code>:lambdaisland.glogi/logger</code> (defaults to the name of the current namespace).</p><p>The cool thing is that now in CLJC (cross-platform) code you can do this.</p><pre><code class="language-clojure">(ns my.project
(:require [#?(:clj io.pedestal.log
:cljs lambdaisland.glogi) :as log]))
(log/info :this "works everywhere!")
</code></pre><h2 id="registering-a-handler">Registering a handler</h2><p>At the top of the post I pointed out that <code>goog.log</code> separates log produces from
log consumers. These consumers are known in <code>goog.log</code> as handlers, they are
functions that receive log events to handle. So <code>Console</code> or <code>FancyWindow</code> will
all install a handler that takes care of actually displaying the log messages.</p><p>You can do cool things with this, you could for instance send them back to the
server, or capture all messages arriving while inside a certain function, to
analyze later.</p><p>Handler are scoped similar to levels, meaning each logger inherits its handlers
from its chain of ancestors. Typically you add handlers to the root logger, but
you could also add them to only a subset of loggers, like all the ones that
start with the name of your project.</p><pre><code class="language-clojure">(defn add-handler
"Add a log handler to the given logger, or to the root logger if no logger is
specified. The handler is a function which receives a map as its argument."
([handler-fn]
(add-handler "" handler-fn))
([name handler-fn]
(.addHandler (logger name)
(fn [^LogRecord record]
(handler-fn {:sequenceNumber (.-sequenceNumber_ record)
:time (.-time_ record)
:level (keyword (str/lower-case (.-name (.-level_ record))))
:message (.-msg_ record)
:logger-name (.-loggerName_ record)
:exception (.-exception_ record)})))))
</code></pre><h2 id="using-with-cljs-devtools">Using with cljs-devtools</h2><p><em>Kudos to Martin Kavalar of <a shape="rect" href="https://nextjournal.com">Nextjournal</a> for coming up with the <code>LogBuffer</code> trick.</em></p><p>People using <a shape="rect" href="https://github.com/binaryage/cljs-devtools/">devtools</a> might have
been a bit underwhelmed by the result, because <code>goog.debug.Logger</code> will
automatically stringify the message argument, so when you try to log data
structures they are simply logged as strings, meaning you lose the nice
formatting and interactive navigation that cljs-devtools provides.</p><p>To work around this we need to step away from <code>goog.debug.Console</code>, since that
one uses its own string formatter under the hood, and we need to convince
<code>goog.debug.Logger</code> to not call <code>js/String</code> on its message argument.</p><p>Digging around the sources you’ll see that if you enable the
<code>goog.debug.LogBuffer</code>, then the message argument is used as-is. To turn the log
buffer on, you need to set its capacity. We’ll just set it to 2 (the lowest
version that works). This will cause the <code>LogRecord</code> instances to be reused,
which is fine, since we convert them immediately in a ClojureScript data
structure.</p><pre><code class="language-clojure">(when-not (LogBuffer/isBufferingEnabled)
(set! LogBuffer/CAPACITY 2))
</code></pre><p>Now you can use <code>glogi/add-handler</code> to register a function which prints the
message to the console.</p><p>Have a look at
<a shape="rect" href="https://github.com/lambdaisland/glogi/blob/master/src/lambdaisland/glogi/console.cljs">lambdaisland.glogi.console</a>,
as there are a few more things to get right.</p><pre><code class="language-clojure">(require 'lambdaisland.glogi.console)
(console/install!)
</code></pre><p>This will log the raw values if devtools is enabled, and fall back to <code>pr-str</code>
if it’s not, for instance when your browser doesn’t support custom formatters.</p><h2 id="logging-in-production">Logging in production</h2><p>This is all fine and well, but chances are that once you try this in production
your logs are gone, which may or may not be what you want.</p><p>There’s a constant used throughout the GCL code base called <code>goog.DEBUG</code>. For
development builds this tends to be true, for production builds its false. Many
pieces of debugging code are guarded with a <code>if (goog.DEBUG) {}</code>. This doesn’t
just skip these parts at runtime, but it also causes the GCC to completely
remove these pieces of code, at least when <code>goog.DEBUG</code> is false and you are
using <code>:advanced</code> optimizations.</p><p>For logging there’s an extra step involved, <code>goog.log</code> has its own constant
called <code>goog.debug.LOGGING_ENABLED</code>. This will be set to the value of
<code>goog.DEBUG</code> unless it has already been set to something else. The same thing
applies here: almost all logging functionality is stripped out of <code>:advanced</code>
builds when <code>LOGGING_ENABLED</code> is false, so there’s also no chance of turning it
back on again at runtime.</p><p>It’s handy though the <code>LOGGING_ENABLED</code> is a separate constant, since that means
you can selectively keep logggin on, while removing other debugging code from
the build.</p><p>To do this use the ClojureScript compiler option <code>:closure-defines</code>.</p><pre><code class="language-clojure">:closure-defines {goog.debug.LOGGING_ENABLED true}
</code></pre><p>If you’re using Figwheel in development then you’ll also have to make sure that
in production builds you’re calling <code>(glogi/enable-console-logging!)</code>, since
this time Figwheel is no longer there to turn your logging on for you.</p><h2 id="closing-thoughts--a-challenge-for-you">Closing thoughts & a challenge for you</h2><p>Thanks to a request from a client I ended up diving deep into <code>goog.log</code>, and I
was happily surprised by what I found. The documentation is a bit lacking, but
the sources are clear, so after some browsing around I got a good picture of how
this library is intended to be used.</p><p>At this point I have a challenge for you: browse through the long list of
<code>goog.*</code> namespaces, pick one that piques your interest, and write your own blog
post about it. I will happily link to it from here.</p></<>>The Art of Tree Shaping with Clojure Zippershttps://lambdaisland.com/blog/2018-11-26-art-tree-shaping-clojure-zip2018-11-26T15:33:21+00:00<<>><p>This is a talk I did for the “Den of Clojure” meetup in Denver, Colorado. Enjoy!</p><p>Captions (subtitles) are available, and you can find the transcript below, as well as slides <a shape="rect" href="http://arnebrasseur.net/talks/2018-clojure-zip-denver/#">over here</a>.</p><p>For comments and discussion please refer to this post on <a shape="rect" href="https://www.reddit.com/r/Clojure/comments/a0k26p/the_art_of_tree_shaping_with_clojure_zippers/">r/Clojure</a>.</p><iframe frameborder="0" scrolling="auto" width="1120" height="730" src="https://www.youtube.com/embed/5Nm56YvTKZY" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe><h2 id="transcript">Transcript</h2><p><span style="color: #448822">Arne:</span></p><p>My name is Arne. I’m from Europe but I flew in because I met Dan in Berlin recently and he told me about this awesome meetup, so I was like, yeah, I gotta check that out. I’ve been here just long enough to get over the light headedness. Yeah, I do Clojure consulting and training. I also make Screencasts about Clojure. I don’t know if anyone’s seen Lambda Island, lambdaisland.com. You can find videos about Clojure.</p><p><span style="color: #884422">Dan:</span></p><p>I heard there’s even an article there that maybe someone around here wrote? Yeah?</p><p><span style="color: #448822">Arne:</span></p><p>That is true. Yeah. Den of Clojure speaker, Joanne Cheng, wrote an article about D3 and ClojureScript, which is really neat.</p><p><span style="color: #884422">Dan:</span></p><p>Awesome.</p><p><span style="color: #448822">Arne:</span></p><p>Yeah.</p><p><span style="color: #884422">Dan:</span></p><p>Cool. Well, thank you for joining us all the way from Berlin.</p><p><span style="color: #448822">Arne:</span></p><p>Thank you very much for having me.</p><p><span style="color: #884422">Dan:</span></p><p>Yeah. Pretty cool to come down here. You’re gonna talk to us about zippers.</p><p><span style="color: #448822">Arne:</span></p><p>Zippers, yes. Functional zippers.</p><p><span style="color: #884422">Dan:</span></p><p>Functional zippers. It’s a fun term, isn’t it?</p><p><span style="color: #448822">Arne:</span></p><p>Who’s used zippers before?</p><p><span style="color: #884422">Dan:</span></p><p>Take it away.</p><p><span style="color: #448822">Arne:</span></p><p>A couple of people, okay. All right. I’ve got a lot of slides so I’m gonna keep the pace. I’m gonna put on a timer so I’ll have a rough idea. If I really go over time, I’ll cut it short, but we’ll see how it goes.</p><p>So, yeah. Welcome to my talk, The Art of Tree Shaping with Zippers. So you already know who I am now, I’m Arne Brasseur, I’m Plexus on the Internet. I’m from Belgium, originally, a tiny country in Europe between France, Germany and the Netherlands. Beer, chocolate, waffles, fries, mussels, all the good stuff. But I’m currently living in Berlin, in Germany.</p><p>I do, like I said, Clojure consulting and training for a living, as well as this thing, Lambda Island. There’s tons of cool Screencasts on there, been going for about two and a half years, there’s 40 something episodes there. It ranges from core stuff like transducers, or small stuff like the some function, but also Datomic, reagent, re-frame, tooling, ClojureScript compiler, Integrant, you name it, there’s 40 something episodes on there now. Some of them are free, some of them you can buy, and then you can also just sign up for an all-access pass. So, yeah. That’s me.</p><p>The Agenda for today is, first, we’re just gonna look at Clojure zip from a user’s perspective. Just walk through some examples, what the API feels like, what you can do with it. Then, we’ll talk a bit more about, okay, what exactly is this whole functional zipper idea, where does it come from and why does it matter, why is it relevant? Then, we’ll look at how Clojure implements zippers. Then, finally, very briefly, I’ll touch on a couple of libraries that build on zippers, stuff that you can go from there.</p><p>The clojure.zip API. Before I can really show you examples, you need some kind of mental model of what we’re talking about when we say “zipper.” So, a zipper, also called a “loc,” or “location,” is like a two element tuple where the first one is some kind of tree-shaped data structure. It can be all kinds of things, but anything that has nodes with children. Then the second one is kind of a pointer to a specific node, like a way of focusing on a specific node. Or you can also think of it as a path in the tree to say, “Okay, now we’re dealing with this particular one.” So, as a metaphor, you can think of it as having a map with a dot saying “you are here.” The map plus the dot, that’s your zipper. With the difference that when you have the zipper you can move that dot around and make changes to your map, but we’ll get to that in a second.</p><p>The zipper API allows you to navigate in your tree. You’re looking at a specific node, and then you can say you wanna go up, down, left, right, and then when you end up at a certain location you can start making changes. When I say “making changes,” it’s Clojure, so it’s immutable, like Clojure’s own data structures, so it really just means that you get a new copy or a new version, an updated version of your zipper. Same with the navigation, it’s also in a functional way. I mean, if you’re used to how Clojure works, this should be quite natural. I’m just pointing out that when I say “replaced,” or “removed,” it’s functional still.</p><p>clojure.zip comes with Clojure itself. You don’t need to install anything extra. If you have Clojure, you have clojure.zip. You just need to require it. It’s quite typical to require it and call it “z.” Then, there’s a couple of different functions to take a data structure and turn it into a zipper to wrap it into that two element tuple.</p><p>For instance, if you have nested vectors, you can think of nested vectors as being a tree structure. So, if you say then <code>vector-zip</code> and you pass it a vector, you get the zipper or the loc over that vector back. So, yeah, I’m using all these terms interchangeably. We’ll look more at where the term “loc” comes from when we talk about the origin of the zipper idea.</p><p>Like I said, a nested vector can be considered a tree, a nested sequence can be considered a tree, XML documents are quite naturally a tree. So, you have an element, an element has children, and then those elements can have their children. But anything that has nodes and child nodes can be a zipper. We’ll look a bit more about how that works when we look at the Clojure implementation.</p><p>I’ll walk you through a very simple example. Here, I have a vector and I’ve turned that into a zipper. On the right-hand side, you see a visualization of what we’re looking at now. We’re looking at the vector as a tree and we’re pointing at the root node. That’s why the top node is colored blue. That blue is your dot saying you are here.</p><p>Then, you can go down, now we’re just looking at that first subvector. You can go right. You can go down again. You can go to the right-most sibling on the current level. Then, when you end up at a certain point, you can do these operations. For instance, you can append an extra child node to the current node. So now, I have appended an extra X and you see that now this node, instead of just four, five, now it has four, five X.</p><p>Then, you can go up again. Now at some point … So, all of the time … You see I’m threading this, the result of each function call is a new zipper. Now at some point you want a plain value again. You want the value of the node you’re currently looking at. That’s what you use the node function for. Given the node that you’re currently looking at that gives you the value of that node.</p><p>It’s also very common that you just wanna go all the way back to the root, all the way back to the top. You say up, up, up until you can’t go up any further, and then get the value of the complete structure. The combination of that is encapsulated in the root function.</p><p>Yes?</p><p><span style="color: #442288">Participant:</span>
Is it required that only the leaf nodes have values?</p><p><span style="color: #448822">Arne:</span>
Well, the only requirements that the zipper API has is that for whatever structure you give it, you can define what is a branch node and what is a leaf node, where a branch node is anything that can have children. In this case, when you’re using vector zip, it’s gonna consider anything that isn’t a vector as a leaf. But you can define that however you want, but you do need sort of a dichotomy of it’s either a branch or it’s a leaf, it can’t be both. Does that answer your question?</p><p><span style="color: #442288">Participant:</span>
[This is a bit hard to understand in the recording, the participant is asking about whether branch nodes can have values, i.e. contain extra information besides the list of children. In the vector example this isn’t clear, because a vector’s value is fully determined by its children.]</p><p><span style="color: #448822">Arne:</span>
Yeah. You can have extra values on that node. Say, an XML document, an element has attributes, absolutely, yeah. You don’t see that in the vector example because a vector doesn’t contain much more information besides its children, but, yes, a node can have extra information besides its children. Absolutely, yeah.</p><p>We’ll get to that now. Here’s a second example, I came up with some hypothetical XML. This is the representation of a shopping cart, which has line items and a customer. In line items, there’s two product line items, and a discount. Clojure has a set of XML parsing functions built-in. So, I’m using <code>clojure.java.io</code> to read that file and then Clojure XML to parse it from XML to a Clojure data structure.</p><p>Then, the result is something like this. This is how <code>clojure.xml</code> represents XML documents, so it uses Clojure maps and vectors where each XML element turns into a map which has a tag attributes and content, which are its children.</p><p>When you get something like this, then you can pass that to the zipper <code>xml-zip</code> function and you get a zipper over that XML document. This is now, yeah, it’s getting a little small here, the visualization. That top one, for instance, that’s cart and then it’s two children have line items, customers. So this is a representation of that XML that we just saw.</p><p>I can do the exact same thing. I can go down, I can go down, right, right, left-most, down. Now I can replace the node. So before it was a string node that said “chocolate” and now I’ll replace it with “luxury chocolates.” Now I go up and then I use the edit function. The edit function is like update or swap or any of these functions that you take a function that you apply to a value to get a new value. In this case, I’m gonna take the value in this node and <code>assoc-in</code> the attributes map a new price, because now it’s “luxury chocolates” so the price has gone up. So, before it says <code>:price 2.5</code>, and now it says <code>:price 2.9</code>. Then I’m done and I go back to the root and I get the root value.</p><p>Now you don’t write actually code like this where you just say, down, down, left, right, replace, … . You’re gonna wrap this in a function and actually check where you are and what you’re doing. But conceptually, this is what you typically do. You walk around your tree, you make changes, and then in the end you go back to the root and get the root value. That’s how zippers work.</p><p>So, you’ve seen most of the API already. I’ll just quickly run through the full API, it’s not that big. There’s a couple different ways to create zippers. We’ve seen a few of those. There’s different functions to navigate. We’ve seen all of these. There’s a bunch of functions to update. We’ve only seen a few of these, but it’s basically insert a sibling on the left, insert a sibling on the right, insert a child on the left, insert a child on the right, “edit” to apply a function to the current node value, remove a node, and replace a node with a new node.</p><p>Then, just a helper functions, which takes an existing node and a new sequence of children, and creates a new node which has all the attributes of the old node but with a new set of children. We’ll get to that again later. You’ll come across them and I’ll make a note later when we talk about the implementation, why that is interesting.</p><p>Then, finally, you can look around. Right? You end up at a certain point and you can say, what are my left siblings? What are my right siblings? What are the children of this node? You can ask for the path. This gives you a sequence of node values from the root up to your current node that you’re currently looking at. Then, you can just ask, is this a branch node or not?</p><p>Then, finally, there’s these tree functions which come together, which give you a straightforward way to do a complete walk of all the nodes in your tree. This has a depth first walk, so you start … Yeah, I’m using <code>iterate</code>. If you’re not familiar with <code>iterate</code>, it says “take that zipper and then apply the next function to it over and over and over again”, and it returns a lazy sequence of all the results.</p><p>Here I have applied it zero times, so I’m still at the root. Then, one time, and you see it starts going down to the next one, to the next one. It’s depth first, so it goes to the bottom, and then the next one, it goes left and up, left and up, until you’ve seen every single node in the tree. Then, if you go next one more time, it goes back to the root and adds a special marker to that zipper that says “this is the end point.” So now, you can use that end predicate to say that, “Okay, am I done walking? Am I done checking every single node?”</p><p>So now you know the complete zipper API in my talk here, but I still have a lot of interesting things to talk about. First of all, I’m gonna give you a couple of caveats. Now you’re gonna probably forget these, but hopefully when you actually get into the thick of it and you start banging your head against the wall, because something isn’t working the way you think it should be, then maybe at some point you will be like, “Oh, actually this is what Arne mentioned, this is the one thing that I should watch out for.”</p><p>These are
- there’s no boundary checks, so don’t fall off your zipper
- There’s no way back from end
- zippers are not cursor
I’ll go over these one by one.</p><p>No boundary checks simply means that if you go down, down, down until you get to a leaf node and you go down once more, you get nil. Up to that point all is well, but then you do one more thing and you get a NullPointerException and you need to start looking like, okay, where did that nil come from? That is how the API is designed, you know, also left, left, you go left one more time, you get a nil. It means that you can do stuff like this, <code>(if-let [left (z/left ...)])</code> if I can still go left, and do something there, so it’s a fine API, but it is something to, you know, you do wanna have some checks and assertions in there to make sure that you don’t need to start chasing your nils after the fact.</p><p>The second thing I wanted to point out. So, I said, okay, when you do this complete depth first walk of your tree, you get this special end value. That end value is in a way a zipper, but not everything works on it the way that it works on regular zippers. For instance, if you try to go to the previous node, again it’s just gonna give you nil.</p><p>The main thing here, main advice here is, this end, either just treat it as a sentinel, just as a way to say, “Okay, now you’re done,” or to get the root value back. But that’s kind of it. Don’t try to do too much with that.</p><p>Then, finally, zippers are not cursors. What I mean by that is, a cursor in text points in between two characters and that means that it can be at the beginning of a line or at the end of the line or on an empty line. Whereas the zipper always points at a concrete node, which means that, for instance, you can’t have a zipper inside an empty collection and then say, “Okay, now I wanna insert something here.” Again, that’s how it works, there’s nothing wrong with that. But depending on the kind of problem you’re solving, sometimes the cursor model is much more intuitive and so you start thinking of your zipper as a cursor because the metaphor kind of matches until it doesn’t.</p><p>Here’s a little example Say, naively, I wanna take these nested vector, go two levels down. So, I’m inside the nested vector now. I wanna remove that one, remove that two and, in place of it, insert a three. What I’m hoping for is a nested vector with a three inside. Let’s see what happens. I go down, I go down, I remove this, and now I’m looking at that middle node. That might already be a little bit surprising. This is 100% by design. Again, there’s nothing wrong with that, but you do need to know how this stuff works. We’ll remove such that it removes the node at your current location, returning the loc that would have proceeded it in a depth first walk. That means, over here, if you would have called that previous, you would have ended up in that middle node, and that’s why it gives you that middle node.</p><p>If we continue our example, you remove it, you try to do an insert, and now you’re trying to insert a sibling of the root node. You can’t have a sibling if you don’t have a parent, and so it’s going to say “Exception: Insert at top.” That’s something to watch out for.</p><p>This is even more obvious when you have an example like this. I’m looking at that one, two, and then I delete it, and now I jump to that Y. Sometimes it’s a little counterintuitive.</p><p>All right, any questions so far? Yes.</p><p><span style="color: #442288">Participant:</span>
Are you familiar with xpath and would zippers be useful compared to those ways of navigating the data structures?</p><p><span style="color: #448822">Arne:</span>
I mean, Xpath, I’ve used Xpath a long time ago. I mean, conceptually, it’s too different from CSS selector, right? I think having these kind of selector APIs are very useful. I think they both have their uses. For instance, say, I’m doing stuff over XML, I will very often use Enlive, which gives me that selector API. It just depends what you’re doing. There are cases where the zipper model works where you really need more context or where it’s more loosely defined. Not everything can easily be captured in a path or in this kind of expression. I think for those cases, this is a really great tool in your toolbox.</p><p><span style="color: #442288">Participant:</span>
You mentioned that the pointer has to point to some node. Does that mean when you create a zipper, you have to give it a root node, or do you create the tree first? Or how do you create a zipper? How do you do it?</p><p><span style="color: #448822">Arne:</span>
Yeah. You need some kind of data structure for it to work on. You can give it an empty branch node of whatever kind of data structure you’re working with, so like an empty vector or an empty XML element. But you do need something like, say, you’re dealing with XML, the minimum you would have to give it is a single tag with nothing in it and then you can add stuff from there.</p><p><span style="color: #442288">Participant:</span>
Okay. Create data structure first and then you create a zipper from that.</p><p><span style="color: #448822">Arne:</span>
Yeah. Typically, you’ll have data structure and you use a zipper to do stuff with it, yeah. I mean, there’ll be more time for questions at the end. I just wanna make sure that this first part is more or less clear.</p><p>Okay. I’ve given you a rough mental model of the zipper, the two element tuple. Now it’s time to refine what a zipper really is. If you look at the doc string for Clojure zip, it’s a very Hickey doc string in that it makes absolute sense if you know exactly what he’s talking about. It says: “Functional hierarchical zipper, with navigation, editing, and enumeration. See Huet.”</p><p>Huet is a French guy, Gerard Huet, who published this article in 1997: “Functional Pearl, the Zipper”, where he says, “Almost every programmer has faced the problem of representing a tree together with a subtree that is the focus of attention, where that focus may move left, right, up, or down the tree. The Zipper is Huet’s nifty name for a nifty data structure, which fulfills this need. I wish I had known of it when I faced this task because the solution I came up with was not quite so efficient or elegant as the Zipper.” It’s a pretty cool article. It uses OCaml, that’s probably the biggest hurdle to reading it. But I remember reading it a couple of years ago, it’s a fun read. I recommend it.</p><p>Huet defines two types of data structures. The first one is the path. He says, “A path is like a zipper,” and in this case, he’s talking about an actual zipper, like off a hoodie or a jacket. “A path is like a zipper, allowing one to rip the tree structure down to a certain location. It contains its list L of left siblings, its father path P, and its list of R of right siblings. It’s a recursive definition, right? Because it contains a sequence of nodes, then a pointer to the previous parent, its own parents, and then a sequence of other nodes, the left and the right. There’s also a special value top, which means that we don’t have a parent path because we’re looking at the top of the structure, so that’s a special marker.</p><p>Then, “A location in the tree addresses a subtree, together with its path. This two element tuple, which is the current node with the path, which is the path we just defined. You just see where that terminology loc or location comes from. That’s from the original article.</p><p>Here I have a different visualization. In these visualization, there’s two types of nodes: there’s the diamond-shaped ones, which are the zippers or the paths; and the round ones are regular nodes. When I have my zipper over my XML document, it’s looking at the root node, so that’s the cart, and it has that marker that says, “okay, we’re at the top.”</p><p>Now, when I go down, now the thing I’m holding is the zipper, which is looking at line items with a parent, which is the previous zipper we had, and then left and right siblings. Then, I can go down again and now I’m looking at that product inside the line items, inside the cart. What you already see here is that the tree has been turned upside down, right? Before, we had the cart at the top and now we have the cart at the bottom. The thing that we’re holding onto is the place where we’re focusing on, the node where we ended up at.</p><p>The reason that that’s cool is that, say, I go right from here, so that means that I’m gonna shuffle left, right siblings and current node, those three pieces of information are gonna change, but the parent path and anything down from there remains unchanged. So I can start doing these local operations and not care about anything that’s above me in the tree. I hope that makes sense, these spatial metaphors.</p><p>We’ll look at Clojure’s implementation later and maybe then it’ll make more sense. So, yeah, it’s a generic way of prying a nested data structure open so that you can locally make changes and then afterwards you put it back together again.</p><p>I go down again. Now I’m gonna do the same stuff that I did before. I have this chocolate node and I’m going to replace it with the string luxury chocolate. Now the zipper has turned red, which means that it has a flag set on it, which says that it has changed. Because as we said before the parent and chain of parent paths doesn’t change when you do an operation like this, which means that that product node still has the old value.</p><p>Only when we start going up again does clojure.zip rebuild those nodes. So now product has changed because now it contains a new string. Now line item has changed because it contains a new product. Finally, we’re back at the cart. This is maybe the most confusing thing about the whole thing, but again it will probably become more clear when we get to the implementation. Does it make sense? Shall I walk through it again? Okay. I’ll keep going.</p><p>Here we are at the Clojure implementation. These vector-zip, xml-zip, seq-zip, in the end they all call the zipper function. This is the main way to create a zipper, to create a loc, and it takes four pieces of information. The first three are functions and the last one is your actual data structure, the root of your tree. Those first three functions are a way to tell Clojure how your data structure works, what kind of data structure you’re dealing with. You need to be able to answer two questions and do one operation. You need to be able to have a node an answer, is it the branch node or not? Can it have children or not? When you have a node you need to be able to say, “Give me a sequence of all the children of this node.” Then, finally, you need to be able to do that make node thing where you have a node and a new set of children and you get a new node which has all attributes of the old nodes but with the new children.</p><p>If you look at the implementation of zipper, this is it. You see that two element vector in there. Right? In returns a vector with that root and with the nil, which is what Clojure uses as its marker to say we’re at the top. Right? The path is nil, there’s no parent path because were at the top.</p><p>The other thing to notice here is that it then attaches a bunch of metadata to that vector. So these functions that you passed in there got stored on that two-element vector as metadata. This is pretty neat. It’s a neat trick and I haven’t seen it in many other places. So, yeah.</p><p>Now, if you look at vector-zip, vector is just calls zipper, in the case of a vector-zip, a root is a branch node if it’s a vector, otherwise we’ll consider it a child node. Seq just gives the stuff inside the of vector. Then, the make node function, mostly just calls back on the children. In this case, it also just make sure it preserves metadata.</p><p>So far it’s fairly straightforward. If you look at the implementation of the actual clojure.zip/branch function, clojure.zip/children, clojure.zip/make-node, they look up those specific functions on the metadata and call those. This is a pretty neat way of doing polymorphism. Clojure has protocols, Clojure has multimethods, which are all different ways of dispatching to different implementations based on the specific value that you’re looking at. But this is yet another way of doing it.</p><p>Vector-zip, on that zipper you’ll see that all it really does is just wrap it in a vector and then add a nil. If you look at the metadata, you’ll see those functions there. So, it’s kind of nothing under the covers. It’s fairly straightforward once you know what’s going on.</p><p><span style="color: #442288">Participant:</span>
I just wanna butt in for a second and say, “Wow, that’s really cool.” Yeah.</p><p><span style="color: #448822">Arne:</span></p><p>So, even if you never used zippers, maybe you end up using this trick. So now, we can actually start looking at the values return from these function calls. Now I’m going to walk through that first example again with the vector-zip. After turning it into a zipper, it’s just this two-element vector. And then you go down and so now you see that the node we’re looking is the node <code>[1 2]</code>, and the parent path, or the path at the moment has this left siblings, right siblings and a parent path, which is currently nil, which is, you know, because the parent of the node we’re looking at is the root.</p><p>Then, the Clojure also stores a sequence of parent nodes, which would use this to reconstitute, to rebuild the values as you go back up again. You can already see if I go left or right here, that left and right and the current node changes, but everything else stays the same. Now I go down one more level and you start seeing this linked structure. We’re looking at the tree now. It has left and right siblings and it has a parent path, which is the one we just had. Right?</p><p>Here we have left one, two, P nodes, P path nil, R nil, that’s inside there now, this is the parent path of the new path. Again, you know, we can go left and right and all it needs to do is change those siblings. Everything else stays the same way.</p><p>Now, append a child, it’s got an extra marker at the bottom. It’s changed now. But you see that X is now in the node that we’re looking at. But if you look at the parent node, that X isn’t there. Not yet, right? Clojure has been able to make this local change. That’s one of the benefits, the efficiency benefits of zippers that you can go deep into your structure, make a bunch of changes, and then only pay the price of rebuilding your path up to the root when you zip back up. We’ll do that now. Once you start going up, you can see that it starts building up those values again.</p><p>I’m actually doing better on time. Then I guess I really flew through this, slide 100. Okay. Two more libraries I wanna at least mention. One is fastzip. Fastzip is really just what it says on the tin, it’s faster zipper. The Clojure implementation with the vectors and the metadata, it’s super cool, it’s also nice that you can just inspect it, that it’s just the vector and you can look at those values. I do typically developed with Clojure.zip, but then when you wanna ship this in production and you wanna get those extra cycles out, you just swap out clojure.zip for fastzip. It’s absolutely the same API but it uses deftype under the hood so it really gives you this high-speed like native dispatch.</p><p>The only thing to watch out for is that you don’t rely on implementation details. Now you all know that a clojure.zip a two element vector, but if you start relying and, like for instance, destructure that vector, yeah, that’s not gonna work with fastzip. But as long as you stick to just the functions in Clojure.zip, it should work 100% the same Clojure. Clojure and ClojureScript, by the way, all of these are cross-platform.</p><p>Then, the final thing I wanna mention is rewrite-clj because this is one of the libraries that built on top zippers. In this case, in particular, it gives you a zipper over a Clojure syntax tree. So what rewrite-clj does is it parses Clojure source code, turns it into a syntax tree that preserves formatting and white space. You can process that, do stuff with it, make modifications without messing up people’s formatting. So, a lot of tools use this, you might’ve come across clj-format, zprint, lein-ancient, refactor-nrepl, mutant mutation testing, all of these use rewrite-clj under the hood.</p><p>Rewrite-clj has its own .zip name space, which largely as API corresponds with clojure.zip. For instance, in rewrite-clj, you go next or you go right, it’s going to skip over white space nodes, which is most of the time what you want. So you can deal with that AST and ignore the white space, but it is still there, when you build it up, you turn it back into text. The formatting is still there.</p><p>You also get some extra goodies, some extra functions that are particular to dealing with source code. So this is pretty nice library. Because, yeah, people keep asking me, “Okay, yeah. You know, I’ve looked at these zippers, but what do you use them for?” So this is, for instance, one real use case that I used them for not too long ago. A project that’s a year or two old, we started out with a certain namespace organizations and halfway through decided to have a bit more of a thought out namespace organization with proper, you know, reverse domain name and all of that stuff.</p><p>So, we had about 50% of our code using the old scheme and the new scheme. At a certain point we wanted to clean that all up. But at the same time it had to all happen as a big bang thing because you don’t want to block everyone who’s working on this. The solution was to write a script, which goes over to complete code base, updates or require statements, updates of namespaced keywords, all that kind of stuff so that we can find an opportune moment to merge as much as possible, run this once, get it over with.</p><p>This is one function in there, you see there’s a loop recur, which takes the input zipper and then just goes next, next, next. It goes over the complete source code until it’s either at the end, then it stops. Or if it’s looking at a list and the first element in that list is a :require keyword. In that case, it goes down into the list and passes the zipper to another function which will do the actual updating of the requires. This is one example of what zippers in the real world might look like.</p><p>This is my last slide. I didn’t go as much over time as I thought I would do. One more thing that I used zippers for recently, so I was working on a project which does a Jupiter notebook kind of interface. So, these are these environments in a web browser where you can evaluate code, and so we had streaming output from the server to the client. There’s two things that are tricky with that because you need to emulate a repl like a terminal. You need to deal with ANSI escape codes, but you also need to deal with carriage returns.</p><p>Like, if you have, for instance, a progress bar, you know, that would work in a terminal, then you need to emulate that in your HTML. If you’ve come across a carriage return, go back to the beginning of the line where the beginning of the line, in this case, is actually a bunch of spans that are encoding your color stuff. There I extensively used zippers to navigate back, “Okay, where is actually the beginning of the line, split this up.” So, that was another use case.</p><p>Yeah. That’s all I have. But I am here for questions. Questions. Yes.</p><p><span style="color: #442288">Participant:</span></p><p>I kind of wanna ask about possible extensions to the zipper idea. I think it’s cool. What if you wanted a zipper except with two pointers? Or what if you wanted a zipper that instead of a tree data structures worked on directed acyclic graph or general graphs. Do you have any ideas?</p><p><span style="color: #448822">Arne:</span></p><p>That’s an interesting research topic. I think the reason that zippers came to be is the idea of … actually I have one more slide here. I briefly mentioned this in passing, but it’s the idea of … Here are two different ways of doing the same thing, right? The bottom one, I’m just using Clojure’s assoc-in to replace a value deep inside a tree, whereas the top one, I’m doing the same thing with zippers. In this case, I mean, both do the exact same thing. But if you have a bunch of those assoc-in, so each time Clojure needs to go from the top to the bottom, make that change, and so you end up with a data structure on the right, which we use as a loc on the left, but anything from, you know, your position where you make the change up to the root, those will all be new nodes.</p><p>If you do a bunch of updates every single time, you’re rebuilding that path from your current position to the root. That’s really why zippers shine, because you can defer that cost as much as possible. So, in a more general graph structure, I think … I don’t know. I can’t answer off the top of my head if you can do the same. I mean, I guess, if your directed acyclic graph still has the concept of a root that you considered a root because you still wanna do it in kind of a functional way, then I guess you can do something similar.</p><p>Having to two pointers at the same time, I think this is going to start getting very, very tricky. You’re going to have to do a lot of bookkeeping and I don’t know if it still worth it … I guess, it also like to see if there’s a use case for it. It might be possible. Yeah.</p><p><span style="color: #442288">Participant:</span>
Would you say that the benefits of using zippers scale with the number of changes that you need to make deep inside of some nested structure?</p><p><span style="color: #448822">Arne:</span></p><p>I mean, that’s one of the benefits of zippers, is in the performance gain. So, yes, performance-wise you’re gonna get more bang for your buck, the more that you can cluster operations together, navigate to a certain position, do your changes there and then all these zip at once. But I think the other thing, people that use zippers, I don’t think performance is often the first reason, I think the main reason is that there’s certain problems where there’s navigation API is just very elegant. That’s probably the main reason that people choose zippers.</p><p>All right. Well, I’ll be here. I’m happy to chat later.</p><p><span style="color: #884422">Dan:</span>
Thank you.</p><p><span style="color: #448822">Arne:</span>
Thank you all very much.</p><p><span style="color: #884422">Dan:</span>
That’s great. Just because I don’t think we were streaming then, Arne, how do I pronounce your name, I’m sorry?</p><p><span style="color: #448822">Arne:</span>
Arne.</p><p><span style="color: #884422">Dan:</span>
Arne, okay.</p><p><span style="color: #448822">Arne:</span>
Arne.</p><p><span style="color: #884422">Dan:</span>
You do freelance Clojure consulting. Are you looking for more clients?</p><p><span style="color: #448822">Arne:</span>
Actually, not right now. I mean, I’m always happy to talk, there might be interesting stuff, especially when it comes to training with some people in Berlin.</p><p><span style="color: #884422">Dan:</span>
You could still … maybe put on your wait list?</p><p><span style="color: #448822">Arne:</span></p><p>Absolutely. But I actually just recently said goodbye to a consulting gig because I wanna spend some time on Lambda Island. I got some open source work that I’m committed to now with Clojurists Together. I’m also organizing a conference next year. It might be a little bit far from most of you. It’s gonna be in Belgium, Heart of Clojure, but it’s going to be an absolutely kick ass conference early August. I’m gonna be busy the next year, is what I’m saying.</p><p><span style="color: #884422">Dan:</span></p><p>I’m going to be out in Belgium speaking in a conference in two weeks. Can you maybe schedule it for them? That’s great. I’ve never been to Belgium, so I’ll check it out. I’ll pre-scout it for next year’s Clojure conference.</p><p><span style="color: #448822">Arne:</span></p><p>Yeah, yeah. That’d be awesome.</p><p><span style="color: #884422">Dan:</span></p><p>That’s great. Cool. Well, thank you for speaking for us and I appreciate your help.</p></<>>Test Wars: A New Hopehttps://lambdaisland.com/blog/2018-11-02-test-wars-new-hope2018-11-02T16:56:03+00:00<<>><p>Yesterday was the first day for me on a new job, thanks to <a shape="rect" href="https://clojuriststogether.org/">Clojurists
Together</a> I will be able to dedicate the coming
three months to improving <a shape="rect" href="https://github.com/lambdaisland">Kaocha</a>, a next
generation test runner for Clojure.</p><p>A number of projects applied for grants this quarter, some much more established
than Kaocha. Clojurists Together has been asking people through their surveys if
it would be cool to also fund “speculative” projects, and it seems people
agreed.</p><p>I am extremely grateful for this opportunity. I hope to demonstrate in the
coming months that Kaocha holds a lot of potential, and to deliver some of that
potential in the form of a tool people love to use.</p><p>In this post I’d like to set the stage by providing a bit of context, and
explaining the problems Kaocha tries to solve, as well as providing a rough plan
for the coming months.</p><h2 id="conventions-and-interfaces">Conventions and interfaces</h2><p>When growing an ecosystem it’s important to have conventions and well defined
interfaces in place. This allows parts of the ecosystem to develop separately,
while still functioning as a whole.</p><p>Clojure has done a fairly good job of providing this shared foundation with
regard to testing, and many tools and libraries have been built on top of it. At
the same time some have chosen to go off and do their own thing, possibly
because the affordances provided by Clojure’s testing conventions did not
support their use cases.</p><p>When talking about a testing ecosystem we’re talking about a bunch of different
things, which may or may not come in a single package.</p><ol><li>Test syntax: how to define tests in code</li><li>Test semantics: how does the system define what is a test</li><li>Assertion libraries: ways to write falsifiable test statements</li><li>Mocking/stubbing libraries: utilities to help with test isolation</li><li>Test fixtures and factories: utilities for building up test data and establishing application states</li><li>Progress reporters: ways to get progress information while tests are running</li><li>Result reporters: summarizing the results of a test run for human or machine consumption</li><li>Test runners: comprehensively load, detect, and run test suites</li></ol><p>Clojure 1.0 provided exactly one of these, number 2. When a var has <code>:test</code>
metadata, then that is considered a test, and can be run with
<code>clojure.core/test</code>. You could argue it also had a bit of 3 with
<code>clojure.core/assert</code>, and a bit of 4 with <code>with-redefs</code>.</p><p>Clojure 1.1 added the <code>clojure.test</code> library which improves on number 3
(assertions) with an extensible macro: <code>(is)</code>, as well as providing syntax (1),
and reporters (6 and 7).</p><p>More importantly it provides conventions so you can bring your own version of
these things. <code>(is ...)</code> is powered by the <code>clojure.core/assert-expr</code>
multimethod, so for instance
<a shape="rect" href="https://github.com/nubank/matcher-combinators">matcher-combinators</a> provides a
custom <code>(is (match? ...))</code> assertion.</p><p>The reporter in <code>clojure.test</code> is also a multimethod. It receives “events” like
<code>:begin-test-var</code>, <code>:pass</code>, or <code>:fail</code>, and can be extended to handle new types
of event, like matcher-combinator’s <code>:mismatch</code>. It can also be swapped out with
<code>with-redefs</code>, for instance when running tests through CIDER a custom reporter
will be used to collect test results.</p><p>There is a chance of conflict here though, what if one library extends <code>report</code>,
and the other replaces it? This illustrates a general problem with
<code>clojure.test</code>, to build tooling on top of it you are forced to rely on details
of the implementation, leading to brittle results that don’t compose well.</p><p>Clojure itself does not contain a test runner. There are some utilities in
<code>clojure.test</code> to execute tests once they are defined, but no comprehensive way
to say “load all test namespaces under this directory path, find all tests in
them, and run them.” The role of a test runner has initially been performed by
Leiningen, which includes <code>lein test</code>. It’s fairly basic but gets the job done.
It will set an appropriate process exit code (important on CI), and provides
some ways to filter or skip tests. It uses stock <code>clojure.test</code> output which is
rather spartan.</p><p>Several libraries and Leiningen plugins came about to provide nicer output for
failed assertions
(<a shape="rect" href="https://github.com/pjstadig/humane-test-output">humane-test-output</a>,
<a shape="rect" href="https://github.com/venantius/ultra">ultra</a>), as well as things like a nice
progress bar, output capturing and running tests in parallel
(<a shape="rect" href="https://github.com/weavejester/eftest">eftest</a>).</p><p>When Boot came around it also needed a test runner, and since <code>lein test</code> is
tightly linked to Leiningen the boot folks implemented
<a shape="rect" href="https://github.com/adzerk-oss/boot-test">boot-test</a>, which uses
humane-test-output under the hood.</p><p>The fine folks from Metosin later created
<a shape="rect" href="https://github.com/metosin/bat-test">bat-test</a>, which is perhaps the most
feature complete of the ones covered so far. It works with boot and leiningen,
supports eftest and cloverage, has global pre/post hooks and more.</p><p>And then <code>tools.deps</code> / Clojure CLI came around and we were back to square one,
waiting for someone to port their existing test runner, or implement a new one.
Cognitect came around with
<a shape="rect" href="https://github.com/cognitect-labs/test-runner">cognitect-labs/test-runner</a>,
which is probably the least feature complete of the bunch.</p><p>Finally there’s <a shape="rect" href="https://github.com/circleci/circleci.test">circleci.test</a>,
which provides some really neat features like enforcing tests are pure and
generating junit.xml output.</p><h2 id="imperative-vs-functional">Imperative vs functional</h2><p>If we have this many test runners already, why create another one? All of the
mentioned projects are available for a subset of Clojure build tools (lein,
boot, Clojure CLI), and implement a subset of all possible test runner features,
but they don’t compose. I can’t take some features of one and a few of the other.</p><p>Clojure’s philosophy is to provide pure functions over common data structures,
so it’s easy to piece things together. The problem is that running tests is
inherintly imperative. You load code, execute tests in order, and observe events
and exceptions that occur. These get aggregated in some kind of mutuble
structure, while progress is reported to the user.</p><p>The problem is that this kind of code doesn’t compose well, and so we end up
with all these different projects with very little opportunity for synergy or
reuse.</p><p>Kaocha addresses this by turning the imperative problem into a functional one.
At the heart of Kaocha is the abstraction of a testable, a Clojure map with an
<code>:id</code>, a <code>:type</code>, and any other information that characterizes this test. When
you <code>(run testable)</code> you get a similar map back, but this time containing test
results: the amount of assertions that ran, errors, failures, captured
exceptions, captured output to stdout, and so forth.</p><p><code>run</code> is by no means pure, since a test can cause (and verify) any number of
side effects, but tooling authors can choose to ignore the side effects. All
they care about is captured and returned as pure data.</p><p>Testables are built up hierarchically. You can <code>run</code> a test-suite, which will
<code>run</code> any namespaces it contains, which in turn will <code>run</code> each test var. The
top level testable is called the test-plan.</p><pre><code class="language-clojure">(def test-plan
{:tests [{:id :unit
:type :clojure.test
:tests [{:id :kaocha.result-test
:type :ns
:tests [{:id :kaocha.result-test/totals-test
:type :var
,,,}]}]}]})
(run test-plan)
;;=>
{:tests [{:id :unit
:type :clojure.test
:tests [{:id :kaocha.result-test
:type :ns
:tests [{:id :kaocha.result-test/totals-test
:type :var
:count 1
:fail 0
:pass 1
:error 0
,,,}]}]}]}
</code></pre><p>Kaocha is still built upon the foundations of <code>clojure.test</code>, and uses the same
“reporter” abstraction for providing real time progress info. This isn’t always
easy, as mentioned before <code>clojure.test</code> makes it easy to step on each other’s
toes when extending or replacing the reporter. Kaocha tries to prevent this by
providing its own conventions for extensions that are more readily composable.
You can add additional reporters, instead of replacing the reporter wholesale.
You can make Kaocha aware of custom event types through keyword hierarchies, and
by implementing multimethods you can influence how custom events are rendered.</p><p>See the <a shape="rect" href="https://cljdoc.org/d/lambdaisland/kaocha/CURRENT/doc/8-extending#reporters">docs about custom
reporters</a>
for more info.</p><h2 id="more-than-one-type-of-tests">More than one type of tests</h2><p>Long gone are the days that <code>clojure.test</code> was the only game in town. What if
you prefer to write your tests with <a shape="rect" href="https://github.com/marick/Midje">midje</a>,
<a shape="rect" href="https://github.com/jimpil/fudje">fudje</a>,
<a shape="rect" href="https://github.com/clojure-expectations/expectations">expectations</a>,
<a shape="rect" href="https://github.com/nubank/selvage">selvage</a>, or
<a shape="rect" href="https://github.com/cucumber/cucumber-jvm-clojure">cucumber</a>?</p><p>Some of these stick to the clojure.test conventions, and can be used with any of
the above test runners, others do not. Midje in particular is interesting here,
as it provides its own version of more or less everything we mentioned: syntax,
test runner, mocking/stubbing and more, and it does it in a very idiosyncratic
way, deliberately moving away from clojure.test.</p><p>So you can’t for instance use Midje with bat-test, or circleci.test, you can
only use Midje with Midje. Conversely you can’t easily use features of Midje
without adopting it wholesale.</p><p>What’s worse, imagine you want to use one testing library for your unit tests,
and another for your integration tests, now you’re dealing with separate tools,
each with their own configuration and interface.</p><p>Kaocha tries to solve this by providing a test type (or test suite) abstraction.
To add support for a given test type to Kaocha you implement two multimethods,
<code>load</code> and <code>run</code>. Getting these right isn’t trivial, as they need to do the
double book keeping mentioned above, invoking the reporter as things are
happening, and collecting and returning the results in a functional way. What
they don’t need to worry about is all the rest of the test runner machinery.
They don’t need to provide a CLI or REPL interface, they don’t need to think
about functionality for filtering or skipping tests, for profiling or output
capturing, all of that is provided by Kaocha.</p><p>Kaocha does not need to know about your test type up front, it just needs to be
on the classpath and follow certain naming conventions. This means that anyone
can implement a new test type and push it to clojars, encouraging
experimentation and innovation.</p><p>Besides supporting command line use (<code>kaocha.runner</code>), or REPL interaction
(<code>kaocha.repl</code>), Kaocha can also be invoked directly by tooling (<code>kaocha.api</code>).
Perhaps in the future editor integrations based on Kaocha will gain support for
all these different test types without any extra work.</p><h2 id="moar-features">Moar features</h2><p>People have many different approaches to writing and running tests, and this
leads to different opinions about the features a test tool should support. One
person’s essentials is an other person’s bloatware.</p><p>Kaocha tries to address this through plugins. Like test types these can be
released by anyone, they are just Clojure libraries that follow certain naming
conventions. Because of the functional, data-driven foundation that Kaocha
provides plugin authors can ignore a lot of the details of running tests, and
focus on the problem they are trying to solve.</p><p>For example the
<a shape="rect" href="https://github.com/lambdaisland/kaocha-junit-xml">kaocha-junit-xml</a> plugin
provides support for writing out a <code>junit.xml</code> file with test results.</p><p>The guts of this plugin look like this. It implements three “hooks”, one to add
an extra command line argument, one that transforms the test configuration, and
that runs after all tests have finished.</p><p>The first two establish a common pattern, allowing configuration through
<code>tests.edn</code> that can be overridden by supplying command line arguments.</p><pre><code class="language-clojure">(defplugin kaocha.plugin/junit-xml
"Write test results to junit.xml"
(cli-options [opts]
(conj opts [nil "--junit-xml-file FILENAME" "Save the test results to a Ant JUnit XML file."]))
(config [config]
(if-let [target (get-in config [:kaocha/cli-options :junit-xml-file])]
(assoc config ::target-file target)
config))
(post-run [result]
(when-let [filename (::target-file result)]
(write-junit-xml filename result))
result))
</code></pre><p><code>write-junit-xml</code> takes the test result, a nested map with the details of every
single test, and transforms it into XML. Quite a difference from
<a shape="rect" href="https://github.com/clojure/clojure/blob/master/src/clj/clojure/test/junit.clj">clojure.test.junit</a>,
which imperatively spits out the XML bit by bit as it goes along.</p><pre><code class="language-clojure">(defn write-junit-xml [filename result]
(with-open [f (io/writer (io/file filename))]
(binding [*out* f]
(clojure.xml/emit (test-result->xml result)))))
</code></pre><p>Note also that this plugin will work with any test type, not just
<code>clojure.test</code>, because they all produce the same Kaocha test-result data
structure.</p><p>Many of the features I have planned for Kaocha will be provided as plugins.
Those that rely on extra third party dependencies will be deployed as separate
jars. Maximum features, minimum bloat!</p><p>Some other plugin ideas</p><ul><li>Cloverage integration</li><li>Detecting side effects (a la circleci.test)</li><li>Configurable global pre/post steps</li><li>Configuring dynamic bindings in tests.edn (e.g. <code>clojure.core/*print-length*</code>)</li><li>Configuration for test.check (report-trials, report-shrinking, default-test-count)</li></ul><h2 id="where-are-we-now">Where are we now?</h2><p>I’ve been working steadily on Kaocha since April, initially focusing on the base
data structures, abstractions, and common functionality. I also put some effort
into ergonomics, making sure the tool is pleasant to use. Most recently this led
to <a shape="rect" href="https://github.com/lambdaisland/deep-diff">deep-diff</a>, which was split out
into its own library.</p><p>The core functionality and extension points are there, including test types,
reporters and plugins, as well as essential features like watching/autoloading,
skipping and filtering, profiling, randomizing test order, and more.</p><p><code>clojure.test</code> is the only test type that’s well supported so far, there’s a
proof of concept of a Midje test type.</p><p>The release process has been streamlined so I can push out new versions quickly,
you can expect new releases multiple times per week. Kaocha’s own tests run on
CircleCI on a mix of Java and Clojure versions, and documentation is published
on <a shape="rect" href="https://cljdoc.org/d/lambdaisland/kaocha/CURRENT">cljdoc</a>.</p><h2 id="whats-next">What’s next?</h2><p>I’ll be funded by Clojurists Together from November through January, and I do
intend to make it count.</p><p>I have a long list of things I want to work on. I’ll start turning these into
Github issues in the coming days so people can follow along.</p><p>Some of the things I mentioned in my Clojurists Together application have
already been implemented, like <a shape="rect" href="https://github.com/lambdaisland/kaocha-junit-xml">junit.xml
output</a>, and pretty diffing of
test failures, which got pulled out into a standalone <a shape="rect" href="https://github.com/lambdaisland/deep-diff">deep-diff
library</a>.</p><p>Boot support is high on the list, with that all three major build tools will be
supported. Cloverage integration is also a priority, partly because I want to
get a better view on Kaocha’s own coverage.</p><p>I’ll also start working on supporting more test types. The current test type
abstraction hasn’t been fully put to the test yet, it will be interesting to see
how well it can support other testing libraries, or whether its design needs to
evolve.</p><p>The big one of these will be a ClojureScript test type. One of my dreams with
Kaocha is to have a dead simple way to drop a <code>tests.edn</code> into your project and
start testing both Clojure and ClojureScript code. This will be a major task as
I <a shape="rect" href="https://clojureverse.org/t/announcing-kaocha-a-new-and-improved-clojure-test-runner/2903">explained elsewhere</a>,
as it involves running the ClojureScript compiler, collecting metadata from the
compiler, spinning up or connecting to a JavaScript runtime, invoking tests in a
JS/CLJS harnass, and collecting the result.</p><p>Parallelization is one of the few features of Eftest and bat-test that Kaocha
does not yet support. It hasn’t been high on the list because Kaocha by default
randomizes the test order, providing a random seed that can be used to recreate
a test run. This is a useful feature for catching unintended dependencies
between tests, but it relies on test order being deterministic, which with
parallelism is no longer the case. Still I understand why people would want to
trade this determism in for higher throughput, and so I do intend to bring in
parallelism, but it might require some reworking of Kaocha’s core.</p><p>There are also a bunch of smaller incremental improvements, like providing nicer
feedback in case of misconfiguration. My current thinking is to get a lot of the
smaller tasks out of the way in the first few weeks in order to get some
momentum, and then start tackling ClojureScript support.</p><p>Towards the end of the three month program I intend to start working on
documentation, and on making sure the project is easy to contribute to.</p><p>Meanwhile I’d love to hear your feedback. Did you try Kaocha yet? What did you
think? Are there features missing that are blocking you from switching?</p><p>You can <a shape="rect" href="https://lambdaisland.com/p/contact/">contact me</a> directly, or comment on this post <a shape="rect" href="https://www.reddit.com/r/Clojure/comments/9tlvr6/test_wars_a_new_hope/">on r/clojure</a>.</p></<>>Two Years of Lambda Island, A Healthy Pace and Things to Comehttps://lambdaisland.com/blog/2018-06-15-second-year-lambda-island2018-06-15T12:49:18+00:00<<>><p>It’s been just over two years since Lambda Island first launched, and just <a shape="rect" href="https://lambdaisland.com/blog/2017-05-13-lambda-island-turns-one">like
last year</a> I’d
like to give you all an update about what’s been happening, where we are, and
where things are going.</p><p>To recap: the first year was <em>rough</em>. I’d been self-employed for nearly a
decade, but I’d always done stable contracting work, which provided a steady
stream of income, and made it easy for me to unplug at the end of the day.</p><p>Lambda Island was, as the Dutch expression goes, “a different pair of sleeves”.
I really underestimated what switching to a one-man product business in a niche
market would mean, and within months I was struggling with symptoms of burnout,
so most of year one was characterised by trying to keep things going and stay
afloat financially, while looking after myself and trying to get back to a good
place, physically and mentally.</p><p>Luckily that all worked out, and during this second year I’ve managed to find a
steady, sustainable pace. I’ve been recharging, I even managed to take some
holidays this time, and I’m slowly rebuilding my previously depleted savings.</p><p>I’m still not able to live off Lambda Island completely, but it provides a good bit of
income. One lesson I’ve learned first hand is that the greatest product is worth
nothing without good marketing, and of all the hats I wear the one that says
“Marketer” is perhaps the one that suits me least.</p><p>Still, I have gained some recognition for my efforts in Clojure’s community and
in Open Source development, as well as for creating content of exceptional
quality, and so there continues to be a steady trickle of new signups. This year
has also seen more teams signing up for company plans, which I think is a great development.</p><p>It makes a lot of sense, Lambda Island has always focused on the kind of things
you’d need to know and use in an actual job, rather than on what’s the latest
hotness, and so teams are finding it a great way to quickly <a shape="rect" href="https://lambdaisland.com/episodes/datomic-quickstart-part-1">introduce people to
Datomic</a>, <a shape="rect" href="https://lambdaisland.com/collections/clojure-foundations">teach
people foundational Clojure
concepts</a>, or improve
their <a shape="rect" href="https://lambdaisland.com/collections/clojure-testing">approach to
testing</a>.</p><h2 id="coaching-and-training">Coaching and Training</h2><p>Hiring and onboarding Clojure developers isn’t always easy. There is only a
limited amount of senior Clojurists in any given locality, and so companies have
to train and mentor more junior profiles, as well as experienced devs coming
from other languages. Lambda Island can be an excellent resource for this.</p><p>I’ve been helping one company with this process directly.
<a shape="rect" href="https://nextjournal.com/">Nextjournal</a> is building an ambitious product using
Clojure and ClojureScript, but most of the devs are coming from Erlang and
Elixir.</p><p>Over the past year I’ve helped them figure out issues with their tooling to make
sure everyone can work comfortably. Through one-on-one coaching sessions and
code reviews I’ve helped people to grasp Clojure’s guiding principles, adopt a
REPL-based workflow, get better at writing idiomatic code, and generally get
over the uncertainty of “are we doing this right?”.</p><p>From talking to people at conferences and meetups it seems there are more
companies that could benefit from this kind of personal coaching, and so I’ve
been talking with some talented Clojure people to see if we could start offering
this as a general service. I’m very excited about this possibility! If you think
your company could benefit from any kind of training or coaching by experienced
Clojurists, or if you want to be the first to know when this service becomes
available, then please drop me a line (arne at lambdaisland dot com). We’re
still figuring out the specific, and any concrete input at this point would be
extremely valuable.</p><h2 id="lambda-island">Lambda Island</h2><p>Lambda Island will continue to regularly publish new content, the next episode
will be a follow-up episode to <a shape="rect" href="https://lambdaisland.com/episodes/transducers">Episode 38.
Transducers</a>. While that one
taught you how to use the built-in transducers, the new episode will dig deeper
into how transducers work, and look at some powerful transducer librarier like
<a shape="rect" href="https://github.com/cgrand/xforms">xforms</a> and
<a shape="rect" href="https://github.com/MastodonC/kixi.stats">kixi.stats</a>.</p><p>Year two has seen fewer new episodes than year one, only about a dozen versus
the 30 episodes I crancked out in those first twelve months. This has turned out
to be a more sustainable pace, and while I would like to increase the frequency
a bit again it will remain closer to once a month than to once a week.</p><p>You may have noticed that the title screens have started looking a lot nicer
though, that’s because this work is now done by the talented <a shape="rect" href="https://twitter.com/moolver_sin">Lubov
Soltan</a>, who also created the branding for the
<a shape="rect" href="http://clojuredays.org/">Dutch Clojure Days</a>. This has been a big step for me
as it’s the first time another person has been involved in producing episodes.</p><div><div class="flex"><div><p><a shape="rect" href="https://lambdaisland.com/episodes/react-components-reagent"><img src="https://i.vimeocdn.com/video/702033997_1280x720.jpg" /></a></p></div><div><p><a shape="rect" href="https://lambdaisland.com/episodes/ultimate-dev-setup"><img src="https://i.vimeocdn.com/video/692032990_1280x720.jpg" /></a></p></div></div><div class="flex"><div><p><a shape="rect" href="https://lambdaisland.com/episodes/integrant"><img src="https://i.vimeocdn.com/video/688845330_1280x720.jpg" /></a></p></div><div><p><a shape="rect" href="https://lambdaisland.com/episodes/transducers"><img src="https://i.vimeocdn.com/video/678258031_1280x720.jpg" /></a></p></div></div></div><h2 id="buying-episodes">Buying episodes</h2><p>Something else I’ve been working on is making it possible to buy individual
episodes. So far the only way to access premium content has been through a
subscription. This made sense at the beginning, as there wasn’t much content
yet, and new videos were coming out regularly. Subscriptions were offered cheap,
and what you were really buying was the promise of future updates.</p><p>Now that there’s a substantial catalog new subscribers get instant access to 40+
episodes, about 9 hours of content. Subscription prices have gone up a bit, but
not nearly as much as they should considering the value you’re getting. On the
other hand consumers have become more price conscious about recurring
charges for online services.</p><p>So the plan is to sell individual episodes instead, they’ll likely be priced
around the ten dollar mark. There will still be subscriptions, but they will be
marketed more as premium “all access passes” aimed at companies.</p><p>This will also make it more rewarding for me to create new content, and to do
better marketing, as I can more directly correlate my efforts to sales. I’ll
also be able to see more clearly which topics work well and which don’t. In the
end there’s no better way to figure out what people want than by letting them vote
with their wallet.</p><p>Of course all existing customers will be “grandfathered in”, you can keep your
existing plan as long as you like, even those that signed up for ridiculously
cheap yearly plans all the way in the beginning. You’ve supported me from the
start, this is the least I can do to say “thank you”!</p><h2 id="privacy">Privacy</h2><p>Finally I’ve been shipping various improvements and fixes for long standing bugs
to the site. Part of this has been to make the site GDPR compliant. Email
notifications are now strictly opt-in, and there’s a <a shape="rect" href="https://lambdaisland.com/p/privacy-policy">Privacy
Policy</a> that complements the existing
<a shape="rect" href="https://lambdaisland.com/p/cookies">Cookie Policy</a>.</p><p>I’ve also removed all third party JS and other assets, with the exception of the
Vimeo player, which unfortunately also injects some analytics tracking of its
own. We might add back some server-side analytics in the future if it makes
sense for marketing purposes, but as it stood the analytics were rarely looked
at, so no need to let BigCorp track you because of it.</p><h2 id="community-and-open-source">Community and Open Source</h2><p>It has been part of Lambda Island’s mission to support the Clojure community and
ecosystem, and plenty has happened on that front this year.</p><p>The big one has been that we <a shape="rect" href="https://clojureverse.org/t/announcing-clojureverse-2-0/875">relaunched
ClojureVerse</a>. This
forum had been up and running for several years, but few people actually knew
about it. We figured there was a need for an alternative online space, one
that’s less formal than the Clojure mailing list, and less noisy than the
Clojurians slack. A warm and welcoming place for thoughtful discourse, and for
sharing what you’re learning and working on.</p><p>On the <a shape="rect" href="http://blog.cognitect.com/blog/2017/1/31/clojure-2018-results#block-yui_3_17_2_1_1521472913853_182124">2018 Clojure Community
Survey</a>
over 17% of respondents mentioned using ClojureVerse, which considering it was
only a few months since the relaunch is a really nice result.</p><p>We’ve also replaced the <a shape="rect" href="https://clojurians-log.clojureverse.org/">Clojurians slack log</a> with a proper Clojure app (<a shape="rect" href="https://github.com/clojureverse/clojurians-log-app">github</a> and <a shape="rect" href="https://clojureverse.org/t/replacing-the-clojurians-slack-log/1614">announcement</a>). The old kludge of Python and Node scripts had become a nightmare to maintain. By moving it to a Clojure web app it has become a lot easier for people to submit contributions. Message parsing and rendering has much improved, and we’re properly showing threaded messages, a Slack feature that didn’t exist when the old site launched.</p><p>There is still work to be done. So far we’ve focused on getting the thing run smoothly and render things properly, and to make sure the site stays accessible when half a dozen indexing bots are crawling it at once. It’s all taken a bit of time, but this is a long term effort, and eventually we’re getting there.</p><p>The main thing that’s still missing is to automate the import of new logs into the database. This is currently still a manual process, which means that the site is often quite a bit behind. This too will get sorted out in time though.</p><p>Lambda Island picks up the tab for the hosting and the domains for both ClojureVerse and the Clojurians log, and does the bulk of the system administration and development for these projects.</p><h3 id="open-source">Open Source</h3><p>In terms of open source it’s also been a good year for Lambda Island. We’ve released new versions of <a shape="rect" href="https://github.com/lambdaisland/uri">lambdaisland/uri</a>, <a shape="rect" href="https://github.com/plexus/chestnut">Chestnut</a>, <a shape="rect" href="https://github.com/lambdaisland/ansi">lambdaisland/ansi</a>; contributed to the Emacs world with <a shape="rect" href="https://github.com/plexus/chemacs">Chemacs</a> and <a shape="rect" href="https://github.com/clojure-emacs/parseclj">parseclj</a>; submitted patches to <a shape="rect" href="https://github.com/clojure/tools.cli">clojure.tools.cli</a>, <a shape="rect" href="https://github.com/weavejester/integrant">integrant</a>, <a shape="rect" href="https://github.com/xsc/rewrite-clj">rewrite-clj</a>, <a shape="rect" href="https://github.com/nubank/matcher-combinators">matcher-combinators</a>, <a shape="rect" href="https://github.com/plexus/cider-nrepl">cider-nrepl</a>, <a shape="rect" href="https://github.com/bhauman/lein-figwheel">lein-figwheel</a>, <a shape="rect" href="https://github.com/jackrusher/sparkledriver">sparkledriver</a>, and <a shape="rect" href="https://github.com/metabase/toucan">toucan</a>. These are often small patches, but integrated over time they have likely made many people’s lives just a little easier.</p><p>I’m currently working on a project that could have a real impact on how people structure and run their test code, but since that’s not quite ready for prime-time yet I’ll save the details for a next installment.</p><h2 id="conclusion">Conclusion</h2><p>Looking in from the outside it may seem like not much has been happening here this past year, but nothing could be further from the truth. Several things have been brewing behind the scenes, and you’ll get to taste the fruit of that labor before long.</p><p>The biggest development is that Lambda Island is no longer a one-man venture, I’ve started collaborating more with others, both formally and informally, and you’ll be seeing much more of that in times to come, which is why I felt I could already dish out a royal “we” a few times in this post.</p><p>If you want to stay up to date about future developments then please <a shape="rect" href="https://lambdaisland.com/register">sign up</a>, and opt-in to receive our newsletter.</p></<>>D3 and ClojureScripthttps://lambdaisland.com/blog/2018-04-26-d3-clojurescript2018-04-26T18:33:30+00:00<<>><p><em>
This is a guest post by <a shape="rect" href="http://joannecheng.me/">Joanne Cheng</a> (<a shape="rect" href="https://twitter.com/joannecheng">twitter</a>), a freelance software engineer and visualization consultant based in Denver, Colorado. She has taught workshops and spoken at conferences about visualizing data with D3. Turns out ClojureScript and D3 are a great fit, in this post she’ll show you how to create your own visualization using the power of D3 and the elegance of ClojureScript.
</em></p><p>I use <a shape="rect" href="https://d3js.org/">D3.js</a> for drawing custom data visualizations.
I love using the library, but I wanted to try one of the several compile to JavaScript options,
and I decided to look into ClojureScript.
<a shape="rect" href="http://joannecheng.me/us-energy-slopegraph/">It ended up working out well for me</a>,
so I’m going to show you how I created a D3.js visualization using ClojureScript!</p><h2 id="what-were-visualizing">What we’re visualizing</h2><p>The <a shape="rect" href="https://www.eia.gov">U. S. Energy Administration</a> releases data on how much energy the US produces and consumes
and what types of energy are used.
<a shape="rect" href="https://www.npr.org/2015/09/10/319535020/coal-gas-nuclear-hydro-how-your-state-generates-power">NPR released slopegraphs</a>
of how much and what types of energy each state in the US
produces.
The article inspired me to dive into the data and create a
<a shape="rect" href="http://charliepark.org/slopegraphs/">slopegraph</a>
that shows how energy generation has changed in the entire country over the last 10 years.</p><p>I downloaded energy generation data from 2005 to 2015 from the U. S. Energy Information Administration data browser, converted it to a Clojure hash map, and used it to create
<a shape="rect" href="http://joannecheng.me/us-energy-slopegraph/">the visualization</a>.</p><p class="flex justify-center"><img src="/img/blog/20180426-d3_clojurescript_visualization.png" alt="" /></p><h2 id="setup">Setup</h2><h3 id="install-leiningen">Install Leiningen</h3><p>If you don’t have Clojure set up on your machine or installed it years ago, telling yourself that you would look into Clojure
<em>one day</em>, you’re going to need to install <a shape="rect" href="https://leiningen.org">Leiningen</a>.
Leiningen is your build tool and dependency manager for this project.
You can use your
<a shape="rect" href="https://github.com/technomancy/leiningen/wiki/Packaging">package manager of choice to install it</a> (preferred)
or check out the <a shape="rect" href="https://leiningen.org/#install">installation instructions on the official website</a>.</p><h3 id="figwheel">Figwheel</h3><p>We’re going to create a new Figwheel project.
<a shape="rect" href="https://github.com/bhauman/lein-figwheel">Figwheel</a> is what is going to create
our new project directory, build our ClojureScript code, and autoreload the
browser every time our code changes.</p><p>In your command line, go to the directory where you want your project to live and run the following command:</p><pre><code class="language-sh">lein new figwheel us-energy-slopegraph
</code></pre><p>Sit back a few seconds, and make sure no errors appear. Here’s what the output should look like:</p><pre><code>Generating fresh 'lein new' figwheel project.
Change into your 'us-energy-slopegraph' directory and run 'lein figwheel'
Wait for it to finish compiling
Then open 'http://localhost:3449/index.html' in your browser
</code></pre><p>Now you can change into your newly created project directory, and start Figwheel:</p><pre><code class="language-sh">cd us-energy-slopegraph
lein figwheel
</code></pre><p>You should now see something like this:</p><pre><code class="language-sh">$ lein figwheel
Figwheel: Cutting some fruit, just a sec ...
Figwheel: Validating the configuration found in project.clj
Figwheel: Configuration Valid ;)
Figwheel: Starting server at http://0.0.0.0:3449
Figwheel: Watching build - dev
Compiling build :dev to "resources/public/js/compiled/us-energy-slopegraph.js" from ["src"]...
Successfully compiled build :dev to "resources/public/js/compiled/us_energy_slopegraph.js" in 1.689 seconds.
........
Prompt will show when Figwheel connects to your application
dev:cljs.user=>
</code></pre><p>After a little while your browser will pop up with Figwheel’s welcome page,
located at <a shape="rect" href="http://0.0.0.0:3449">http://0.0.0.0:3449</a>. Now your ClojureScript
project is all set up!</p><p>Every time you save a file in the project directory, Figwheel will automatically recompile
and reload your browser, so you don’t need to manually refresh the page.</p><p>Once it’s fully loaded, Figwheel will switch to a REPL, a command line prompt that lets you evaluate bits of ClojureScript.
This allows you to enter bits of ClojureScript code directly into your terminal, and it will be executed by your browser.</p><p>Type this in and press “enter”:</p><pre><code class="language-clojure">(js/alert "Now look at your browser!")
</code></pre><h3 id="installing-d3js">Installing D3.js</h3><p>In your <code>us-energy-slopegraph</code> project directory, open the file called <code>project.clj</code>.
Insert <code>[cljsjs/d3 "4.12.0-0"]</code> in that list of library names after <code>:dependencies</code>:
It should look something like this:</p><pre><code class="language-clojure"> :dependencies [[org.clojure/clojure "1.8.0"]
[org.clojure/clojurescript "1.9.946"]
[org.clojure/core.async "0.3.443"]
[cljsjs/d3 "4.12.0-0"]]
</code></pre><p>Now you’re going to want Figwheel to install this dependency.
Restart Figwheel by hitting <code>Ctrl-D</code> and running <code>lein figwheel</code> again.
Figwheel will install any new dependencies added to <code>project.clj</code>.</p><p>Open up <code>src/us_energy_slopegraph/core.cljs</code> — you’ll mostly be working in this file.
You may notice there are a few lines of boilerplate code.
Remove the <code>println</code> and the <code>defonce app-state</code> lines - we don’t need those for this project.
<code>core.cljs</code> should now look something like this:</p><pre><code class="language-clojure">(ns example-project.core
(:require ))
(enable-console-print!)
(defn on-js-reload []
)
</code></pre><p>The first line is the namespace definition.
This is also where we’ll load the D3.js package that we just installed.
Require <code>cljsjs.d3</code> in this file by altering the first two lines to look like this:</p><pre><code class="language-clojure">(ns us-energy-slopegraph.core
(:require [cljsjs.d3]))
</code></pre><p>By loading this namespace you make sure the necessary JavaScript files get
loaded. These typically define global variables, which you can access with the
<code>js</code> prefix, like <code>js/d3</code>. If you wanted to add React to this ClojureScript
project, you would add it as a dependency, require <code>cljsjs.react</code>, and access it
with <code>js/React</code>.</p><div class="ba bg-washed-blue b-black-50 mv4 mh3 pa3"><h3 id="clojurescript-and-javascript-libraries">ClojureScript and JavaScript libraries</h3><p>Using existing JavaScript libraries from ClojureScript can be challenging. ClojureScript does aggressive code optimizations, but these only work when the compiler is able to analyze the complete project, including third party libraries, and this isn’t always possible for code found “in the wild”.</p><p>So called “externs” provide extra hints to the compiler about external references that should not be optimized away. The <a shape="rect" href="http://cljsjs.github.io/">CLJSJS project</a> bundles popular JavaScript libraries together with the necessary externs, and publishes them on Clojure’s code repository: Clojars.</p><p>More recent versions of ClojureScript support NPM packages directly, so in time the need for CLJSJS will go away, but for now it’s still the most convenient way to include JavaScript libraries in your ClojureScript project.</p></div><h2 id="code">Code</h2><p>Let’s first add some scaffolding to make sure reloading works smoothly.</p><p>When you open <a shape="rect" href="http://localhost:3449/index.html">http://localhost:3449/index.html</a> in your browser, it’s loading <code>resouces/public/index.html</code>.
Let’s make a few changes so it works better with this project.</p><p>First add this CSS link to the <code><head></code> section:</p><pre><code class="language-html"><head>
...
<link href="https://rawgit.com/lambdaisland/us-energy-slopegraph/master/resources/public/css/style.css" rel="stylesheet" type="text/css">
</head>
</code></pre><p>And then replace the <code><body></code> tag with this:</p><pre><code class="language-html"><body>
<div id="slopegraph">
</div>
<script src="js/compiled/us_energy_slopegraph.js" type="text/javascript"></script>
<script>us_energy_slopegraph.core.main()</script>
</body>
</code></pre><p>Save and reload the page in the browser.
We’ve added a <code><div></code> for the visualization, and an extra <code><script></code> tag.
This script calls an exported function called <code>main</code> in the <code>us-energy-slopegraph.core</code>
namespace and is called once when the page loads. We’ll implement this <code>main</code> function in ClojureScript.</p><p>Add this <code>main</code> function (make sure it’s before the <code>on-js-reload</code> function) in <code>core.cljs</code> and export it so it’s callable from JavaScript.</p><pre><code class="language-clojure">(defn ^:export main []
(println "This is the main function."))
</code></pre><p>Also call this <code>main</code> function in <code>on-js-reload</code></p><pre><code class="language-clojure">(defn on-js-reload []
(println "This is the on-js-reload function.")
(main))
</code></pre><p>Now make some changes in <code>core.cljs</code> — you can just change the string in the <code>println</code> function —
and save the file.
When you look at your browser’s JavaScript console you’ll see that
<code>on-js-reload</code> gets called every time you make changes.</p><p>Why are we using this pattern? Figwheel has an autorefresher built in that will
load what we write into the browser. We want to create a <code>main</code> function so we
can define exactly what happens when the the app loads the first time, and what
happens on subsequent reloads.</p><h2 id="svg">SVG</h2><p>One of D3’s core concepts is the <a shape="rect" href="https://bost.ocks.org/mike/join/"><strong>data join</strong></a>.
This tutorial won’t go into the deep parts of D3.js,
so if you’re completely unfamilar with it, I suggest looking into
<a shape="rect" href="https://bost.ocks.org/mike/join/">Mike Bostock’s blog post</a> or the book
<a shape="rect" href="http://alignedleft.com/tutorials/d3">Interactive Data Visualization for the Web</a>.</p><p>We’re going to be drawing our visualization using
<a shape="rect" href="https://developer.mozilla.org/en-US/docs/Web/SVG">SVG</a>.
To start, we’ll need to create an SVG element using D3.js.</p><p>Before you continue you might want to grab
<a shape="rect" href="https://github.com/lambdaisland/us-energy-slopegraph/blob/master/src/us_energy_slopegraph/core_template.cljs">this code template</a>
off Github. ClojureScript can be quite particular about the order you put things
in. The template contains empty stubs for most of the functions so you can simply fill them in, and named sections that should make it clear where to put the rest.</p><pre><code class="language-clojure">;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; data
(def height 450)
(def width 540)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Lifecycle
(defn append-svg []
(-> js/d3
(.select "#slopegraph")
(.append "svg")
(.attr "height" height)
(.attr "width" width)))
</code></pre><p><code>append-svg</code> is a function that selects an HTML element with the
id of <code>slopegraph</code>, appends an <code>svg</code> element, and sets <code>height</code> and <code>width</code> attributes on it.
The <a shape="rect" href="https://developer.mozilla.org/en-US/docs/Web/SVG/Element/svg">Mozilla Web docs</a> is a resouce that can tell us what attributes we can set on certain SVG elements are great to have as a reference when working with SVG.
We defined the <code>height</code> and <code>width</code> values in the first two lines
of the code example above because we’ll be using them throughout
the code example.</p><p>In JavaScript <code>append-svg</code> would look like this:</p><pre><code class="language-javascript">function append_svg() {
return d3.select("#slopegraph").append("svg").attr("height", height).attr("width", width);
}
</code></pre><p>Try calling <code>(append-svg)</code> from the Figwheel REPL running in your terminal.</p><pre><code class="language-bash">(us-energy-slopegraph.core/append-svg)
</code></pre><p>Now inspect the page in your browser (F12).
You should see an SVG element in your HTML now!</p><pre><code class="language-html"><div id="slopegraph">
<svg height="450" width="540">
</svg>
</div>
</code></pre><p>Now call <code>append-svg</code> from the <code>main</code> function so that you have
an SVG element to work with when the page loads.</p><pre><code class="language-clojure">(defn ^:export main []
(append-svg))
</code></pre><p>Save the changes, and inspect the HTML on <code>index.html</code> again.
Do you notice anything unusual?</p><p class="flex justify-center"><img src="/img/blog/20180426-duplicate-svg.png" alt="" /></p><p>There are two SVG elements!
When we save our changes, <code>on-js-reload</code> gets called and calls <code>main</code>,
which calls <code>append-svg</code>.</p><p>To solve this issue for now, let’s remove the SVG element when <code>on-js-reload</code>
gets called.
Add the <code>remove-svg</code> function to <code>core.cljs</code>, and modify <code>on-js-reload</code>
so we call <code>remove-svg</code> before <code>main</code> is called.</p><pre><code class="language-clojure">(defn remove-svg []
(-> js/d3
(.selectAll "#slopegraph svg")
(.remove)))
(defn on-js-reload []
(remove-svg)
(main))
</code></pre><p>As soon as you save this change <code>core.cljs</code> will reload, Figwheel will invoke
<code>on-js-reload</code>, which then removes any leftover <code><svg></code> elements, before adding
a new one. No more duplicate SVG elements!</p><p>You’ll notice I used the <a shape="rect" href="https://clojure.org/guides/threading_macros">thread-first macro
(->)</a> in <code>append-svg</code> and
<code>remove-svg</code> when chaining calls to D3 methods. The thread-first macro takes a
starting value and a number of function calls. The starting value is used as the
first argument for the first function, then the result is used as first argument
for the next function, and so forth. This helps us “chain” the D3 selection
methods together. Without it <code>remove-svg</code> would look like this:</p><pre><code class="language-clojure">(defn remove-svg []
(.remove (.selectAll js/d3 "#slopegraph svg")))
</code></pre><p>Those periods in <code>.remove</code> and <code>.selectAll</code> indicate that we’re calling
JavaScript methods, rather than plain ClojureScript functions.</p><p>You may also want to check out the <code>..</code> operator.
It works similarly to thread-first, but you don’t have to prefix all those
JavaScript methods with a <code>.</code>.
The <a shape="rect" href="https://lambdaisland.com/episodes/clojurescript-interop">ClojureScript Interop episode</a> on Lambda Island
explains more about this syntax.</p><p>The benefit of using <code>-></code> over <code>..</code> is that it allows you to freely mix ClojureScript functions and JavaScript methods.</p><h3 id="the-data">The Data</h3><p>I used a separate script to transform the CSV data I downloaded from
<a shape="rect" href="https://www.eia.gov/electricity/data/browser/">U. S. Energy Information Administration</a>
into a ClojureScript hash map containing the percentages of a few energy sources generated (<em>Note: “Renewables” is the sum of biomass, geothermal, solar, and wind</em>).
This is the data we’ll be working with to draw our slopegraph.
Insert this hash map definition into <code>core.cljs</code>.</p><pre><code class="language-clojure">
(def data {2005 {:natural-gas 0.2008611514256557
:coal 0.48970650816857986
:nuclear 0.19367190804075465
:renewables 0.02374724819670379}
2015 {:natural-gas 0.33808321253456974
:coal 0.3039492492908485
:nuclear 0.1976276775179704
:renewables 0.08379872568702211}})
</code></pre><h2 id="drawing">Drawing!</h2><p>Let’s create a function called <code>draw-slopegraph</code> that takes a D3 selection called <code>svg</code> and the <code>data</code> hash map.
Update <code>main</code> to call <code>draw-slopegraph</code> with the svg element created by <code>append-svg</code> and the <code>data</code> hash map.
This will make sure that our visualization reflectes the changes we make in our code.</p><pre><code class="language-clojure">(defn draw-slopegraph [svg data])
,,,
(defn ^:export main []
(let [svg (append-svg)]
(draw-slopegraph svg data)))
</code></pre><p>Each column of our slope graph represents a year. The column on the left will display 2005 data, the column on the right will display 2015 data.
If you look back at the slopegraph examples in the <a shape="rect" href="https://www.npr.org/2015/09/10/319535020/coal-gas-nuclear-hydro-how-your-state-generates-power">NPR article</a>, you’ll notice that each text element is positioned vertically based on the associated numeric value.
The text element in this case is the hash map key (‘natural-gas’, ‘coal’, etc), and the numeric value is the hash map value (in percent).
We’ll use D3.js to position those names correctly for each column.</p><p>Create a function called <code>draw-column</code>.
<code>draw-column</code> will take a D3.js selection (where the text elements be drawn inside), a year’s worth of data, the column index, and any additional svg attributes.
<code>draw-column</code> will do the standard D3.js procedure for creating new elements from data:
create a placeholder selection, bind the data, call <code>enter</code>, create new elements (in this case, <a shape="rect" href="https://developer.mozilla.org/en-US/docs/Web/SVG/Element/text">SVG text elements</a>), then set attributes for those text elements.</p><pre><code class="language-clojure">(defn draw-column [svg data-col index custom-attrs]
(-> svg
(.selectAll (str "text.slopegraph-column-" index))
(.data (into-array data-col))
(.enter)
(.append "text")))
</code></pre><p>Notice the call to <code>into-array</code> when we pass our data to the <code>.data</code> call.
One of the things you’ll find when using JavaScript libraries from ClojureScript
is that sometimes you need to turn ClojureScript data structures into plain
JavaScript ones.
<a shape="rect" href="https://cljs.github.io/api/cljs.core/into-array">into-array</a> turns a ClojureScript sequence
type into a native JavaScript array.
I could have also used
<a shape="rect" href="https://cljs.github.io/api/cljs.core/clj-GTjs">clj->js</a>,
but <code>clj->js</code> recursively turns <em>all</em> ClojureScript values in <code>data</code> into native JavaScript types,
which means we can’t use ClojureScript methods on those JavaScript values.</p><p>Now modify <code>draw-slopegraph</code> to make sure we’re calling <code>draw-column</code> for each year of data.</p><pre><code class="language-clojure">(defn draw-slopegraph [svg data]
(let [data-2005 (get data 2005)
data-2015 (get data 2015)]
(draw-column svg data-2005 0 {"x" (* width 0.25)})
(draw-column svg data-2015 0 {"x" (* width 0.75)})))
</code></pre><p>We’re setting the <code>x</code> attribute for each column to align them horizontally.</p><p>Save your changes, open <code>index.html</code> and inspect the HTML. You won’t see anything on the page, but you’ll see that we just created text SVG elements for each item in <code>data-col</code>.</p><p class="flex justify-center"><img src="/img/blog/20180426-text-svg-elements.png" alt="" /></p><p>We need to set the <code>y</code> attribute of those <code>text</code> elements, which will set the vertical position.
We’re going to create a helper function that will return a <a shape="rect" href="https://github.com/d3/d3-scale">D3 scale function</a> which will help us translate those values into pixel values.</p><pre><code class="language-clojure">;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; helpers
(def height-scale
(-> js/d3
(.scaleLinear)
(.domain #js [0 1])
(.range #js [(- height 15) 0])))
</code></pre><p>For the display text itself, we want to display the name of the energy source and the percent value.
For example, <code>:natural-gas 0.2008611514256557</code> will be <code>Natural gas 20.08%</code>.
Some more helper functions should help us out with that.</p><p>Modify the require statement to include <code>clojure.string</code>.</p><pre><code class="language-clojure">(ns us-energy-slopegraph.core
(:require [cljsjs.d3]
[clojure.string :as str]))
</code></pre><p>Then add these two helpers:</p><pre><code class="language-clojure">(defn format-percent [value]
((.format js/d3 ".2%") value))
(defn format-name [name-str]
(-> name-str
(str/replace "-" " ")
(str/capitalize)))
</code></pre><p><code>format-percent</code> uses <a shape="rect" href="https://github.com/d3/d3-format">D3’s formatters</a> to convert the decimal number into a percent string.
<code>format-name</code> does some simple string manipulation to clean up the name string.
Feel free to try out these two functions using the REPL!</p><pre><code class="language-clojure">dev:cljs.user=> (us-energy-slopegraph.core/format-name "natural-gas")
"Natural gas"
dev:cljs.user=> (us-energy-slopegraph.core/format-percent 0.19293818293)
"19.29%"
</code></pre><p>Now let’s set the attributes in the <code>text</code> elements we created in <code>draw-column</code>.
Add the <code>attrs</code> function below and modify <code>draw-column</code>.</p><pre><code class="language-clojure">;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; helpers
(defn attrs [el m]
(doseq [[k v] m]
(.attr el k v)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; draw functions
(defn draw-column [svg data-col index custom-attrs]
(-> svg
(.selectAll (str "text.slopegraph-column-" index))
(.data (into-array data-col))
(.enter)
(.append "text")
(.classed (str "slopegraph-column" index) true)
(.text (fn [[k v]] (str (format-name (name k)) " " (format-percent v))))
(attrs (merge custom-attrs
{"y" (fn [[_ v]] (height-scale v))}))))
</code></pre><p>The <code>attrs</code> function is a helper function that helps us pass in SVG attributes to a selection using a whole hash map,
rather than calling <code>attr</code> for every attribute we want to set without needing to install <a shape="rect" href="https://github.com/d3/d3-selection-multi">an external library</a>.</p><p>Save your changes and look at <code>index.html</code> again.
You should now see text!</p><p class="flex justify-center"><img src="/img/blog/20180426-slopegraph-text.png" alt="" /></p><p>We’re still missing the line and the header, but let’s first refactor the code to make drawing those easier. The data join pattern in D3.js is going to be used again and again, so let’s pull out some of that code into a separate function.</p><pre><code class="language-clojure">;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; helpers
(defn data-join [parent tag class data]
(let [join (-> parent
(.selectAll (str tag "." class))
(.data (into-array data)))
enter (-> join
(.enter)
(.append tag)
(.classed class true))]
(-> join (.exit) (.remove))
(.merge join enter)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; draw functions
(defn draw-column [svg data-col index custom-attrs]
(-> svg
(data-join "text" (str "text.slopegraph-column-" index) data-col)
(.text (fn [[k v]] (str (format-name (name k)) " " (format-percent v))))
(attrs (assoc custom-attrs
"y" (fn [[_ v]] (height-scale v))))))
</code></pre><p>With that <code>data-join</code> in place we’re now ready to implement <code>draw-header</code> and <code>draw-line</code></p><pre><code class="language-clojure">(def column-1-start (/ width 4))
(def column-space (* 3 (/ width 4)))
(defn draw-header [svg years]
(-> svg
(data-join "text" "slopegraph-header" years)
(.text (fn [data _] (str data)))
(attrs {"x" (fn [_ index]
(+ 50 (* index column-space)))
"y" 15})))
(defn draw-line [svg data-col-1 data-col-2]
(-> svg
(data-join "line" "slopegraph-line" data-col-1)
(attrs {"x1" (+ 5 column-1-start)
"x2" (- column-space 5)
"y1" (fn [[_ v]]
(height-scale v))
"y2" (fn [[k _]]
(height-scale (get data-col-2 k)))})))
</code></pre><p>And call them from <code>draw-slopegraph</code></p><pre><code class="language-clojure">(defn draw-slopegraph [svg data]
(let [data-2005 (get data 2005)
data-2015 (get data 2015)]
(draw-column svg data-2005 1 {"x" column-1-start})
(draw-column svg data-2015 2 {"x" column-space})
(draw-header svg [2005 2015])
(draw-line svg data-2005 data-2015)))
</code></pre><p>Again I’m not going to go into the details of the D3 API, I hope that intuitively you can get a sense of what is going on. The line connecting the two columns together uses a <a shape="rect" href="https://github.com/d3/d3-shape#lines">d3 line generator</a> which works a little differently than the text elements we drew.</p><p>There are a lot of things we can do to make this visualization more readable.
Maybe we can attach mouse handlers so we can highlight one column at a time, like
this <a shape="rect" href="https://bl.ocks.org/eesur/ee8d0ab88229d0e07f7f">slopegraph example here</a>?
You may want to
<a shape="rect" href="https://github.com/lambdaisland/us-energy-slopegraph/">clone the finished project</a>
and try to add that feature yourself using ClojureScript!</p><h3 id="final-thoughts">Final thoughts</h3><p>Learning a new language can be an overwhelming task.
There’s the pain of learning new syntax and paradigms, then also build tools, editor tools, a decent number of conceptual talks, and a whole lot more.
Getting all these thrown in the face can feel like too much work to get started with a language.</p><p>Picking up figwheel meant that I could learn Clojure in a responsive environment
with a pretty small setup process.
I was able to apply my JavaScript knowledge to ClojureScript and start writing
small D3.js visualizations to learn the language.
From there, I slowly learned about better ways to write my code,
picked up new editor and REPL tools,
and continued my Clojure education.</p><p>If learning Clojure seems overwhelming, a
<a shape="rect" href="https://github.com/bhauman/lein-figwheel/wiki/Quick-Start">Figwheel</a>
project may be right for you!</p><h3 id="additional-clojure-resources">Additional Clojure Resources</h3><ul><li>If you want to dive straight into Clojure in a graphical environment with zero setup, I also highly suggest <a shape="rect" href="http://maria.cloud/">Maria.cloud</a>.</li><li><a shape="rect" href="https://github.com/bhauman/lein-figwheel">Figwheel</a></li></ul><h3 id="d3js-resources">D3.js resources</h3><ul><li><a shape="rect" href="https://bost.ocks.org/mike/join/">Thinking With Joins</a></li><li><a shape="rect" href="http://alignedleft.com/tutorials/d3">Interactive Data Visualization for the Web</a></li></ul></<>>Reloading Woeshttps://lambdaisland.com/blog/2018-02-09-reloading-woes2020-11-24T12:48:33+00:00<<>><p><em>Update: seems Stuart Sierra’s blog post has dropped off the internet. I’ve updated the link to refer to the Wayback Machine’s version instead.</em></p><h2 id="setting-the-stage">Setting the Stage</h2><p>When doing client work I put a lot of emphasis on tooling and workflow. By
coaching people on their workflow, and by making sure the tooling is there to
support it, a team can become many times more effective and productive.</p><p>An important part of that is having a good story for code reloading. Real world
projects tend to have many dependencies and a large amount of code, making them
slow to boot up, so we want to avoid having to restart the process.</p><p>Restarting the process is done when things get into a bad state. This can be
application state (where did my database connection go?), or Clojure state (I’m redefining this middleware function, but the webserver isn’t picking it up.)</p><h3 id="toolsnamespace-and-componentintegrant">tools.namespace and component/integrant</h3><p>We can avoid the cataclysm of a restart if we can find some other way to get to a clean slate. To refresh Clojure’s view of the world, there’s <a shape="rect" href="https://github.com/clojure/tools.namespace">clojure.tools.namespace</a>, which is able to <em>unload</em> and then <em>reload</em> namespaces.</p><p>To recreate the app state from scratch there’s the “reloaded workflow” as <a shape="rect" href="https://web.archive.org/web/20160315031517/http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded">first described by Stuart Sierra</a>, who created <a shape="rect" href="https://github.com/stuartsierra/component">Component</a> for this reason.</p><p>Combining these two allows you to tear down the app state, reload all your Clojure code, then recreate the app state, this time based on the freshly loaded code. Combining them like this is important because when you use tools.namespace to reload your code, it effectively creates completely new and separate versions of functions, records, multimethods, vars, even namespace objects, and any state that is left around from before the reload might still be referencing the old ones, making matters even worse than before.</p><p>So you need to combine the two, which (naively) would look something like this.</p><pre><code class="language-clojure">(ns user
(:require [com.stuartsierra.component :as component]
[clojure.tools.namespace.repl :as c.t.n.r]))
(def app-state nil)
(defn new-system []
(component/system-map ,,,))
(defn go []
(alter-var-root! app-state (fn [_]
(-> (new-system)
component/start))))
(defn reset
"Stop the app, reload all changed namespaces, start the app again."
[]
(component/stop app-state)
;; Clojure tools namespace takes a symbol, rather than a function, so that it
;; can resolve the function *after code reloading*. That way it's sure to get
;; the latest version.
(c.t.n.r/refresh :after 'user/go))
</code></pre><p>This pattern is encoded in <a shape="rect" href="https://github.com/weavejester/reloaded.repl">reloaded.repl</a>; <a shape="rect" href="https://github.com/danielsz/system">System</a> also contains an implementation. I recommend using an existing implementation over rolling your own, as there are some details to get right. It’s also interesting to note that reloaded.repl uses <a shape="rect" href="https://github.com/weavejester/suspendable">suspendable</a>, an extension to Component that adds a suspend/resume operation.</p><div class="ba b--black-20 pa3 w5 fr ml3 mv3"><p>To dig deeper into this topic, check out the Lambda Island episodes on <a shape="rect" href="https://lambdaisland.com/collections/clojure-component-system">Component and System</a>.</p><p><a shape="rect" href="https://lambdaisland.com/collections/clojure-component-system"><img src="https://i.vimeocdn.com/video/637685834_295x166.jpg?r=pad" alt="" /></a></p></div><p>The author of reloaded.repl, James Reeves, eventually became dissatisfied with Component, and created <a shape="rect" href="https://github.com/weavejester/integrant">Integrant</a> as an alternative, with <a shape="rect" href="https://github.com/weavejester/integrant-repl">integrant-repl</a> as the counterpart of reloaded.repl.</p><p>With reloaded.repl your own utility code now looks like this:</p><pre><code class="language-clojure">(ns user
(:require reloaded.repl
com.stuartsierra.component))
(defn new-system []
(com.stuartsierra.component/system-map ,,,))
(reloaded.repl/set-init! new-system)
</code></pre><p>Now you start the system with <code>(reloaded.repl/go)</code>. To stop the system, reload all changed namespaces, and start the system again, you do <code>(reloaded.repl/reset)</code>.</p><p>With integrant-repl things look similar.</p><pre><code class="language-clojure">(ns user
(:require integrant.repl
clojure.edn))
(defn system-config []
(-> "system_config.edn" slurp clojure.edn/read-string))
(integrant.repl/set-prep! system-config)
</code></pre><p>Now <code>(integrant.repl/go)</code> will bring your system up, and with <code>(integrant.repl/reset)</code> you can get back to a clean slate.</p><h3 id="cider-refresh">cider-refresh</h3><p>CIDER (Emacs’s Clojure integration) supports tools.namespace through the <code>cider-refresh</code> command. This only does code reloading though, but you can tell it to run a function before and after the refresh, to achieve the same effect.</p><pre><code class="language-clojure">;; emacs config
;; reloaded.repl
(setq cider-refresh-before-fn "reloaded.repl/suspend")
(setq cider-refresh-after-fn "reloaded.repl/resume")
;; integrant-repl
(setq cider-refresh-before-fn "integrant.repl/suspend")
(setq cider-refresh-after-fn "integrant.repl/resume")
</code></pre><p>You can also configure this on a per-project basis, by creating a file called <code>.dir-locals.el</code> in the project root, which looks like this.</p><pre><code>((nil . ((cider-refresh-before-fn . "integrant.repl/suspend")
(cider-refresh-after-fn . "integrant.repl/resume"))))
</code></pre><p>Now calling <code>cider-refresh</code> (Spacemacs: <code>, s x</code>) will again stop the system, reload all namespaces, and restart.</p><h2 id="plot-twist">Plot twist</h2><p>So far so good, this was a lengthy introduction, but I wanted to make sure we’re all on the same page, now let’s look at some of the things that might spoil our lovely reloading fun.</p><h3 id="aot">AOT</h3><p>Clojure has a nifty feature called AOT or “ahead of time compilation”. This causes Clojure to pre-compile namespaces to Java classes and save those out to disk. This can be a useful feature as part of a deployment pipeline, because it can speed up booting your app. It has some serious drawbacks though, as it messes with Clojure’s dynamic nature.</p><p>What tends to happen is that at some point people find out about AOT, they think it’s amazing, and enable it everywhere. A bit later errors start popping up during development that just make no sense.</p><p>AOT should only be used for deployed applications. Don’t use it during development, and don’t use it for libraries you publish.</p><p>Even when following this advice you can get into trouble. Say you have aot enabled as part of the process of building an uberjar for deployment.</p><pre><code class="language-clojure">(defproject my-project "0.1.0"
,,,
:profiles {:uberjar
{:aot true}})
</code></pre><p>You do a <code>lein uberjar</code> to test your build locally. This will create AOT compiled classes and put them on the classpath (under <code>target/</code> to be precise). Next time you try to <code>(reset)</code> it will tell you it can’t find certain Clojure source files, even though they’re <em>right there</em>. I have lost hours of my life figuring this out, and have more than once found my <a shape="rect" href="https://stackoverflow.com/questions/44246924/clojure-tools-namespace-refresh-fails-with-no-namespace-foo">own Stackoverflow question+answer</a> when googling for this. Watch out for the ghost of AOT!</p><div class="ba b--black-30 pa3"><p>By the way, here’s a handy oneliner: <code>git clean -xfd</code>. It removes any files not tracked by git, including files in <code>.gitignore</code>. It’s the most thorough way of cleaning out a repository. Do watch out, this might delete files you still want to keep! With <code>git clean -nxfd</code> you can do a dry run to see what it plans to delete.</p></div><h3 id="defonce-wont-save-you"><code>defonce</code> won’t save you</h3><p>Ideally all your state is inside your system, but maybe there’s something else that you want to carry over between reloads. No need to judge, life is compromise.</p><p>You might think, “I know, I’ll just <code>defonce</code> it!”, but <code>defonce</code> won’t save you. tools.namespace will completely remove namespaces with everything in them before loading them again, so that var created by <code>defonce</code> the first time is long gone by the time <code>defonce</code> gets called again, and so it happily defines it anew.</p><p>What you can do instead is add some namespace metadata telling tools.namespace not to unload the namespace.</p><pre><code class="language-clojure">(ns my-ns
(:require [clojure.tools.namespace.repl :as c.t.n.r]))
(c.t.n.r/disable-unload!) ;; this adds :clojure.tools.namespace.repl/unload false
;; to the namespace metadata
(defonce only-once (Date.))
</code></pre><p>This namespace will still get reloaded, so if you have functions in there then you’ll get the updated definitions, but it won’t be unloaded first, so <code>defonce</code> will work as expected. This does also mean that if you remove (or rename) a function, the old one is still around.</p><p>To completely opt-out of reloading, use <code>(c.t.n.r/disable-reload!)</code>. This implies <code>disable-unload!</code>.</p><h3 id="cljc--clojurescript-source-maps">cljc + ClojureScript source maps</h3><p>When ClojureScript generates source maps, it needs a way to point the browser at the original source files, so it copies <code>cljs</code> and <code>cljc</code> files into your web server document root. (your “public/” directory). Seems innocent enough, but it can confuse tools.namespace.</p><p>It is quite a common practice to search for files that are requested over HTTP in the classpath, so the “public/” directory tends to be on the classpath as well. tools.namespace scans the classpath for Clojure files (<code>clj</code> or <code>cljc</code>) and finds those <code>cljc</code> files that ClojureScript copied there, but their namespace name doesn’t correspond with their location, and things breaks. There is a JIRA ticket for this for tools.namespace: <a shape="rect" href="https://dev.clojure.org/jira/browse/TNS-45">TNS-45</a>, with several proposed patches, but no consensus yet about the right way forward.</p><p>The easiest way around this is to limit tools.namespace to only scan your own source directories.</p><pre><code class="language-clojure">(c.t.n.r/set-refresh-dirs "src/clj")
</code></pre><h3 id="cider-does-its-own-thing">CIDER does its own thing</h3><p><code>cider-refresh</code> roughly does the same thing as <code>clojure.tools.namespace.repl/refresh</code>, and it uses tools.namespace under the hood, but it reimplements <code>refresh</code> using lower level tools.namespace primitives. This leads to subtle differences.</p><p>Currently <code>cider-refresh</code> does not honor <code>set-refresh-dirs</code>. It honors the unload/load metadata, but does not follow the convention that <code>:clojure.tools.namespace.repl/load false</code> implies <code>:clojure.tools.namespace.repl/unload false</code>.</p><p>I proposed <a shape="rect" href="https://github.com/clojure-emacs/cider-nrepl/pull/503">a patch</a> to address both of these issues. In general if <code>cider-refresh</code> seems to have issues, then try to use <code>(reset)</code> directly in the REPL.</p></<>>The Bare Minimum, or Making Mayonnaise with Clojurehttps://lambdaisland.com/blog/2017-12-29-the-bare-minimum-clojure-mayonnaise2017-12-29T14:54:49+00:00<<>><h2 id="making-mayonnaise">Making Mayonnaise</h2><p>Imagine you have a grandfather who’s great at making mayonnaise. He’s been
making mayonnaise since before the war, and the result is truly excellent.
What’s more, he does this with a good old fashioned whisk. He’s kept his right
arm in shape throughout decades just by beating those eggs and oil and vinegar.</p><p>Now he’s bought himself a handheld electric mixer after hearing his friend rave
about hers, but after a few tries he gives up and goes back to his whisk. He
says he just can’t get the same result. This seems slightly odd, so the next
time you go over you ask him to show you how he uses the mixer.</p><p>Imagine your face when he takes one of the mixer’s beaters and starts beating
the stuff by hand. Turns out he actually had no idea how to use the mixer, so he
just took the bit that seemed familiar, and started using it the way he was used
to.</p><h2 id="what-does-this-have-to-do-with-clojure">What does this have to do with Clojure?</h2><p>A far fetched story perhaps, but it’s the best metaphor I could come up with to
illustrate the difference between “writing Clojure”, and “doing Clojure”.</p><p>Sometimes people approach Clojure in a similar way. They’ve heard good things
about it, and decide to try it out, while sticking to their familiar tools and
workflow. They hear Clojurists talk about their Emacs and other exotica, and
write it off as a strange fetish for obscure technology. They know coding. They
already have an editor they know and love, thank you very much. All they want to
know is how this code in <code>$LANGUAGE</code> translates to Clojure.</p><p>I made this same mistake some years ago when I decided to give SmallTalk a try.
I looked at a couple of implementations, but they all seemed to come with this
massive environment, including a UI, editor, and something called “images” that
just did not make sense. All I wanted was to learn a language, all I needed (I
thought), was a way to stick some code into a source file, compile and run it.</p><p>But SmallTalk, like Clojure, isn’t just a language. It’s a complete philosophy
about creating software, with specific ideas about how programmers will interact
with their environment. This makes it strange and unfamiliar, but if you really
want to learn one of these languages you need to dive in wholesale. You will
never be able to assess and appreciate what makes them stand apart by skirting
around the edges of your comfort zone.</p><p>The sad part is that this stuff isn’t made very explicit. It’s the kind of
knowledge that’s passed down from mentor to mentee, taught by example. If you’re
learning Clojure all on your own, you may never realize what you’re missing.</p><p>So I’m here to tell you what the bare minimum setup is you need to not just be
“writing Clojure”, but to actually be “doing Clojure”.</p><p>In short, there are two things that you simply can not do without. I’ll call
these “managed parentheses”, and “in-place evaluation”. Take note that I did not
say “use Emacs” (or Vim or ed(1) or whatever). Many editors and IDE’s can
provide these features, but you do have to make sure your environment has these
features, and you have to learn how to use them.</p><h2 id="managed-parentheses">Managed Parentheses</h2><p>Managed parentheses means that your editor makes it near impossible to have
unbalanced parentheses in your code. You should never ever have to manually
insert a closing parenthesis.</p><p>There are several ways to achieve this. The most well known approach is called
“paredit”, originally built for Emacs, but with equivalent packages for most
major editors. A different approach that’s been gaining popularity, and with
less of a learning curve is Parinfer. The general term for this is “structural
editing”.</p><p>You can find eplanations of how these work in detail elsewhere. Just take it
from me that they manage your parentheses, so you don’t have to. What they
really do is cause a mental shift, where you <em>no longer think about
parentheses</em>, but instead think about blocks of code (also called forms,
expressions, or sexps).</p><p>Clojure is a LISP. This in itself is a superpower. For decades people have been
saying that treating programs as text is backwards, that we need higher level
tools that allow us to manipulate the structure of the program directly. LISP
drags you halfway there. It takes getting used to, but it pays off massively.</p><p>Don’t just take it from me, this is what user <a shape="rect" href="https://clojureverse.org/t/share-the-nitty-gritty-details-of-your-clojure-workflow/1208/17?u=plexus">bbrinck had to say on ClojureVerse</a>:</p><blockquote><p>I have found that many beginners don’t realize that they will be much, much more productive once they learn paredit or parinfer. I worked in Clojure for a few years in my spare time without structural editing. I knew paredit existed, but I didn’t have a good sense of the time investment necessary vs the productivity difference.</p><p>It was only when I pair programmed with some experienced Clojurists that I realized how much time I was wasting on bugs related to parentheses that they never experienced.</p></blockquote><h2 id="in-place-evaluation">In place evaluation</h2><p>Most languages have REPLs, command line tools that let you enter and evaluate
program code one line at a time. They’re great for exploration, for quickly
trying things out, or for running commands on the fly.</p><p>In the LISP family of languages (as well as SmallTalk) this concept is taken one
step further by integrating this REPL deeply into the editor experience. This
allows you to put your cursor next to a piece of code in a source file, and
evaluate it right there.</p><p>Because of LISP’s uniform structure you can unambigously single out any single
sub-expression, and send that to the editor-connected-REPL, immediately seeing
the result. This brings coding to a new level of interactivity. You are interacting
with a full, running system, which you can inspect, update, and alter. You can
even connect to your production system, to look at its state, or to quickly
redefine a function to patch a bug.</p><p>You can also evaluate a piece of code, and <em>insert the result back into your
source file</em>. People often say that in Clojure code is data, focusing on the
manipulation of code structure with things like macros, but the equivalence goes
both ways. Data is code, so any Clojure data structure, when printed, is a valid
Clojure expression. Depending on where you’re coming from this will be either a
no-brainer or a mind bender. In any case it’s extremely powerful. I said it
before, LISP is a superpower.</p><h2 id="what-else">What else?</h2><p>These two are the absolute minimum. If you honestly want to give Clojure a try,
then make sure you’re using an environment that has these features, and learn
how to use them. Some popular options include</p><ul><li>CIDER (Emacs)</li><li>Cursive (IntelliJ)</li><li>ProtoREPL (Atom)</li><li>fireplace.vim</li></ul><p>There are others. The point is to not blindly adopt what someone tells you, but
to be able to assess yourself whether the tool you’re using clears this bar.</p><p>These two things are only the absolute bare minimum. For a real world setup I
would recommend a few more things. For all of these</p><ul><li>check that your environment can do it</li><li>learn how to use it</li><li>make it part of your workflow</li></ul><h3 id="automatic-indentation">Automatic indentation</h3><p>LISP, when properly indented, looks a lot like Python. There’s general concensus
about how to indent Clojure, and so it’s something that easily can, and should
be automated.</p><p>Either have a way to automatically correct the indentation of a line (e.g. by
pressing TAB), or use something that automatically keeps the indentation of the
whole file up to date (I recommend the latter, for Emacs this is called
“agressive-indent-mode”).</p><h3 id="jump-to-definition">Jump to definition</h3><p>Before coming to Clojure I did a lot of Ruby. In Ruby when you’re looking at a
method invocation it can be hard to figure out where that implementation lives,
because it depends on runtime types. Not so in Clojure, the system always knows
exactly where a given function is defined. Your editor can ask its integrated
REPL where that is, so you can jump to the right file and line.</p><p>This is great for navigating your own project, as well for jumping into library
code to better understand how it works. And while you’re there you could put a
couple <code>println</code> statements in that library code, and re-evaluate it. Never
second guess again what values are flowing through your system.</p><h3 id="show-function-signature">Show function signature</h3><p>When your cursor is on a function name, your editor should show somewhere
unintrusively what the signature of that function looks like. This little bit of
information, the arity and argument names of a function, are 80% of the
documentation you’ll need. Having it right there will help you stay in a state
of flow.</p><h3 id="look-up-documentation">Look up documentation</h3><p>Functions in Clojure come with built-in documentation. All of core has these
docs, and most libraries do as well. You really want to be able to pull these up
with the touch of a button.</p><h3 id="rainbow-parentheses">Rainbow parentheses</h3><p>More of a nice to have, but worth a mention. This feature gives matching
parentheses matching colors. I really like this, but it’s probably not
essential.</p><h2 id="staying-in-the-flow">Staying in the flow</h2><p>If you have all of the above you’re already ahead of most programmers in the
industry when it comes to tooling and workflow. Consider yourself lucky. The
next step is to make sure you can stay in the flow as much as possible, to
eradicate events that upset and interrupt your REPL based workflow.</p><h3 id="reloadable-system-state">Reloadable system state</h3><p>A downside of dynamically working with a long running process is that eventually
state will accrue, so that the in-memory state of your Clojure process no longer
reflects the source files on disks. There could be functions that you’ve deleted
or renamed, but that still exist in memory. You could have application state
that no longer reflects a valid state of the program. There are a bunch of
gotchas in Clojure that can cause these errors, for instance redefining a
<code>defrecord</code>, or a <code>defmulti</code>.</p><p>This can cause code to work fine, but fail as soon as you restart the process,
or it can cause things to fail even though they really should work. In both
cases what you need is a clean slate.</p><p>Initially you’ll just take this in your stride and restart the REPL process.
Once you have felt enough of this pain though you’ll look for a solution. What
you’ll end up with will likely be a component-style library to manage your
application state, something like Component, Mount, or Integrant, combined with
clojure.tools.namespace for intelligent unloading and reloading of all your
namespaces.</p><h3 id="hot-loading-of-new-dependencies">Hot loading of new dependencies</h3><p>When developing applications, especially greenfield apps, you’ll regularly be
adding more third party libraries as you go. Each time you add one of these to
<code>project.clj</code>, <code>deps.edn</code>, or <code>build.boot</code>, you need to restart your Clojure
process.</p><p>This can be solved through libraries like Pomegranate, which will install and
hot-load new dependencies. Configure them in <code>~/.lein/profiles.clj</code> or
<code>~/.clojure/deps.edn</code> so you have them available on every project.</p><h2 id="further-reading">Further reading</h2><ul><li><a shape="rect" href="https://clojureverse.org/t/share-the-nitty-gritty-details-of-your-clojure-workflow/1208">Share the nitty gritty details of your Clojure workflow (ClojureVerse, December 2017)</a></li><li><a shape="rect" href="http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded">My Clojure Workflow, reloaded (Stuart Sierra, June 2013)</a></li><li><a shape="rect" href="https://vimeo.com/223309989">(Video) REPL Drive Development (Stuart Haloway, June 2017)</a></li></ul></<>>Clojure Gotchas: "contains?" and Associative Collectionshttps://lambdaisland.com/blog/2017-10-02-clojure-gotchas-associative-contains2017-10-02T16:57:57+00:00<<>><p>When learning a programming language we rarely read the reference documentation front to back. Instead someone might follow some tutorials, and look at sample code, until they’re confident enough to start a little project for practice.</p><p>From that point on the learning process is largely “just in time”, looking up exactly the things you need to perform the task at hand. As this goes on you might start to recognize some patterns, some internal logic that allows you to “intuit” how one part of the language works, based on experience with another part.</p><p>Developing this “intuition” — understanding this internal logic — is key to using a language effectively, but occasionally our intuition will be off. Some things are simply not obvious, unless someone has explained them to us. In this post I will look at something that frequently trips people up, and attempt to explain the underlying reasoning.</p><h2 id="contains"><code>contains?</code></h2><p>Let’s start with perhaps the greatest source of frustration in <code>clojure.core</code>, the <code>contains?</code> function. Based on its name you’d be excused for thinking that it checks whether a collection contains a certain value.</p><pre><code class="language-clojure">(def beatles ["John" "Paul" "Ringo" "George"])
(contains? beatles "Ringo")
;;=> false
</code></pre><p>Ahum… what the h*ck, Clojure?</p><p>A symptom of the same underlying cause is confusion over <code>get</code> vs <code>nth</code>. Why do we need both? When you do you use which?</p><pre><code class="language-clojure">(get beatles 1) ;;=> "Paul"
(nth beatles 1) ;;=> "Paul"
</code></pre><h2 id="collection-traits">Collection Traits</h2><p>To understand what’s going on we need to look at the <em>underlying abstractions</em> to Clojure’s data types. All of Clojure’s concrete types like vectors, sets, or maps implement one or more of these “abstractions”. You can also think of them as “having certain traits”.</p><ul><li>Seq(uential)</li><li>Seqable</li><li>Associative</li><li>Counted</li><li>Indexed</li><li>Sorted</li><li>Reversible</li></ul><p>Functions in <code>clojure.core</code> do their work based on these abstractions. For instance, you can call <code>first</code> on any Clojure collection because they all implement Seq or Seqable, whereas <code>get</code> or <code>update</code> will only work on collections that implement Associative.</p><p>This <a shape="rect" href="http://insideclojure.org/2016/03/16/collections/">blog post by Alex Miller</a> explains these traits in great detail.</p><p>Of these Seq and Seqable are arguable the most important to understand, closely followed by Associative. There’s a <a shape="rect" href="https://lambdaisland.com/episodes/clojure-seq-seqable">free Lambda Island episode</a> with everything you need to know about seqs, which is a trait shared by all Clojure collections, as well as some other things like strings or Java arrays.</p><pre><code class="language-clojure">(seqable? '(1 2)) ;;=> true
(seqable? [1 2]) ;;=> true
(seqable? {1 2}) ;;=> true
(seqable? #{1 2}) ;;=> true
(seqable? "12") ;;=> true
</code></pre><p>Functions like <code>first</code>, <code>rest</code>, <code>map</code>, or <code>cons</code> can operate on any seq/seqable. That’s also why the return type of <code>map</code> or <code>cons</code> may be different from what you passed it. The only guarantee is that the result is again a seq.</p><p>Maps and vectors are associative. You can think of the first as a mapping from keys to values, and the second as a mapping from indexes to values. <code>get</code>, <code>assoc</code>, <code>dissoc</code>, <code>update</code>, and any of their <code>*-in</code> variants work on associative data structures.</p><pre><code class="language-clojure">(associative? '(1 2)) ;;=> false
(associative? [1 2]) ;;=> true
(associative? {1 2}) ;;=> true
(associative? #{1 2}) ;;=> false
(associative? "12") ;;=> false
</code></pre><pre><code class="language-clojure">(get {1 2} 1) ;;=> 2
(get [1 2] 0) ;;=> 1
(update-in {:x {:y [0 0 0]}} [:x :y 1] inc)
;;=> {:x {:y [0 1 0]}}
</code></pre><p><code>contains?</code> works on Associative collections, where it checks if the collection contains a mapping for the given input. For maps that means it will check if a given key is there, and for vectors it returns true if the <em>index</em> exists.</p><pre><code class="language-clojure">(contains? {:a :b} :a) ;;=> true
(contains? {:a :b} :c) ;;=> false
(contains? [:a :b] 0) ;;=> true
(contains? [:a :b] 1) ;;=> true
(contains? [:a :b] 2) ;;=> false
</code></pre><h2 id="sets-as-funny-maps">Sets as funny maps</h2><p>Sets are a bit of a special case. While sets are not associative, several of the associative operations, including <code>contains?</code>, will also work on sets.</p><p>Suppose that Clojure didn’t have sets, in that case you could make do by using maps, with each value identical to its key. It turns out that for many operations that is exactly how sets behave. Whenever I’m not sure of whether a set would work in a given context, I try to imagine what would happen if I used a map instead.</p><pre><code class="language-clojure">(def real-set #{:a :b})
(def pseudo-set {:a :a, :b :b})
(real-set :a) ;;=> :a
(pseudo-set :a) ;;=> :a
(get real-set :a) ;;=> :a
(get pseudo-set :a) ;;=> :a
(contains? real-set :a) ;;=> true
(contains? pseudo-set :a) ;;=> true
</code></pre><p>Now that you know exactly how <code>contains?</code> works, you can attempt to make sense of its docstring.</p><pre><code class="language-clojure">cljs.user=> (doc contains?)
-------------------------
cljs.core/contains?
([coll v])
Returns true if key is present in the given collection, otherwise
returns false. Note that for numerically indexed collections like
vectors and arrays, this tests if the numeric key is within the
range of indexes. 'contains?' operates constant or logarithmic time;
it will not perform a linear search for a value. See also 'some'.
nil
</code></pre><p>So to recap, for maps <code>contains?</code> checks for the presence of a key, for sets it does what you would expect, since keys=values, and for vectors it checks whether the <em>index</em> is present.</p><pre><code class="language-clojure">(contains? #{:a :b} :a) ;;=> true
(contains? #{:a :b} :c) ;;=> false
(contains? {:a :b} :a) ;;=> true
(contains? {:a :b} :b) ;;=> false
(contains? [1 2] 1) ;;=> true
(contains? [1 2] 2) ;;=> false
</code></pre><p>If you do want to search a collection that is “only” a seq, the common idiom is to use <code>some</code> + a set literal. This returns the matched value, which is truthy.</p><pre><code class="language-clojure">(def beatles ["John" "Paul" "Ringo" "George"])
(some #{"Ringo"} beatles)
;;=> "Ringo"
</code></pre><p>An important caveat: if the value you’re looking for could be <code>nil</code> or <code>false</code>, then the set literal won’t work, and you need a more explicit check.</p><pre><code class="language-clojure">(def haystack [false])
(some #{false} haystack)
;;=> nil
(some #(= false %) haystack)
;;=> true
</code></pre><p>The difference between <code>get</code> and <code>nth</code> is now hopefully also clear. While <code>get</code> does a lookup in an associative collection, <code>nth</code> will find the nth element in a sequential collection. If the collection happens to also be indexed, then <code>nth</code> works in constant time, otherwise it will take linear time to walk the sequence.</p><pre><code class="language-clojure">;; Vectors: they're roughly the same
(get [:a :b :c] 1) ;;=> :b
(nth [:a :b :c] 1) ;;=> :b
;; Lists: only nth makes sense
(get '(:a :b :c) 1) ;;=> nil
(nth '(:a :b :c) 1) ;;=> :b
;; Sets: only get makes sense
(get #{:a :b :c} :a) ;;=> :a
(nth #{:a :b :c} 1) ;;=> ! Exception
;; Maps: only get makes sense
(get {:a :b} :a) ;;=> :b
(nth {:a :b} 1) ;;=> ! Exception
</code></pre><p>This diagram from a <a shape="rect" href="https://www.slideshare.net/alexmiller/clojure-the-art-of-abstraction-7161663">presentation by Alex Miller</a> gives you an overview of which functions work on which abstractions.</p><p><img src="https://devblog.arnebrasseur.net/images/collection_traits.png" alt="Collection Traits" /></p><p>This may seem overly complicated, but in practice most of these are pretty intuitive, so you don’t need to hang this diagram over your bed to study it every evening. Some functions “color outside the lines”, like <code>contains?</code> working on Associative collections, but also sets, or <code>nth</code> really being more of an Indexed operation, but also accepting seqs. In practice this isn’t as confusing as it may seem. The main cases that do trip people up I’ve tried to explain above.</p><p><em>Many thanks to <a shape="rect" href="http://www.daiyi.co/">Daiyi</a> for providing feedback on an earlier draft.</em></p><p><em><a shape="rect" href="https://www.maria.cloud/gist/ecb5c59087477ec5c4fb92f1c2f68e42?eval=true">Try out an interactive version of this post in Maria</a></em></p><p><em><a shape="rect" href="https://www.reddit.com/r/Clojure/comments/73tpoe/clojure_gotchas_contains_and_associative/">🡑 Discuss/vote for this post on Reddit</a></em></p></<>>Dates in Clojure: Making Sense of the Messhttps://lambdaisland.com/blog/2017-07-26-dates-in-clojure-making-sense-of-the-mess2019-05-29T06:52:36+00:00<<>><p><em>Update 2018-11-27: while most of this article is still relevant, I no longer recommend using JodaTime as the main date/time representation for new projects. Even existing projects that aren’t too invested in JodaTime/clj-time should consider migrating to <code>java.time</code> and <code>clojure.java-time</code> across the board.</em></p><p><em>Update 2 2019-05-29: Also check out the talk <a shape="rect" href="https://www.youtube.com/watch?v=UFuL-ZDoB2U">Cross Platform DateTime Awesomeness</a> by Henry Widd, given at Clojure/north 2019</em></p><p>You can always count on human culture to make programming messy. To find out if
a person is a programmer just have them say “encodings” or “timezones” and watch
their face.</p><p>In Java, and hence Clojure, there are half a dozen different ways to represent
dates and times, which can lead to confusion, needless type coercion, and
inconsistent code. In this post I’ll give you a quick lay of the land, and some
tips to make it all a bit easier to deal with.</p><h2 id="representing-time">Representing Time</h2><h3 id="unix-time">Unix Time</h3><p>The traditional way to represent time, the system that has been handed down to
us by The Elders, is known
as <a shape="rect" href="https://en.wikipedia.org/wiki/Unix_time">Unix Time</a>. A Unix timestamp is a
number representing the amount of seconds passed since
the <a shape="rect" href="https://en.wikipedia.org/wiki/Epoch_(reference_date)">start of time</a>, which
is conveniently defined as midnight, January 1, 1970.</p><p><img src="https://farm1.staticflickr.com/36/75450245_ddc16cbb18_z_d.jpg" alt="1970's family pic" /></p><p><em>A happy family shortly after the start of time <a shape="rect" href="https://www.flickr.com/photos/gregor_y/75450245">(credits)</a></em></p><h3 id="javautildate">java.util.Date</h3><p>Some 822 million seconds later Java 1.0 came out. It was the year of the Spice
Girls, the Macarena, and mutable date classes.</p><pre><code class="language-clojure">user> (def date (java.util.Date.))
#'user/date
user> date
#inst "2017-06-09T06:59:40.829-00:00"
user> (.setTime date 822391200000)
nil
user> date
#inst "1996-01-23T10:00:00.000-00:00"
user>
</code></pre><p>A <code>Date</code> wraps a milli-second based Unix timestamp, meaning it’s not time zone
aware.</p><pre><code class="language-clojure">user> (java.util.TimeZone/setDefault (java.util.TimeZone/getTimeZone "UTC"))
user> (.toString date)
"Tue Jan 23 10:00:00 UTC 1996"
user> (java.util.TimeZone/setDefault (java.util.TimeZone/getTimeZone "Europe/Berlin"))
user> (.toString date)
"Tue Jan 23 11:00:00 CET 1996"
</code></pre><p>That’s not all that’s wrong with <code>java.util.Date</code>, so in Java 1.1 they tried
to rectify things by deprecating half of the <code>Date</code> API and instead introducing
a (obviously mutable) <code>GregorianCalendar</code>.</p><h2 id="jodatime">JodaTime</h2><p>It’s 2002, Brazil has beaten Germany in the FIFA World Cup, and someone decides
the time is ripe for a proper date/time library. In no time at
all <a shape="rect" href="http://www.joda.org/joda-time/">JodaTime</a> is born!</p><p>It doesn’t take long or Joda becomes the de facto time library on the JVM, and
in 2010 it becomes the basis
for <a shape="rect" href="https://github.com/clj-time/clj-time">clj-time</a>.</p><p>Joda contains two principles classes for representing time,
<code>org.joda.time.Instant</code>, which is essentially a Unix timestamp, and
<code>org.joda.time.DateTime</code>, which explicitly stores year/month/day/etc.</p><p>Having both these classes makes the distinction clear between a “machine time”
and “human time”. For machines things are relatively straightforward: time is an
ever increasing number. The same instant in time corresponds with the same
number, no matter where you are or how you look at it. There are no jumps or
gaps, time is strictly monotonous.</p><p>The human view of time is a lot more complex. There are time zones, daylight
saving time, not to mention different calendar systems and historical time
adjustments. For adding timestamps to your logs machine time will work fine, but
when your system communicates dates and times with end users you need a richer
model.</p><h2 id="jsr-310">JSR-310</h2><p>For years this was the state of things: you used the built-in <code>java.util.Date</code>
despite its shortcomings, or you relied on JodaTime as an external dependency.
That you needed a library to do something as essential as working with dates and
times seemed a bit odd, and so Oracle involved the JodaTime author, Stephen
Colebourne, to work on a new Date/Time library for the JDK, affectionately known
as <a shape="rect" href="https://jcp.org/en/jsr/detail?id=310">JSR-310</a>.</p><p>JSR-310 introduces a new <code>java.time</code> package, which includes <code>java.time.Instant</code>
and <code>java.time.ZonedDateTime</code>. While clearly inspired by JodaTime, there are
also some differences. The author
<a shape="rect" href="http://blog.joda.org/2009/11/why-jsr-310-isn-joda-time_4941.html">explains in this article</a>
that there are some shortcomings in JodaTime that they sought to address.</p><p>Development was completed in 2014, and <code>java.time</code> shipped with Java 8. That’s
basically last week in Java years, it will be a while before support for this
has become widespread.</p><p>On the clj-time issue tracker there’s
an <a shape="rect" href="https://github.com/clj-time/clj-time/issues/196">ongoing discussion</a> about
what this will mean for clj-time. While there’s a lot of interest for clj-time
to adopt JSR-310, it seems the current concensus is that if they do this will be
with a new artifact id (a new library name). The differences between JodaTime
and JSR-310 are big enough to make breaking API changes inevitable. By using a
different name it will be possible to switch to the new version, while libraries
you depend on can keep using the old version.</p><h2 id="clojurejava-time">clojure.java-time</h2><p>Meanwhile someone has gone ahead and created an idiomatic Clojure library for
the <code>java.time.*</code> API, aptly
named <a shape="rect" href="https://github.com/dm3/clojure.java-time">clojure.java-time</a>, so perhaps
this will be what clj-time 2.0 could have been. If you are starting a new
project today you might skip clj-time altogether, and instead use
clojure.java-time for your date and time needs, assuming you’re at least on Java
8.</p><p>If you do go that route you should do your homework though. <code>java.time</code> contains
a handful of different date/timestamp classes, and you should know which is which. The
<a shape="rect" href="https://github.com/dm3/clojure.java-time/blob/master/README.md">clojure.java.time README</a>
has several links for further reading.</p><h2 id="clojureinstant">clojure.instant</h2><p>It’s also worth pointing out that since Clojure 1.8 there’s a built-in namespace
called
<a shape="rect" href="https://clojure.github.io/clojure/clojure.instant-api.html">clojure.instant</a>,
which contains a few utility methods for parsing dates.</p><h2 id="javasqltimestamp">java.sql.Timestamp</h2><p>You thought we were done? Guess again! Besides <code>java.util.Date</code>, Java has always
included yet another timestamp class: <code>java.sql.Timestamp</code>. The reason is that
timestamps in the SQL standard have a nanosecond precision, while
<code>java.util.Date</code> only stores milliseconds. So a <code>java.sql.Timestamp</code> inherits
from <code>java.util.Date</code>, but adds an extra nanosecond field.</p><p>If you’re using JDBC to read or store timestamps, then you’ll run into this
class.</p><h2 id="making-things-bearable">Making things bearable</h2><p>Having all these different types for representing essentially the same thing is
a recipe for massive frustration, but with a few small tweaks we can make it all
a lot less painful.</p><p>The trick is to pick one of the above, and use it across the board. For now
<code>org.joda.time.Instant</code> is still a likely candidate, it has widespread library
support, and you can combine it with <code>clj-time</code> for writing idiomatic Clojure
code. If instead you want to be future proof you can opt for
<code>java.time.Instant</code>, in combination with <code>clojure.java-time</code>. Several people
have reported they’re already getting good use out of <code>java.time</code>. The snippets
below assume you’re sticking with JodaTime.</p><p>Whenever time values enter your program, convert them to JodaTime. Maybe you’re
reading JSON, EDN, Transit, you’re pulling values off an API or out of a
database. Don’t let them propagate through your code as <code>java.util.Date</code>s (or
worse, <code>long</code>s representing Unix timestamps), but convert them to
<code>org.joda.time.DateTime</code> as soon as possible.</p><p>The same goes for outputting values. You don’t want to see an exception any time
you encode some JSON with a timestamp, or save a date to the database.</p><p>Many libraries can be configured to do this automatically, so you only have to
worry about it once.</p><h3 id="jdbc">JDBC</h3><p>JDBC is Java’s standard for communicating with relational databases, and the
<code>clojure.java.jdbc</code> library provides a convenient Clojure wrapper. By default
JDBC expects, and outputs, <code>java.sql.Timestamp</code>. But by loading the
<code>clj-time.jdbc</code> namespace it switched to JodaTime instead. Nice, that was easy!</p><pre><code class="language-clojue">;; load/save DB times as joda DateTime
(require 'clj-time.jdbc)
</code></pre><h3 id="cheshire">Cheshire</h3><p>The go-to library for reading and writing JSON from Clojure is called Cheshire.
JSON doesn’t have a standard way of encoding time values. If you do need to
encode timestamps the common thing to do is to either use a string containing an
ISO-8601 formatted timestamp, or to use a number representing a UNIX timestamp.</p><p>This snippet configures Cheshire to output <code>org.joda.time.DateTime</code> as a string.</p><pre><code class="language-clojure">(ns my.ns
(:require [cheshire.factory]
[cheshire.generate :as cheshire]
[clj-time.coerce :as tc]
[clj-time.core :as t])
(:import [org.joda.time.DateTime]
[com.fasterxml.jackson.core.JsonGenerator]
[java.text.SimpleDateFormat]
[java.util SimpleTimeZone]))
(defn cheshire-add-jodatime-encoder! []
(cheshire/add-encoder
org.joda.time.DateTime
(fn [^org.joda.time.DateTime d ^JsonGenerator jg]
(let [sdf (SimpleDateFormat. cheshire.factory/default-date-format)]
(.setTimeZone sdf (SimpleTimeZone. 0 "UTC"))
(.writeString jg (.format sdf (tc/to-date d)))))))
(cheshire-add-jodatime-encoder!)
</code></pre><h2 id="ednclojure">EDN/Clojure</h2><p>In EDN and Clojure source code you can use an <code>#inst</code> tagged literal to
represent timestamps. These by default will be read as <code>java.util.Date</code>.</p><p>Having a literal syntax for timestamps can be very handy, one example would be
to write out test data containing timestamps.</p><p>In my apps I tend to configure Clojure’s reader and printer to represent
<code>org.joda.time.DateTime</code> as <code>#joda/inst "...timestamp..."</code>, providing me the
same convenience. When doing REPL-driven development (directly or inside a
source buffer) this is particularly useful, since results that are printed out
can be read again, they are proper time literals.</p><p>Clojure’s printer can be extended through the <code>print-method</code> and <code>print-dup</code>
multimethods.</p><pre><code class="language-clojure">;; Configure the printer
(defmethod print-method org.joda.time.DateTime
[dt out]
(.write out (str "#joda/inst \"" (.toString dt) "\"") ))
(defmethod print-dup org.joda.time.DateTime
[dt out]
(.write out (str "#joda/inst \"" (.toString dt) "\"") ))
;; Utility function for the reader
(defn ->joda-inst [time-str]
(org.joda.time.DateTime/parse time-str))
</code></pre><p>The reader can be taught about new tagged literals through a
<code>data_readers.clj</code> file placed on the classpath. I found it works best to put it
under <code>resources</code>, that way Leiningen is smart enough to merge it with other
<code>data_readers.clj</code> files provided by libraries when packing up an uberjar.</p><pre><code class="language-clojure">;; resources/data_readers.clj
{joda/inst lambdaisland.util.time/->joda-inst}
</code></pre><h2 id="tu-quoque-clojurescript">Tu quoque, ClojureScript?</h2><p>This article has focused on JVM based Clojure, but what about ClojureScript?
Things aren’t quite as chaotic on the JavaScript side. It seems in general
people are content using <code>js/Date</code>. There is a separate
<a shape="rect" href="https://google.github.io/closure-library/api/goog.date.DateTime.html">goog.date.DateTime</a> included
with the Google Closure library, which in turn is used
by <a shape="rect" href="https://github.com/andrewmcveigh/cljs-time">cljs-time</a>, so you might run into it.</p><h2 id="for-a-good-time">For a good time</h2><p>Java’s time/date landscape can be difficult to navigate, but a map of the
territory can help you along the way. And with the right tools in your belt it
can be a pleasant journey after all.</p><p><em>Do you have your own snippets for making libraries or subsystems play well with either JodaTime or java.time? <a shape="rect" href="/contact">Send them in</a> and I will add them to the article</em></p><p><em>A special thanks to Ben Lovell, Tim Gilbert, and Sean Corfield for pointing me to clojure.java-time.</em></p></<>>Clojure Gotchas: Surrogate Pairshttps://lambdaisland.com/blog/2017-06-12-clojure-gotchas-surrogate-pairs2020-01-15T13:39:44+00:00<<>><p><em>tl;dr: both Java and JavaScript have trouble dealing with unicode characters from <a shape="rect" href="https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_Multilingual_Plane">Supplementary Planes</a>, like emoji 😱💣.</em></p><p>Today I started working on the next feature of <a shape="rect" href="https://github.com/lambdaisland/uri">lambdaisland/uri</a>, URI normalization. I worked test-first, you’ll get to see how that went in the next Lambda Island episode.</p><p>One of the design goals for this library is to have 100% parity between Clojure and ClojureScript. Learn once, use anywhere. The code is all written in <code>.cljc</code> files, so it can be treated as either Clojure or ClojureScript. Only where necessary am I using a small amount of reader conditionals.</p><pre><code class="language-clojure">#?(:clj
(defmethod print-method URI [this writer]
(.write writer "#")
(.write writer (str edn-tag))
(.write writer " ")
(.write writer (prn-str (.toString this))))
:cljs
(extend-type URI
IPrintWithWriter
(-pr-writer [this writer _opts]
(write-all writer "#" (str edn-tag) " " (prn-str (.toString this))))))
</code></pre><p><em>Example of a reader conditional</em></p><p>For this feature however I’m digging quite deeply into the innards of strings, in order to do <a shape="rect" href="https://en.wikipedia.org/wiki/Percent-encoding">percent-encoding</a> and decoding. Once you get into hairy stuff like text encodings the platform differences become quite apparent. Instead of trying to smooth over the differences with reader conditionals, I decided to create two files <code>platform.clj</code> and <code>platform.cljs</code>. They define the exact same functions, but one does it for Clojure, the other for ClojureScript. Now from my main namespace I require <code>lambdaisland.uri.platform</code>, and it will pull in the right one depending on the target that is being compiled for.</p><pre><code class="language-clojure">(ns lambdaisland.uri.normalize
(:require [clojure.string :as str]
;; this loads either platform.clj, or platform.cljs
[lambdaisland.uri.platform :refer [string->byte-seq
byte-seq->string
hex->byte
byte->hex
char-code-at]]))
</code></pre><p>The first challenge I ran into was that I needed to turn a string into a UTF-8 byte array, so that those bytes can be percent encoded. In Clojure that’s relatively easy. In ClojureScript the Google Closure library came to the rescue.</p><pre><code class="language-clojure">;; Clojure
(defn string->byte-seq [s]
(.getBytes s "UTF8"))
(defn byte-seq->string [arr]
(String. (byte-array arr) "UTF8"))
;; ClojureScript
(require '[goog.crypt :as c])
(defn string->byte-seq [s]
(c/stringToUtf8ByteArray s))
(defn byte-seq->string [arr]
(c/utf8ByteArrayToString (apply array arr)))
</code></pre><p>To detect which characters need to be percent-encoded I’m using some regular expressions. Things seemed to be going well, but when re-running my tests on ClojureScript I started getting some weird results.</p><pre><code>;; Clojure
(re-seq #"." "🥀")
;;=> ("🥀")
;; ClojureScript
(re-seq #"." "🥀")
;;=> ("�" "�")
</code></pre><p><strong>Update: Ben Lovell (<a shape="rect" href="https://twitter.com/socksy/">@socksy</a>) pointed out that modern JavaScript has a flag you can add to regular expressions to make them unicode aware, like so: <code>/some-regex/u</code>. In ClojureScript you can use this syntax to achieve the same effect: <code>(re-seq #"?(u)." "🥀")</code></strong></p><p>This, gentle folks, is the wonder of surrogate pairs. So how does this happen?</p><p>Sadly I don’t have time to give you a complete primer on Unicode and its historical mistakes, but to give you the short version…</p><p>JavaScript was created at a time when people still assumed Unicode would never have more than 65536 characters, and so its strings use two bytes to represent one character, always. This is known as the UCS-2 encoding.</p><p>Unicode has grown a lot since then, and now also has a lot of codepoints with numbers greater than 65536. These include many old scripts, less common CJK characters (aka Hanzi or Kanji), many special symbols, and last but not least, emoji!</p><p>So they needed a way to represent these extra characters, but they also didn’t want to change all those systems using UCS-2 too much, so UTF-16 was born. In UTF-16 the first 65536 codepoints are still encoded the same as in UCS-2, with two bytes, but the ones higher up are encoded with 4 bytes using some special tricks involving some gaps in the Unicode space. In other words, these characters take up the width of two characters in a JavaScript string. These two characters are known as a “surrogate pair”, the first one being the “high surrogate”, and the other one the “low surrogate”.</p><p>So this is what JavaScript strings do now, but the rest of the language never got the memo. Regular expressions, string operations like <code>.substr</code> and <code>.slice</code> all still happily assume it’s 1995, and so they’ll cut surrogate pairs in half without blinking.</p><p>ClojureScript builds on those semantics, so you are liable to all the same mess.</p><pre><code class="language-clojure">(seq "🚩 ")
;;=> ("�" "�")
</code></pre><p>I managed to work around this by first implementing <code>char-seq</code>, a way of looping over the <em>actual</em> characters of a string.</p><pre><code class="language-clojure">(defn char-code-at [str pos]
#?(:clj (.charAt str pos)
:cljs (.charCodeAt str pos)))
(defn char-seq
"Return a seq of the characters in a string, making sure not to split up
UCS-2 (or is it UTF-16?) surrogate pairs. Because JavaScript. And Java."
([str]
(char-seq str 0))
([str offset]
(if (>= offset (count str))
()
(let [code (char-code-at str offset)
width (if (<= 0xD800 (int code) 0xDBFF) 2 1)] ; detect "high surrogate"
(cons (subs str offset (+ offset width))
(char-seq str (+ offset width)))))))
</code></pre><p>I imagine this snippet might come in handy for some. Notice how it’s basically identical for Clojure and ClojureScript. This is because Java suffers from the same problem. The only difference is that there <em>some</em> of the language got the memo. So for instance regular expressions correctly work on characters, but things like substring or <code>.charAt</code> are essentialy broken.</p><p>Hopefully ClojureScript will eventually fix some of this mess, for instance by having a <code>seq</code> over a string return the real characters, but for performance reasons it’s likely they will want to stick closely to JavaScript semantics, so I wouldn’t count too much on this happening.</p><p>In the meanwhile what we can do is document the things you need to watch out for, and write cross-platform libraries like <a shape="rect" href="https://github.com/lambdaisland/uri">lambdaisland/uri</a> that smooth over the differences. 👍</p></<>>Simple and Happy; is Clojure dying, and what has Ruby got to do with it?https://lambdaisland.com/blog/2017-05-25-simple-and-happy-is-clojure-dying-and-what-has-ruby-got-to-do-with-it2017-05-25T19:32:53+00:00<<>><p>The past week or so a lot of discussion and introspection has been happening in the Clojure community. Eric Normand responded to my <a shape="rect" href="https://lambdaisland.com/blog/13-05-2017-lambda-island-turns-one">one year Lambda Island post</a> with <a shape="rect" href="https://purelyfunctional.tv/issues/purelyfunctional-tv-newsletter-225-spec-deep-learning-new-orleans/">some reflections</a> on the size and growth of the community.</p><p>And then Zack Maril lamented on Twitter: <a shape="rect" href="https://twitter.com/ZackMaril/status/863915988424589312">“I’m calling it, clojure’s dying more than it is growing”</a>. This sparked a mega-thread, which was still raging four days later. A <a shape="rect" href="https://www.reddit.com/r/Clojure/comments/6brhsq/interesting_discussion_on_clojure_growth_and/">parallel discussion thread</a> formed on Reddit. Someone asked if their were any <a shape="rect" href="https://twitter.com/mark_engelberg/status/865370061489356800">Clojure failure stories</a>. They were pointed at <a shape="rect" href="https://www.youtube.com/watch?v=doZ0XAc9Wtc">this talk</a> from RubyConf 2016, which seemed to hit a lot of people right in the feels, and <a shape="rect" href="https://twitter.com/ztellman/status/865649146476113923">sparked a subthread</a> with a life of its own.</p><p>Meanwhile Ray, one of the hosts of the <a shape="rect" href="https://defn.audio/">defn</a> podcast <a shape="rect" href="https://twitter.com/raymcdermott/status/866216201709662208">reacted</a> to the original tweet: “I’m calling it: Clojure is alive and well with excellent defaults for productive and sustainable software development.” This sparked another big thread.</p><p>Finally <a shape="rect" href="https://twitter.com/phillmv">@phillmv</a>, the person who did the talk that sparked so much debate, wrote his own summary of the discussion while reiterating some of his points in a blog post: <a shape="rect" href="https://blog.appcanary.com/2017/hard-isnt-simple-ruby-clojure.html">Simple Ain’t Easy, but Hard Ain’t Simple: Leaving Clojure for Ruby</a>.</p><p>It was all a bit much to be honest, but if we care about a healthy and growing Clojure community then it’s important that we have these conversations. Let’s try to unpack some of the points that were raised.</p><h2 id="clojure-is-dying">Clojure is dying</h2><p>This statement initially set fire to the whole debate, but there seems to be little proof that this is true. This <a shape="rect" href="https://news.ycombinator.com/item?id=14302762&__s=r8nvw64mp3jsgweqgmcs">ask HN thread</a> about who’s using Clojure shows plenty of companies adopting it recently. The <a shape="rect" href="https://clojure.org/community/companies">list of companies on Clojure.org</a> mentions Amazon, Facebook, Deutsche Bank, eBay, PayPal, Oracle, ThoughtWorks, Red Hat, Salesforce, Walmart, and 271 others. Onyx raised <a shape="rect" href="http://www.onyxplatform.org/jekyll/update/2016/08/17/Funding.html">500’000$US in funding</a> last year, and the Taiwanese government is <a shape="rect" href="https://blog.pol.is/pol-is-in-taiwan-da7570d372b5">using Clojure</a> for its citizen participation platform.</p><p>The <a shape="rect" href="http://clojurians.slack.com/">community Slack</a> grew from just over 6000 to close to 10K members in the past 12 months. Meetup.com lists 432 <a shape="rect" href="https://www.meetup.com/find/?allMeetups=false&keywords=clojure&radius=Infinity">meetup groups</a> that mention Clojure. We could definitely use more Clojure conferences, but with 8 annual conferences worldwide there are already twice as many as a few years back, and most of them seem to be growing in size year on year.</p><p>One personal observation: at my first EuroClojure in Berlin in 2013, most people I spoke to were enthusiasts. They programmed Java or Ruby or C# at their day jobs, and were dreaming of landing a Clojure job one day. At last year’s EuroClojure in Bratislava it was the opposite, it seemed most people were doing Clojure professionally in some shape or form.</p><p>So it seems industry adoption has grown significantly, and continues to gather steam. Mindshare in the open source and developer enthusiast communities has grown more slowly, and it’s worthwhile asking why that is.</p><h2 id="simple-vs-easy-vs-programmer-happiness">“Simple vs. Easy” vs. “Programmer Happiness”</h2><p>What also hit the internet last week was David Nolen’s talk from GOTO 2017, <a shape="rect" href="https://www.youtube.com/watch?v=lzXHMy4ewtM">Idée Fixe</a>. It’s a great talk. In his casual style he delivers some great insights. The central theme of the talk is “fixed ideas” standing in the way of progress. By becoming fixed on ideas which worked in the past, we lose the openness of mind needed to come up with novel ideas, even when experience or a changing context demonstrate that these old ideas no longer serve us.</p><p>Much of last week’s debate has centered on AppCanary’s choice to move back to Ruby, after a bad experience trying to build their platform in Clojure, and so it was a lot of “Clojure vs Ruby”, both on a language level, and on a cultural and philosophical level. I have been doing Ruby since 2006 and Clojure since 2013, I spoke at Ruby and Clojure conferences, and I’m intimately familiar with both of these languages and communities. <em>I have some feels about this, y’all</em>.</p><p>Each of these has their most prominent mantra. For Clojure it’s “Simple vs. Easy”, while the Rubyist chant is “Optimized for Programmer Happiness”. What do these mean, really? And what are the blind spots, the things we are simply unable to see, because these fixed ideas are in the way?</p><h2 id="simple-vs-easy">Simple vs. Easy</h2><p>At Strange Loop 2011 Rich Hickey delivered one of his classic talks, <a shape="rect" href="https://www.infoq.com/presentations/Simple-Made-Easy">Simple Made Easy</a>, which would have a lasting effect on the discourse around Clojure. In it he contrasts “simple” with “easy”. Simple is the opposite of complex, it is a measure of how entangled things are, of how much is folded and twisted into one. Simple software is focused, it does not interleave multiple concerns.</p><p>Easy is the opposite of hard, it is a measure of the effort required to perform a task. This is largely subjective, something we are familiar with requires less effort, and so is perceived as easy.</p><p>When the selling point of a framework is that you can “build a blog in 15 minutes”, or that a DSL “reads like English”, then you’re talking about ease. Ease helps us when initially writing code. It lets us move fast, and that is not undesirable.</p><p>In the longer run other aspects come to the foreground. Why is it that, as projects age, the speed of development tends to slow down? That it becomes harder to make changes? How do you build something non-trivial over time, while remaining confident in its quality and correctness?</p><p>For that you need to be able to reason about the system. Human minds can only juggle so many things at the same time, and because of this simplicity reigns supreme. We can become familiar with hard things and thus make them easy, but dealing with complexity will continue to hurt your brain. Years of practice won’t change that.</p><p>And so Rich’s assertion is that many balls of mud sprang from a focus on ease, while sweeping complexity under the mat. And so he decided to build a language based on those features that promote simplicity, and rejecting those that lead to complexity.</p><p>Mutable state intertwines everything it touches. Objects mix functionality and state. Variables add a dimension of time to any code that references them.</p><p>Contrast that with immutable values, which are always local in time and space; pure functions, which can be reasoned about in isolation; reference types which make the dimension of time explicit by <a shape="rect" href="https://www.youtube.com/watch?v=-6BsiVyC1kM">separating values from identities</a>.</p><p>Clojure is not designed to be hard, to the contrary, I think a lot of thought has gone into making it easy to use as well, but when ease and simplicity are at odds, then simplicity always wins.</p><h2 id="programmer-happiness">Programmer Happiness</h2><p>Ruby’s creator, Yukihiro Matsumoto (Matz for short), has <a shape="rect" href="https://siliconangle.com/blog/2011/08/31/qa-with-yukihiro-matz-matsumoto-the-creator-of-ruby/">stated repeatedly</a> that Ruby is designed to make programmers happy.</p><blockquote><p>The goal of Ruby is to make programmers happy. I started out to make a programming language that would make me happy, and as a side effect it’s made many, many programmers happy.</p></blockquote><p>He doesn’t really spell out what those things are that make him happy, but we can gather some insight by looking at what he borrowed from other languages, and what he left out. What may surprise you is that, before creating Ruby, Matz’s primary language was Lisp. Perl and Python are also cited as major influences. He wanted to build a modern object oriented scripting language, more suitable for building large systems than Perl, which he considered a toy language, and more pervasively object-oriented than Python, and so he borrowed the object model from Smalltalk.</p><p>When it comes to modeling software Ruby comes closest to Smalltalk, while syntactically, Ruby feels more like Perl, once the favorite scripting language of hackers and tinkerers the world over. Perl is the brainchild of <a shape="rect" href="https://en.wikipedia.org/wiki/Larry_Wall">Larry Wall</a>, by education a linguist. Perl is heavily inspired by natural language, it championed the idea that programming languages should be designed for the human doing the programming, not for the machine running the program. In his talk <a shape="rect" href="http://www.wall.org/~larry/pm.html">Perl, the first postmodern computer language</a>, Larry says,</p><blockquote><p>So, to drag the subject back to computers, one of the characteristics of a postmodern computer language is that it puts the focus not so much onto the problem to be solved, but rather onto the person trying to solve the problem. I claim that Perl does that, and I also claim that, on some level or other, it was the first computer language to do that.</p></blockquote><p>Much of that talk goes into the contrast of a reductionist Modernism vs a more holistic postmodernism, it’s a fascinating read, I highly recommend it.</p><blockquote><p>Modernism, as a form of Classicalism, was always striving for simplicity, and was therefore essentially reductionistic. That is, it tended to take things to pieces. That actually hasn’t changed much. It’s just that Modernism tended to take one of the pieces in isolation and glorify it, while postmodernism tries to show you all the pieces at once, and how they relate to each other.</p></blockquote><p>This still hasn’t given us a clear sense of what is meant with “programmer happiness”, but I’ll venture that it has to do with approaching things in a natural, organic way. Designing a language that appeals to the analytic and the creative and emotional brain equally. A language that loosens the straightjacket and lets you express yourself freely. Ruby promotes <a shape="rect" href="https://martinfowler.com/bliki/HumaneInterface.html">humane interfaces</a></p><blockquote><p>The essence of the humane interface is to find out what people want to do and design the interface so that it’s really easy to do the common case.</p></blockquote><p>Even when picking up a library for the first time you can often guess method names without looking at the docs. It just “does what you would expect”. Some even add aliases. Would you like some <code>is_a?</code> or do you prefer <code>instance_of?</code>? It doesn’t matter, they do the exact same thing!</p><h2 id="fixed-ideas">Fixed ideas</h2><p>Is there any surprise that these two sides seem to be speaking past each other? It’s like they live in different worlds. How can a Clojurist wearing their “Simple vs Easy” sunglasses ever comprehend the joy of creation sparking in the heart of a Rubyist? And how do we explain to the Ruby programmer the beauty of functional code, of having code and data laid bare, nothing up the sleeve, nothing tucked under the covers, just plain values, endlessly flowing, when all they can see is boilerplate code and single letter variable names?</p><p>And yet each side has much to learn from the other, once we get past our fixed ideas.</p><p>“Simple Made Easy” has instilled in many minds the notion that simple and easy are two sides of a spectrum, that they are necessarily at odds. But simple/complex and easy/hard are separate dimensions. While there’s great merit in striving for simplicity, it would be a mistake to disregard ease. I think as Clojure programmers, as library authors, we have been too focused on the reductionist, on building solid parts. But it’s only when these parts come together and meet the real world that magic happens.</p><p>The real world is messy, it’s a complex tangle, a braiding of concerns and constraints. Dealing with that is why we are programmers, this is the real measure of our worth. At the heart of our systems we must strive for “simple”, but at the edges we must inspire, delight, make the impossible effortless. We must do this for our customers, and we must do this for our peers.</p><p>Every tool, every library forms an interface, a language that people must first master to convey their intent. Pay as much attention to designing these interfaces as you would designing their functional internals. This is the face with which your code looks upon the world. Make it smile with its eyes.</p><p>To <a shape="rect" href="http://www.wussu.com/laotzu/laotzu64.html">quote</a> Laozi,</p><blockquote><p>People usually fail when they are on the verge of success.<br clear="none" />
So give as much care to the end as to the beginning;<br clear="none" />
Then there will be no failure.</p></blockquote><p>And Rubyists, you too have something to gain from this exchange. Yes, you’ve been hearing for years of the benefits of immutable data, the elegance of pure functions. These are interesting ideas, but don’t just read about them, try them out, use them in anger, and pay attention to how this influences the resulting system. You can <a shape="rect" href="https://leanpub.com/happylambda">do this in Ruby</a>, but it will be hard to break out of old habits, so instead I recommend picking up a simpler, functional language, Clojure, OCaml, Elixir, and putting it through its paces. Afterwards you’ll look at your Ruby projects with fresh eyes, and a new sense of where complexity comes from.</p><p>I’ll leave you with one more data point to ponder: Clojure has polymorphism, but it <a shape="rect" href="https://groups.google.com/forum/#!msg/clojure/pZFl8gj1lMs/qVfIjQ4jDDMJ">eschews concrete derivation</a>. It has interfaces and protocols, but no inheritance chains. It’s not a design decision that gets put in the limelight much, but it is one I encourage you to study. This alone has tremendous impact on how maintainable systems are.</p><h2 id="minswan">MINSWAN</h2><p>One interesting thing that surfaced in the debates is that while some describe the Clojure community as “incredibly friendly and helpful”, others describe it as hostile. Well, which one is it? Again I think it’s interesting to compare and contrast with the Ruby community.</p><p>From my experience, the average interaction on Clojure’s Slack, mailing lists or Github issues is polite and friendly. I see people tirelessly and patiently answering questions in the <code>#beginners</code> channel and elsewhere. People really want to help.</p><p>And yet it’s a very different vibe from what I see in the Ruby world. There people really go out of their way to be nice to each other. Github issues are accompanied by a bit more elaborate explanations, some emoji or animated gifs, because it’s a low bandwith medium and you really want to make sure the human on the other side doesn’t take things the wrong way. A phrase that was popular in the early days was “MINSWAN”, “Matz is Nice So We Are Nice”. It’s something that deeply permeates this community, it’s part of what made Rails Girls such a huge success, and it’s why I still love going to Ruby conferences. It’s a vibe that’s hard to beat.</p><p>There are trolls in Rubyland too, but they get called out immediately. There is very little tolerance for less than friendly behavior. Virtually every conference and meetup has, and enforces, a Code of Conduct. Being welcoming, especially to beginners, is an essential part of Ruby’s DNA. How to be and remain open and welcoming is an important part of the discourse.</p><p>Clojure is different, it’s a mixed bunch. There are old-school lispers, converted Java programmers, people coming from other dynlangs like Ruby or Python, and they all carry different expectations. If you’ve been active in open source for more than a decade you know that a thick skin was a basic requirement. Harsh interactions were par for the course. People do what they are used to, and without a guiding principle like MINSWAN it’s harder to call people out and hold everyone to higher standards.</p><p>In the Clojure world, as in Clojure code, brevity is considered a feature, but there’s a thin line between being concise, and being curt. People still try to be friendly and nice, but the dial isn’t turned up all the way to 11. This is painfully true when it comes to the official docs, which seem to be written for machines rather than humans, it shows in the error messages, and it shows in many small online interactions.</p><p>Languages that draw a younger, more homogeneous crowd: Elm, Elixir, JavaScript, seem to be better at prioritizing the humanistic aspect. Both in the affordances of the software (Elm’s famous error messages), and in person-to-person interactions. I think we can, and should, take a leaf out of their book.</p><h2 id="catering-for-the-novice">Catering for the Novice</h2><p>People cite many pain points when coming to Clojure. Chas Emerick <a shape="rect" href="https://twitter.com/cemerick/status/865657364317298690">summarized</a> it well:</p><blockquote><p>✅ jvm pain<br clear="none" />
✅ docs/error msgs<br clear="none" />
✅ deployment pain<br clear="none" />
✅ tooling/debugging pain<br clear="none" />
✅ domain modeling difficulty</p></blockquote><p>There’s truth in all of these, and we as a community will have to stand up if we want to change these, but I want to focus on one more thing in particular: meeting the needs of the absolute beginner.</p><p>The <a shape="rect" href="https://en.wikipedia.org/wiki/Dreyfus_model_of_skill_acquisition">Dreyfus model</a> distinguishes five levels of competence, from novice to mastery. At the absolute beginner level people execute tasks based on “rigid adherence to taught rules or plans”. Beginners need <em>recipes</em>. They don’t need a list of parts, or a dozen different ways to do the same thing. Instead what works are step by step instructions that they can internalize. As they practice them over time they learn the reasoning behind them, and learn to deviate from them and improvise, but they first need to <em>feel like they’re doing something</em>.</p><p>I have taught many people Ruby on Rails. Rails is tremendously complex (and complected), but it gets this part right. In a few sessions you can teach a person to create an interactive website backed by a database. It’s frustrating to teach because you’re walking a tight rope. Just follow the steps. Don’t peek under the covers. Don’t stare into the abyss. But that feeling of instantly being productive is what draws people in, what makes them think, “hey, maybe I <em>can</em> do this”.</p><p>People criticize this as optimizing the wrong thing. Yes, initial prototyping goes really fast, but long term maintainability suffers. This criticism isn’t without merit, but it tends to overlook this important aspect. “The Rails Way” provides a framework that makes teaching easy. It means you can just dive in and ignore all the peripheral stuff until later.</p><p>For ClojureBridge we tend to teach graphics programming with Quil, with Nightcode as the editor. There the experience is similar. Within the first hour people are doing stuff, they’re building things. And there are plenty of recipes out there for Quil that people can adopt and run with.</p><p>But in general, and for web development in particular, Clojure fails to deliver that experience. This may sound like a case for more of a “framework” and less of a “library” approach, but I think the bigger issue is documentation. We need more practical tutorials and how-to guides, that provide clear recommendations of tools and libraries to use, that explain things step by step, with empathy, with beginners in mind. Guides that you can follow from top to bottom and that <em>actually work</em>, even when you’re on Windows, or Linux, or have never heard of a classpath.</p><p>We’re also missing a dialogue about <em>how to program</em>, both in the small: how do you structure a function, what are common idioms, how do you manipulate data structures; and in the large: doing domain modeling, architecting systems, designing interfaces. Ruby folks talk about these things all the time, but talks like these are eerily absent from Clojure conferences, and yet it’s something people consistently cite as a stumbling block.</p><p>If we want Clojure to continue to grow, we need to take responsibility for how it is perceived. Clojure is not always perceived as a friendly language. We can retort, “it was very friendly to <em>me</em>”, or we can listen without interrupting the speaker, believe they are speaking from good faith, and do better.</p><p><a shape="rect" href="https://www.reddit.com/r/Clojure/comments/6d9say/simple_and_happy_is_clojure_dying_and_what_has/">🡅 You can comment (and upvote) on Reddit.com</a></p><hr /><p><strong>Update</strong>: The <a shape="rect" href="https://clojure.org">https://clojure.org</a> web site has a guide section for tutorials and guides and accepts pull requests.</p><ul><li><a shape="rect" href="https://clojure.org/community/contributing_site">https://clojure.org/community/contributing_site</a></li><li><a shape="rect" href="https://clojure.org/guides">https://clojure.org/guides</a></li></ul><p><em>Many thanks to <a shape="rect" href="https://twitter.com/cemerick">Chas Emerick</a>, <a shape="rect" href="https://twitter.com/ericnormand">Eric Normand</a>, and <a shape="rect" href="https://twitter.com/pragtob">Tobias Pfeiffer</a> for reviewing an earlier version of this article.</em></p></<>>Loading Clojure Libraries Directly From Githubhttps://lambdaisland.com/blog/2017-05-17-loading-clojure-libraries-directly-from-github2017-05-17T13:59:23+00:00<<>><p>Did you ever fix a bug in an open source library, and then had to wait until the maintainer released an updated version?</p><p>It’s happened to me many times, the latest one being <a shape="rect" href="https://github.com/metabase/toucan">Toucan</a>. I had run into a limitation, and found out that there was already an <a shape="rect" href="https://github.com/metabase/toucan/issues/6">open ticket</a>. It wasn’t a big change so I decided to <a shape="rect" href="https://github.com/metabase/toucan/pull/10">dive in and address it</a>. Just a little yak shave so I could get on with my life.</p><p>Now this pull request needs to be reviewed, and merged, and eventually be released to Clojars, but ain’t nobody got time for that stuff. No sir-ee.</p><p>So what do I do? Do I push my own version to Clojars? Call it <code>lambdaisland/toucan</code>? If the project seems abandoned, and it could help others as well, then maybe I would, the way I did with <a shape="rect" href="https://github.com/pjlegato/ring.middleware.logger/pull/21#issuecomment-285943214">ring.middleware.logger</a>, but Toucan is actively maintained, so releasing a fork might cause confusion (and maybe even be taken the wrong way).</p><p>For cases like these it’s good to know that you can use Github as a Maven repository. That’s right, you don’t need to deploy or build any jars. Any Clojure library with a proper <code>project.clj</code> will do, thanks to the power of <a shape="rect" href="https://jitpack.io/">Jitpack</a>.</p><p>All you need to do is create a git tag and push it Github.</p><pre><code class="language-sh">$ git tag v1.0.2-with-hydrate-fix
$ git push plexus --tags
</code></pre><p>Now add the Jitpack maven repository, and you can load your library straight aways.</p><pre><code class="language-clojure">(defproject lambdaisland "0.160.0"
:dependencies [,,,
[com.github.plexus/toucan "v1.0.2-hydrate-fix"]]
:repositories [["jitpack" "https://jitpack.io"]])
</code></pre><p>Notice the format of the dependency vector: <code>[com.github.your-github-name/your-project-name "git-tag"]</code>.</p><p>Let’s try that out:</p><pre><code class="language-sh">$ lein deps
Could not transfer artifact com.github.plexus:toucan:pom:v1.0.2-hydrate-fix from/to jitpack (https://jitpack.io): Read timed out
This could be due to a typo in :dependencies or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
</code></pre><p>This is usually the case, the first time you try to download the jar, Jitpack still needs to fetch the code and build it, so this tends to time out. But try again a minute later and, tada \o/</p><pre><code class="language-sh">$ lein deps
Retrieving com/github/plexus/toucan/v1.0.2-hydrate-fix/toucan-v1.0.2-hydrate-fix.pom from jitpack
Retrieving com/github/plexus/toucan/v1.0.2-hydrate-fix/toucan-v1.0.2-hydrate-fix.jar from jitpack
</code></pre><p><a shape="rect" href="https://jitpack.io/">Jitpack</a> also has paid plans, for when you want to deploy stuff straight from private repositories, but they’re free for open source, so kudos to them!</p></<>>Lambda Island Turns One, The Story of a Rocky Ridehttps://lambdaisland.com/blog/2017-05-13-lambda-island-turns-one2017-05-13T11:49:17+00:00<<>><p>One year ago to date I launched Lambda Island, a service that offers high quality video tutorials on web development with Clojure and ClojureScript. It’s been quite a ride. In this post I want to look back at the past year, provide some insight into how this experience has been for me, and give you a glimpse of what the future has in store.</p><p>This story really starts in December 2015. After three years of doing contract work for <a shape="rect" href="http://ticketsolve.com">Ticketsolve</a> I decided it was time for a change. I have been self-employed for many years, but I knew that sooner or later I wanted to try my hand at selling a product, rather than selling my time.</p><p>In January and February I took some time for soul-searching, and recharging. I went to speak at RubyConf Australia, and got to hang out with some old friends around Australia and New Zealand. Once back in Berlin I got busy creating Lambda Island.</p><p>I decided to build the site from scratch. There are plenty of reasons why this isn’t the best idea, many would argue I’d be better off using an existing platform, but so far I haven’t regretted the choice. I felt it was important for my street cred to be running a Clojure app in production.</p><p>I did choose to make use of a handful of well chosen SaaS providers to do the heavy lifting. <a shape="rect" href="https://www.chargebee.com/">Chargebee</a> does all the subscription management and invoicing, and I’m very happy with their service. EU VAT regulation is complex and a pain to implement, but they handle that swimmingly.</p><p>Payment is handled through Chargebee by <a shape="rect" href="https://stripe.com/">Stripe</a>, video hosting is done by <a shape="rect" href="https://vimeo.com/">Vimeo</a>, emails are sent with <a shape="rect" href="https://www.mailgun.com/">Mailgun</a>, drip campaigns go out with <a shape="rect" href="https://www.sendwithus.com/">Sendwithus</a>, and the whole thing runs on <a shape="rect" href="https://www.upcloud.com/">Upcloud’s</a> virtual servers. Upcloud generously sponsored the last ClojureCup, and the credit we won there kept the servers up for almost a year.</p><p>I also had to figure out how to create screencasts. I had made some videos before, but there’s a big difference between just recording a live coding session, and actually creating a high quality screencast. With trial and error I managed to come up with a recording and editing workflow that worked for me, and I’ve been improving upon that ever since. I’ll leave the details of that for another post.</p><p>After figuring out some legal aspects and creating the first few videos I was finally ready to launch, and on 13 May 2016 I welcomed my first customers.</p><div style="display:flex; align-items: center; justify-content: center;"><blockquote class="twitter-tweet" data-lang="en"><p dir="ltr" lang="en">
I have customers now. The world will never be the same again.
</p><p>— 𐌐𐌋𐌄𐌗𐌖𐌔 (@plexus) <a shape="rect" href="https://twitter.com/plexus/status/731089209684074496">May 13, 2016</a></p></blockquote></div><script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script><p>I would need about 700 subscribers to make what I was making before with contract work, but I was prepared to take a big pay cut. I figured with 200 customers I would make enough to live, or 300 to live modest but comfortably. If I could get about 30 new subscribers each month, then by the time my savings would run out I would be self-sufficient, all by the end of 2016.</p><p>Spoiler: that’s not quite what happened. In the past month I finally broke through the first 100, and it’s been a big struggle to stay afloat.</p><p>Here you see the number of activations (initial signups or people who cancelled earlier and the re-activated their subscription), and number of cancellations (a.k.a. “The Churn”), for each month.</p><p><img src="/img/blog/20170513-activations.png" alt="" style="width:100%" /><img src="/img/blog/20170513-churn.png" alt="" style="width:100%" /></p><p>It’s been quite a rollercoaster ride. It all started off well. Twenty-five people signed up in the first two weeks. Many of those are still happy customers today. June and July were slow, but in August things were starting to pick up, and all through fall I had ten to fifteen new members each month.</p><p>At that point I had been working long days and weekends for close to half a year. I started getting more migraines and tension headaches, I had been getting sick more and more often, and mentally I was not in a good place. At the same time my financial reserve was getting dangerously low.</p><p>Around September I knew I needed a break, but I also urgently needed to make some money. I started looking for freelance Clojure work, but that wasn’t that easy to come by, especially not remote or part-time. I ended up taking on a project for <a shape="rect" href="https://www.mein-grundeinkommen.de/">Mein Grundeinkommen</a>, a non-profit working to supply people with basic income. The break would have to wait.</p><p>This brought some financial relief, but it did keep me busy for a while, so in October and November only two new videos came out. After that I found a gig teaching programming a few hours a week. Together with what was slowly starting to come in from Lambda Island I could already cover a good part of my costs.</p><p>At the same time I started making some life style changes. I made an effort to block the weekends and evenings for things unrelated to work. I cut my alcohol consumption way back. I wasn’t a big drinker before, but I found I was getting very sensitive to it, and just a few drinks would have me feeling sick and depressed for days. I started doing more sports, and made sure to get enough sleep. Since January I’ve been doing an hour of <a shape="rect" href="https://en.wikipedia.org/wiki/Alexander_technique">Alexander Technique</a> once a week with a teacher, and for over a month now I’ve been meditating once a day with <a shape="rect" href="https://www.headspace.com/">Headspace</a>.</p><p>I still haven’t gotten the break that I needed, and I still get sick too easily, but I’m in a much better place than I was six months ago. I’m feeling productive again and optimistic for the future. I’ve learned some hard lessons from it all, and in that sense it’s been an important experience. I’ll also be taking two weeks of actual holiday this summer, one week with my family, and one week at the <a shape="rect" href="http://ejc2017.org/">European Juggling Convention</a>, which I’m sure will do me good.</p><p>So that’s where we are now. One year in. One hundred subscribers. Why hasn’t it taken off the way that I had hoped?</p><p>I am convinced the product itself is not the problem. People seem to genuinely appreciate the content I make. I did a survey to see why people cancelled, and the most common reason is they simple no longer use Clojure. The only complaint about the product itself is that there’s not enough new content coming out. This is understandable, but it’s also not something I can do much about at this point. And at 12 USD per month I think it’s fair to say people are getting their money’s worth.</p><p>In fact I’ve been meaning to raise the price for some time now, but since February sign-ups are barely keeping pace with cancellations, which has made me hesitant. At the same time I’m afraid that people think the low price reflects the quality of the content. So before long prices will have to go up, to make it clear that this is a luxury product, not fast food.</p><p>So if the product is good, then it follows that the problem is lack of marketing. Saying I haven’t done enough marketing is stating the obvious. So far all I’ve really made use of is social media and content marketing. Twitter and r/Clojure are two good sources of traffic.</p><p>There’s a rule of thumb that half of your time and effort should be spent on marketing, and the other half on developing the product. Most of my time goes into creating content, and hustling freelance gigs. I have a long list of things to try out though, but acting on them will take time.</p><p>So for the time being I need to accept that Lambda Island will remain more of a side gig then a primary source of income. Approaching it from that angle will give me a lot more peace of mind, and will give me time to gradually grow it into something that can stand on its own two feet.</p><p>So my plan is to stick to the current two-to-three week publishing schedule. I have a two freelance gigs that I’m wrapping up at the moment, and after that I’m looking to do ClojureScript contract work for two days a week. That should give me enough financial stability to worry less, and spend those three other days working productively on Lambda Island.</p><p>Thanks for reading this far. I felt it was important to tell this story, so people in the community know what’s up. I strongly believe in the power of open and honest communication, and with posts like these I want to take you all along on this journey.</p><p>Some people have asked what they can do to help. Here are a few things you can do,</p><ul><li>Sign up for a subscription</li><li>Get your company to sign up for a team membership</li><li>Get your company to sponsor a video</li><li>Try the free trial and give me feedback</li><li>Tell your friends about Lambda Island, tell people what you like about it</li><li>Share links to the blog posts, freebies, the REPL guide, on Twitter/Facebook/Reddit/Slack</li></ul><p>Thanks for listening, here’s to the next year of Lambda Island!</p><p>🡅 <a shape="rect" href="https://www.reddit.com/r/Clojure/comments/6axfdy/lambda_island_turns_one_the_story_of_a_rocky_ride/">Comment on /r/Clojure</a></p></<>>Writing Node.js scripts with ClojureScripthttps://lambdaisland.com/blog/2017-05-02-nodejs-scripts-clojurescript2017-05-02T17:24:45+00:00<<>><p>In the
<a shape="rect" href="https://lambdaisland.com/episodes/building-cli-apps-with-lumo-part-1">two most recent</a> <a shape="rect" href="https://lambdaisland.com/episodes/building-cli-apps-with-lumo-part-2">Lambda Island episodes</a> I
covered in-depth how to create command line utilities based
on <a shape="rect" href="https://github.com/anmonteiro/lumo">Lumo</a>, how to combine them with third
party libraries, and how to deploy them to <a shape="rect" href="http://npmjs.com/">npmjs.com</a>.</p><p>However there’s a different way to create tools with ClojureScript and
distribute them through NPM, without relying on Lumo. In this blog post I want
to quickly demostrate how to do just that.</p><p>To recap, Lumo is a ClojureScript environment based on Node.js, using
bootstrapped (self-hosted) ClojureScript. This means the ClojureScript compiler,
which is written in Clojure and runs on the JVM, is used to compile itself to
JavaScript. This way the JVM is no longer needed, all you need is a JavaScript
runtime to compile and run ClojureScript code, which in this case is provided by
Node.js. On top of that Lumo uses <a shape="rect" href="https://github.com/nexe/nexe">nexe</a>, so Lumo
can be distributed as a single compact and fast executable binary.</p><p>With Lumo you can wrap your ClojureScript code in an NPM package, and have it
run on the fly. The benefit of this is that there’s no separate compilation
step, so you don’t have to fiddle with ClojureScript compiler configuration. You
get a fast feedback cycle during development, and your code runs exactly the
same way during development as in production (or in this case: someone else’s
computer.)</p><p>But there are also downsides to this approach. Compiling code and loading
dependencies on the fly means the startup time of your app will suffer. It also
means you need to have all dependencies available at runtime, instead of at
compile time, and you don’t get to benefit from Google Closure’s advanced code
optimizations, so all of this contributes to a bigger installed size on disk.</p><p>If you’re not afraid to mess with ClojureScript compiler options
(<a shape="rect" href="https://lambdaisland.com/episodes/clojurescript-compiler">which you shouldn’t be</a>),
then this is how you do it.</p><p>Here’s an exampl script. It turns its arguments into <a shape="rect" href="https://en.wikipedia.org/wiki/Leet">l33t sp34k</a></p><pre><code class="language-clojure">;; src/l33t/core.cljs
(ns l33t.core
(:require [clojure.string :as str]
[cljs.nodejs :as nodejs]))
(nodejs/enable-util-print!)
(defn -main [& args]
(-> (str/join " " args)
(str/replace #"cker\b" "xor")
(str/replace #"e|E" "3")
(str/replace #"i|I" "1")
(str/replace #"o|O" "0")
(str/replace #"s|S" "5")
(str/replace #"a|A" "4")
(str/replace #"t|T" "7")
(str/replace #"b|B" "6")
(str/replace #"c|C" "(")
println))
(set! *main-cli-fn* -main)
</code></pre><p>Now you need a way to compile this to Javascript. You could
use <a shape="rect" href="https://github.com/boot-clj/boot">Boot</a>, or
a
<a shape="rect" href="https://github.com/clojure/clojurescript/wiki/Quick-Start">small custom build script</a>.
I’m going to use <a shape="rect" href="https://github.com/emezeske/lein-cljsbuild">lein-cljsbuild</a>.</p><pre><code class="language-clojure">;; project.clj
(defproject l33t "0.1.0"
:dependencies [[org.clojure/clojure "1.9.0-alpha15"]
[org.clojure/clojurescript "1.9.521"]]
:plugins [[lein-cljsbuild "1.1.5"]]
:cljsbuild {:builds [{:id "prod"
:source-paths ["src"]
:compiler {:main l33t.core
:output-to "package/index.js"
:target :nodejs
:output-dir "target"
;; :externs ["externs.js"]
:optimizations :advanced
:pretty-print true
:parallel-build true}}]})
</code></pre><p>Notable here is <code>:target :nodejs</code>, this is necessary to make the final script
run on Node. I’m setting <code>:optimizations</code> to <code>:advanced</code>, to get the full power
of the Google Closure compiler, and to prevent all my code from getting
optimized away, I’m setting <code>:main</code> to <code>l33t.core</code>, the main namespace.</p><p>When you compile this with</p><pre><code>lein cljsbuild once prod
</code></pre><p>It will create <code>package/index.js</code>. Now create <code>package/package.json</code> to look like this:</p><pre><code class="language-javascript">{
"name": "l33t",
"bin": {
"l33t": "index.js"
}
}
</code></pre><p>So this is what you have now:</p><pre><code>.
├── package
│ ├── index.js
│ └── package.json
├── project.clj
└── src
└── l33t
└── core.cljs
</code></pre><p>Go into <code>package</code> and do <code>npm link .</code> so you can test out the package.</p><pre><code>$ cd package
$ npm link .
/home/arne/.nvm/versions/node/v6.10.2/bin/l33t -> /home/arne/.nvm/versions/node/v6.10.2/lib/node_modules/l33t/index.js
/home/arne/.nvm/versions/node/v6.10.2/lib/node_modules/l33t -> /home/arne/projects/l33t
$ l33t I am a leet hacker
</code></pre><p>The script runs, but it doesn’t output anything. The reason is that it tries to
find <code>process.argv</code>, but because of the advanced compilation, this has been
munged to something like <code>process.FgI</code>. Externs to the rescue!</p><pre><code class="language-javascript">// externs.js
var process = {};
process.argv = [];
</code></pre><p>Comment out the <code>:externs</code> line in <code>project.clj</code>, recompile, and you’re good to go.</p><pre><code>$ l33t I am a leet hacker
1 4m 4 l337 h4x0r
</code></pre><p>Alternatively, you can include
<a shape="rect" href="https://github.com/cljsjs/packages/tree/master/nodejs-externs">cljsjs/nodejs-externs</a> as
a dependency, so the whole Node.js API is immediately covered.</p><p><img src="https://lambdaisland.com/img/github-small-icon.png" alt="" /><a shape="rect" href="https://github.com/lambdaisland/l33t">browse the code on Github</a></p><p>⬆ <a shape="rect" href="https://www.reddit.com/r/Clojure/comments/68uhxo/writing_nodejs_scripts_with_clojurescript/">discuss this post on Reddit</a></p></<>>Announcing lambdaisland/uri 1.0.0https://lambdaisland.com/blog/2017-02-27-announcing-lambdaisland-uri2017-02-27T15:20:20+00:00<<>><p>I just released <code>lambdaisland/uri</code>, a pure Clojure/ClojureScript URI library. It is available <a shape="rect" href="https://github.com/lambdaisland/uri">on Github</a> and <a shape="rect" href="http://clojars.org/lambdaisland/uri">Clojars</a>.</p><p>This is a small piece of the code base that powers <a shape="rect" href="https://lambdaisland.com">lambdaisland.com</a>. It’s inspired by Ruby’s <a shape="rect" href="https://github.com/sporkmonger/addressable">Addressable::URI</a>, the most solid URI implementation I’ve seen to date, although it only offers a small part of the functionality that library offers.</p><p>It’s written in pure Clojure/ClojureScript, with only minimal use of <code>.cljc</code> reader conditionals to smooth over differences in regular expression syntax, and differences in core protocols. It does not rely on any URI functionality offered by the host, such as <code>java.net.URL</code>, so it’s usable across all current and future Clojure implementations (Clojure, ClojureScript, ClojureCLR).</p><p>Currently all this library offers is parsing a URI to its components (scheme, host, path, etc.), manipulating those components individually, and performing RFC compliant joining. <code>lambdaisland.uri.URI</code> instances print with a <code>#lambdaisland/uri "foo://bar"</code> reader tag, and a reader function is provided for use with <code>clojure.edn/read</code>.</p><p>The main missing parts are URI normalization and validation. These will be added in later versions. The intention is for this to become a modern Clojure URI library that can serve us for the time to come. Contributions are more than welcome. The code is released under the <a shape="rect" href="https://www.mozilla.org/media/MPL/2.0/index.txt">Mozilla Public License 2.0</a>, a <a shape="rect" href="https://tldrlegal.com/license/mozilla-public-license-2.0-(mpl-2)">non-viral copyleft license (link to ‘tl;dr legal’)</a>.</p><p>The code has extensive test coverage, and has CI set up for Clojure and ClojureScript (doo+phantomjs) to make sure it stays cross-platform compatible.</p><p>Example code:</p><pre><code class="language-clojure">(require '[lambdaisland.uri :refer [uri join]])
;; uri :: String -> lambdaisland.uri.URI
(uri "//example.com/foo/bar")
;;=> #lambdaisland/uri "//example.com/foo/bar"
;; A URI is a record, use assoc to update specific parts
;; Use `str` if you want the URI back as a string
(str
(assoc (uri "//example.com/foo/bar")
:scheme "https"
:user "arne"
:password "supersecret"
:host "lambdaisland.com"
:port "3333"
:path "/hello/world"
:query "q=5"
:fragment "section1"))
;;=> "https://arne:supersecret@lambdaisland.com:3333/hello/world?q=5#section1"
;; RFC compliant joining of relative URIs
(join "//example.com/foo/bar" "./~arne/site/" "../foo.png")
;;=> #lambdaisland/uri "//example.com/foo/~arne/foo.png"
;; Arguments to `join` are coerced, you can pass strings, java.net.URI, or any x
;; for which `(str x)` returns a URI string.
(join (java.net.URI. "http://example.com/foo/bar") (uri "./~arne/site/") "../foo.png")
;;=> #lambdaisland/uri "http://example.com/foo/~arne/foo.png"
;; URI implements IFn for keyword based lookup, so it's fully
;; interface-compatible with Clojure maps.
(:path (uri "http://example.com/foo/bar"))
;; Instances of URI are printed with a #lambdaisland/uri reader tag. To read
;; them back from EDN, use the provided readers.
(require '[clojure.edn :as edn])
(edn/read-string
{:readers lambdaisland.uri/edn-readers}
"#lambdaisland/uri \"http://example.com/foo/~arne/foo.png\"")
</code></pre><h3 id="about-the-author">About the author</h3><p><em>
Arne divides his time between making Clojure tutorial videos for <a shape="rect" href="https://lambdaisland.com">Lambda Island</a>, and working on open-source projects like <a shape="rect" href="https://github.com/plexus/chestnut">Chestnut</a>. He is also available for in-person training and mentoring. You can support Arne through <a shape="rect" href="https://www.patreon.com/plexus">his Patreon page</a>, which also gives you access to exclusive content.
</em></p></<>>re-frame Subscriptions Got Even Betterhttps://lambdaisland.com/blog/2017-02-11-re-frame-form-1-subscriptions2017-02-11T11:10:43+00:00<<>><p>Up until recently, to use re-frame subscriptions in Reagent views, you had to
use a form-2 component.</p><p>A form-2 component is a function that returns another function, which does the
actual rendering of the component to hiccup. In contrast, a form-1 component
renders the hiccup directly.</p><pre><code class="language-clojure">;; form-1
(defn todo-item [todo]
[:div.view
[todo-checkbox (:id todo) (:completed todo)]
[:label {:unselectable "on"} title]
[:button.destroy {:on-click #(dispatch [:todos/remove (:id todo)])}]])
;; form-2
(defn todo-item [todo]
(fn [todo]
[:div.view
[todo-checkbox (:id todo) (:completed todo)]
[:label {:unselectable "on"} title]
[:button.destroy {:on-click #(dispatch [:todos/remove (:id todo)])}]]))
</code></pre><p>It’s a small difference, but it has interesting uses. With a form-2 component
the outer function is only called once. When re-rendering a component, only the
inner function is called. This gives you a place to put one-time initialization
code. In that sense it’s a bit like React’s
<a shape="rect" href="https://facebook.github.io/react/docs/react-component.html#componentwillmount"><code>componentWillMount</code></a>
hook.</p><p>Since the inner function is a closure, you can have it close over variables by
using a <code>let</code> block. A common pattern is to close over a Reagent atom to hold
component-local state.</p><pre><code class="language-clojure">(require '[reagent.core :as r])
(defn todo-input []
(let [title (r/atom "")]
(fn []
[:input#new-todo {:type "text"
:value @title ;; <------------ deref!
:placeholder "What needs to be done?"
:auto-focus true
:on-change #(reset! title (-> % .-target .-value))}])))
</code></pre><p>By deref’ing the atom inside the render function Reagent will know that the
component depends on this ratom, and re-render it when the atom updates.</p><p>The same is true for Reagent reactions, such as returned by re-frame’s
<code>subscribe</code> function. You only want the reaction to be created once, but it
should be dereferenced for its current value whenever the component re-renders.</p><p><em> The re-frame
<a shape="rect" href="https://github.com/Day8/re-frame/blob/master/docs/SubscriptionFlow.md">docs about “subscription flow”</a>
contain a great introduction to ratoms and reactions. They are also covered in
depth in
<a shape="rect" href="https://lambdaisland.com/episodes/re-frame-subscriptions">Lambda Island episode 20, re-frame Subscriptions</a></em></p><pre><code class="language-clojure">(require '[re-frame.core :refer [subscribe])
(defn toggle-all-checkbox []
(let [all-complete? (subscribe [:todos/all-complete?])]
(fn []
[:span
[:input {:type "checkbox"
:checked @all-complete?}] ;; <---------------- deref!
[:label {:for "toggle-all"} "Mark all as complete"]])))
</code></pre><p>Re-frame 0.8 introduced a subscription cache, meaning that multiple calls to
subscribe with the same arguments will return the same identical reaction.</p><p>In version 0.9
<a shape="rect" href="https://github.com/Day8/re-frame/issues/218">this became official</a>, and so now
you can subscribe and dereference a subscription in one go.</p><pre><code class="language-clojure">(defn toggle-all-checkbox []
[:span
[:input {:type "checkbox"
:checked @(subscribe [:todos/all-complete?])}]
[:label {:for "toggle-all"} "Mark all as complete"]])
</code></pre><p>It’s a neat change, and one that makes re-frame yet again a little bit more user
friendly.</p><p>You could go one step further by creating a little helper function that
subscribes and derefs in one go:</p><pre><code class="language-clojure">(def deref-subscribe [query-v]
(deref (subscribe query-v]))
</code></pre><p>Or if you like it point-free:</p><pre><code class="language-clojure">(def deref-subscribe (comp deref subscribe))
</code></pre><p>There was <a shape="rect" href="https://github.com/Day8/re-frame/issues/218">some discussion</a> to add
this helper to re-frame itself, but the jury is still out on what to call it.
Proposed names include <code>listen</code>, <code>watch</code>, and <code>input-signal</code>.</p><p>Personally I started using <code><sub</code> in my projects, and I’m liking it a lot. It
provides a little visual queue as to where the inputs of a component are. I’ve
even consider using <code>>evt</code> as an alias for <code>dispatch</code>.</p><p>I would perhaps even go one step further and
<a shape="rect" href="https://twitter.com/jamesiry/status/828656533584048131">embrace the unicode future</a>
by using ⇐ and ⇒, but it seems ClojureScript doesn’t like Unicode identifiers (´•̥̥̥︵•̥̥̥` )</p><pre><code class="language-clojure">;; sadly doesn't work
;; (def ⇐ (comp deref re-frame.core/subscribe))
;; (def ⇒ re-frame.core/dispatch)
(def <sub (comp deref re-frame.core/subscribe))
(def >evt re-frame.core/dispatch)
(defn toggle-all-checkbox []
[:span
[:input {:type "checkbox"
:checked (<sub [:todos/all-complete?])
:on-change #(>evt [:todos/toggle-all])}]
[:label {:for "toggle-all"} "Mark all as complete"]])
</code></pre><p>If you’re wondering how you would type those arrows, then it’s about time you
set up a <a shape="rect" href="https://en.wikipedia.org/wiki/Compose_key">Compose Key</a>!</p></<>>Why React & re-frame are fast in 7 Performance Hackshttps://lambdaisland.com/blog/2017-02-02-why-react-reagent-fast-7-performance-hacks2017-02-02T13:38:02+00:00<<>><p>This one had been sitting on my drive for a while, I thought it would be nice to share it with y’all. It’s a recording from a talk I did at the <a shape="rect" href="https://www.meetup.com/Clojure-Berlin/events/233941806/">Clojure Berlin Meetup last October</a>, about the various performance tricks that React, Reagent, and re-frame employ to make sure your apps are as spiffy as can be.</p><p>The <a shape="rect" href="http://arnebrasseur.net/talks/2016-react-fast/#1">slides are on-line</a> as well.</p><div class="video aspect-ratio"><iframe frameborder="0" scrolling="auto" src="https://player.vimeo.com/video/202177001" width="1119" height="629" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe></div><p><br clear="none" /><br clear="none" /></p><p><a shape="rect" href="https://www.reddit.com/r/Clojure/comments/5rmrgz/why_react_reframe_are_fast_in_7_performance_hacks/">Discuss this video on Reddit</a>.</p></<>>Game Development with Clojure/ClojureScripthttps://lambdaisland.com/blog/2016-12-08-game-development-with-clojure-clojurescript2016-12-08T21:27:46+00:00<<>><p>This weekend it’s <a shape="rect" href="https://ldjam.com/">Ludum Dare</a> again, the world’s longest
running game jam. The idea is that, alone or with a team, you build a game in a
weekend based on a certain theme.</p><p>We got a little team together here in Berlin, and so I’ve been reviewing what
options there are for someone wanting to build a game in Clojure or
Clojurescript.</p><p>The good news is there are plenty of options, as you’ll see from the list below.
You can do desktop games, browser based games with canvas or webgl, and you can
even create <a shape="rect" href="https://unity3d.com/">Unity 3D</a> games, all from your comfortable
Clojure parentheses.</p><p>That’s more or less where the good news ends. Most of these frameworks and
libraries are still alpha-level software, many have a “use at your own risk,
you’re on your own” kind of warning on the project page. And the
documentation… well… let’s just say that you’re likely better off studying
<a shape="rect" href="https://en.wikipedia.org/wiki/Oracle_bone">cracked turtle bones</a> to figure out
how these libraries work.</p><p>Disclaimer: most of these I haven’t actually tried, certainly not in anger, so
much of what follows is from what I could gather looking at the project page and
perusing some code samples and (if present) docs. Feel free to “well actually”
me <a shape="rect" href="https://twitter.com/lambdaisland">on Twitter</a> if you feel I’m
misrepresenting things.</p><h2 id="desktopmobile-offerings">Desktop/mobile offerings</h2><h3 id="quilhttpquilinfo"><a shape="rect" href="http://quil.info/">Quil</a></h3><p>Platforms: desktop, web<br clear="none" />
Based on: Processing / Processing.js<br clear="none" />
Maturity: 👴<br clear="none" />
Documentation: 😊<br clear="none" /></p><p>Quil is Clojure’s version of the graphics programming toolkit Processing. It’s
beginner friendly, and really lends itself to exploration, which is also why
ClojureBridge chose it for the official curriculum.</p><p>It’s been around for a while and “just works”. The project site has decent API
docs, there’s a wiki with useful info, and you can find plenty of examples
floating around, like the ones I created last year for
<a shape="rect" href="http://landofquil.we-do-fp.berlin/">The Land of Quil</a>, or the ones that are
being worked on as part of the
<a shape="rect" href="https://github.com/clojurebridge-berlin/study-group-curriculum">ClojureBridge Berlin Study Group Curriculum</a>.</p><p>It’s also the only offering on this list that’s actually cross platform. You do
have to keep a few small things in mind, but it’s very doable to have a single
code base work for both desktop and in the browser, which is pretty cool.</p><p>The only downside is that for game development Quil is pretty limited. Its sweet
spot lies more in generative art and interactive graphics. You get a bunch of
drawing primitives, and some functions for mouse and keyboard input, but beyond
that you’re on your own. No sprites, physics, collision detection, scenes,
layers, sound, tweening, or event system. It’s still a lot of fun to play around
with, but if you only have a weekend then maybe tone down your ambitions.</p><h3 id="play-cljhttpsgithubcomoakesplay-clj"><a shape="rect" href="https://github.com/oakes/play-clj">Play-clj</a></h3><p>Platforms: desktop, Android<br clear="none" />
Based on: <a shape="rect" href="https://libgdx.badlogicgames.com/">libGDX</a><br clear="none" />
Maturity: 👩<br clear="none" />
Documentation: 😊<br clear="none" /></p><p>Play-clj is Zach Oakes game framework for Clojure. There’s a big warning on the
README that he’s focusing his efforts now on Play-cljs, which I’ll cover further
down. Zach is the author of <a shape="rect" href="https://sekao.net/nightcode/">Nightcode</a>, a
beginner friendly editor/IDE for Clojure. He’s also created a Nightcode spin-off
called <a shape="rect" href="https://sekao.net/nightmod/">Nightmod</a>, specifically for game
development using Play-clj. It calls itself “a tool for making live-moddable
games”. The idea is clear: bring REPL-like interactive development to game dev.</p><p>Despite the warning Play-clj seems pretty solid. It’s the only offering in this
list that explicitly mentions mobile as a target. Play-clj games use a
<a shape="rect" href="https://en.wikipedia.org/wiki/Entity%E2%80%93component%E2%80%93system">Entity-Component System</a>
which is pretty standard in game development nowadays. It’s based on libGDX
which is a feature rich game dev framework, and as such provides many features
you’d expect, like sprites or physics simulation.</p><p>There’s also comprehensive documentation, not just API docs but also (bless the
author) and actual step-by-step tutorial. It doesn’t seem like much to ask but
it’s a rare offering in the documentation wasteland you’ll encounter further
down.</p><p>Oh and Zach (I can say Zach, right?) spoke about Play-clj at Clojure/conj!
<a shape="rect" href="https://www.youtube.com/watch?v=0GzzFeS5cMc">Check out the talk: Making Games at Runtime With Clojure</a>.</p><h3 id="arcadiahttpsgithubcomarcadia-unityarcadia"><a shape="rect" href="https://github.com/arcadia-unity/Arcadia">Arcadia</a></h3><p>Platforms: desktop<br clear="none" />
Based on: <a shape="rect" href="https://unity3d.com/">Unity</a><br clear="none" />
Maturity: 👦<br clear="none" />
Documentation: 🤔<br clear="none" /></p><p>Arcadia is definitely the odd in one out in this list, but it’s such a cool and
promising project that it wouldn’t stand not to mention it. Most readers of this
blog will know that there is a JVM and a Javascript based version of Clojure,
but there’s a lesser known implementation called ClojureCLR, targeting
Microsoft’s Common Language Runtime, the virtual machine that powers .NET.</p><p>This is what enabled Arcadia to integrate Clojure with Unity, a bona fide
A-level professional game development environment. This also means the workflow
is a bit other than usual. Unity comes with its own coding environment, and to
get started with Arcadia you clone the Arcadia source into your Unity project,
although this might change as the project matures.</p><p>Unity is a commercial offering, it is not free software/open source, but there’s
a free plan for hobby use.</p><p>Documentation is still limited, but there are a handful of wiki pages that do
seem to contain all you need to get rolling.</p><p>Arcadia is development by <a shape="rect" href="https://twitter.com/timsgardner">Tim Gardner</a> and
<a shape="rect" href="https://twitter.com/ra">Ramsey Nasser</a> from
<a shape="rect" href="http://kitchentablecoders.com/about/">Kitchen Table Coders</a>, the same group
that <a shape="rect" href="https://twitter.com/swannodette">David Nolen</a> is part of.</p><h2 id="browser-based">Browser based</h2><p>Now we come to the web part, games that are playable directly in your browser.
Note that we already mentioned Quil at the top. Since it’s cross platform it
really belongs in both categories.</p><p>To evaluate these libraries it’s good to first have a look at what powers them
under the hood. Whatever superpowers they have will be inherited directly from
whatever javascript library is driving them.</p><p>Some of these only provide a thin wrapper for Clojurescript, meaning they don’t
bring anything new to the table, but merely provide a more idiomatic syntax,
avoiding lots of interop calls. Others try to add value of their own.</p><h3 id="play-cljshttpsoakesgithubioplay-cljs"><a shape="rect" href="https://oakes.github.io/play-cljs/">Play-cljs</a></h3><p>Platforms: browser<br clear="none" />
Based on: <a shape="rect" href="http://p5js.org/">p5.js</a>
Maturity: 👶<br clear="none" />
Documentation: 🤔<br clear="none" /></p><p>P5.js is a reimagination of Processing.js, the library Quil is based on. It
wants to “make coding accessible for artists, designers, educators, and
beginners”. That’s a great goal, but just like with Quil this means that handy
features specifically for game development are a bit thin on the ground.</p><p>Play-cljs seems to mostly be a thin wrapper around p5.js, although it’s possible
the author plans to build more gaming features in. Note that despite the one
letter difference, play-clj and play-cljs are completely different animals, with
completely different feature sets and APIs.</p><p>Zach tends to do a pretty good job at documenting things, although there’s no
step-by-step tutorial in this case. You do get a bunch of example games, and
generated API docs, which for such a minimal library really might be enough.</p><h3 id="chocolatierhttpsgithubcomalexkehayiaschocolatier"><a shape="rect" href="https://github.com/alexkehayias/chocolatier">Chocolatier</a></h3><p>Platforms: browser<br clear="none" />
Based on: <a shape="rect" href="http://www.pixijs.com/">Pixi.js</a><br clear="none" />
Maturity: 👶<br clear="none" />
Documentation: 😠<br clear="none" /></p><p>Chocolatier seems pretty impressive, it’s based on Pixi.js which prizes itself
at being “the fastest, most flexible 2D WebGL renderer”. It has few hundred
Github stars, and it seems a lot of thought has gone into its design. The author
even did a
<a shape="rect" href="https://www.youtube.com/watch?v=TW1ie0pIO_E">conj talk about Functional Game Design</a>,
and the README talks a lot about its “modified entity component system”, but I’m
really missing a gentler introduction on how to actually <em>use</em> the thing.</p><p>There’s a big example in the README that’s more or less provided without
comment. There’s a documentation folder, with a single file <code>Intro.md</code>,
containing a single line: “TODO: write great documentation”.</p><p>It’s really unfortunate, I feel like if I had a day or two to sink my teeth into
this, preferrably well rested and with few distractions, I could get to a point
where I could have a lot of fun with it. Sigh… oh well, I guess as a creator
of accessible technical content it’s good to know there’s still plenty of work
to be done.</p><h3 id="phzrhttpsgithubcomdparisphzr"><a shape="rect" href="https://github.com/dparis/phzr">Phzr</a></h3><p>Platforms: browser<br clear="none" />
Based on: <a shape="rect" href="http://phaser.io/">Phaser</a><br clear="none" />
Maturity: 👶<br clear="none" />
Documentation: 🤔<br clear="none" /></p><p>Phzr is just a thin wrapper around Phaser, one of the more popular Javascript
game engines. Documentation is limited, and instead you’re simply referred to
the Phaser docs. This of course is the benefit of thin wrappers, as long as the
underlying library is well documented you’re all golden. So I think this
actually would make a pretty good choice, although the question is if you really
need a wrapper then.</p><h3 id="re-framehttpsgithubcomday8re-framewiki--svg"><a shape="rect" href="https://github.com/Day8/re-frame/wiki">re-frame</a> + SVG</h3><p>Platforms: browser<br clear="none" />
Based on: Reagent/React<br clear="none" />
Maturity: 👦<br clear="none" />
Documentation: 🤔<br clear="none" /></p><p>Not a game library as such, but still something I would consider a valid option.
Browser support for SVG has improved rapidly in recent years, and React hasn’t
missed the boat either. The result is that you get drawing primitives right in
your DOM, which can then be styled and animated, and which generate DOM events that
you can respond to.</p><p>In that sense it’s no less basic than Quil or play-cljs. You could use and React
wrapper like Reagent, Om, Quiescent, or Rum, but my favorite by far is re-frame,
and I believe its event system is uniquely suited for use with games.</p><p>And re-frame has some pretty decent docs. It’s a bit of a weird style, and the
README is a bit of work to get through, but it does explain well how it works,
and there are wiki pages explaining all the core concepts. Some even say the
re-frame docs are like the hidden documentation for Reagent, the library
re-frame is built on. Some Reagent concepts are only properly explained here.</p><p>You might have to be a bit more performance-conscious than with other options,
but make good use of React/Reagent’s optimizations to avoid unnecessary rendering, and keep in mind that
<a shape="rect" href="https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/">some things are cheaper to animate than others</a>.</p><p>Finally if I would go this route I would combine it with the excellent
<a shape="rect" href="https://github.com/thi-ng/geom/">thi.ng/geom</a> 2D/3D geometry toolkit for
Clojure/Clojurescript. The same goes for Quil or Play-cljs.</p><h3 id="javascript-engines">Javascript engines</h3><p>Platforms: browser<br clear="none" />
Maturity: 👵<br clear="none" />
Documentation: 😃<br clear="none" /></p><p>Finally a very compelling option is to simply forget about Clojurescript
specific libraries, and instead pick any of the popular Javascript game engines.
There are
<a shape="rect" href="https://html5gameengine.com/">literally dozens of high quality libraries out there</a>.
Forget about being idiomatic, just interop the shit out of it!</p><p>The benefits are obvious: you get a battle tested foundation backed by a large
community, and actual documentation. Just use it as is, and wrap as you go where
it makes sense.</p></<>>Using ClojureScript with Google Apps Scripthttps://lambdaisland.com/blog/2016-10-01-clojurescript-and-google-apps-script2016-10-01T11:10:33+00:00<<>><p><img src="/img/game-of-life.gif" alt="" /></p><p><em>Game of Life in Google Sheets, written in ClojureScript.
<a shape="rect" href="https://github.com/plexus/gas-of-life">Github</a> and
<a shape="rect" href="https://docs.google.com/spreadsheets/d/1ddK2Oh4NdhQ19nFbstI5wIDl39FdBjV3U72bQ5yjxj8/edit#gid=0">demo</a></em></p><h2 id="background-story">Background story</h2><p>At ClojureBridge Berlin we put a lot of work into managing the sign-up process
for attendees. With workshops of this kind it’s common for people to sign up but
not actually show up. We’ve figured out a process to make sure we have nearly
100% attendance, but there’s a lot of manual work involved to make it work.</p><p>We use Google Forms for the registration. We typically get two to three times
more sign-ups than we have space for. About three weeks before the workshop the
registration closes, and we invite the first batch of attendees, selected
randomly. These people get an email giving them a week to confirm that they’re
still able to come. The rest are informed that they’re on the waiting list.</p><p>A week later we know how many spots have opened up, either because people
cancelled or because they didn’t respond, and we can invite a second batch. We
keep repeating this process until days before the workshop.</p><p>All of this is done by manually keeping multiple spreadsheets based on email
conversations… not a great experience.</p><p>The weekly ClojureBridge Berlin project group is working on a web app that will
make this a lot smoother, but that won’t be finished in time for the upcoming
workshop, so I decided to investigate if we could automate some of the things we
are currently doing inside Google Sheets.</p><h2 id="google-apps-script">Google Apps Script</h2><p>If like many of us you’re also trapped in Google’s golden cage, you might be
interested to know just how much of Docs, Sheets, and other Google services is
scriptable. “<a shape="rect" href="https://developers.google.com/apps-script/">Google Apps Script</a>”
is what Google calls third party JavaScript that runs on their servers and
interacts with their services.</p><p>There are two ways to get started with GAS. The easiest is to open a
spreadsheet, form, or word processing document, and go to Tools > Script editor.
This will create a “bound” script, one that’s linked to the document. This has
some limitations, but for personal use it’s the easiest option.</p><p>The other option is to go to http://script.google.com/ and create a new
stand-alone project. You’ll need this is you want to write add-ons that others
can add to their documents. You can also create Google Chrome add-ons, or even
actual web apps that run on Google’s cloud.</p><p>I’m going to focus on sheets. One thing you can do is create custom functions
that you can use as formulas in cells, “macros” in spreadsheet lingo. You can
also add custom menus, add a sidebar, and display modal or non-modal popups.</p><p>As entry point there are time and event based “triggers” you can hook into, like
“onOpen”, “onChange”, or “onSubmit”.</p><p>So to make this concrete, my plan is to create a sidebar in the “onOpen”
trigger, which will contain a small UI for filtering attendees and updating
their status.</p><h2 id="its-just-javascript">It’s just JavaScript</h2><p>While Google calls it “Google Apps Script” and insists you use the “.gs”
extension, it’s really just JavaScript, and so we can use ClojureScript just
fine. For instance, to insert a row of data you could do this</p><pre><code class="language-clojure">(.. js/SpreadsheetApp
getActiveSheet
(appendRow #js ["hello", "world"]))
</code></pre><p>Note that you do need to be
<a shape="rect" href="https://lambdaisland.com/episodes/clojurescript-interop">up to speed with js interop</a>.
The great thing is that with a bit of sugar you get nice idiomatic ClojureScript
doing what it’s good at, processing data.</p><pre><code class="language-clojure">(defn current-sheet []
(->> (.. js/SpreadsheetApp
getActiveSheet
getDataRange
getValues)
array-seq
(map array-seq)))
(filter (fn [[name age]]
(< age 18))
(current-sheet))
</code></pre><p>The <a shape="rect" href="https://developers.google.com/apps-script/overview">guides</a> and
<a shape="rect" href="https://developers.google.com/apps-script/reference/calendar/">API docs</a> for
GAS are pretty good, so go there if you want to figure out how to do something.</p><p>To get it working I created a small project with just a single ClojureScript build.</p><pre><code class="language-clojure">;; project.clj
(defproject attendomat "0.1.0-SNAPSHOT"
:license {:name "Mozilla Public License 2.0" :url "https://www.mozilla.org/en-US/MPL/2.0/"}
:dependencies [[org.clojure/clojure "1.9.0-alpha13"]
[org.clojure/clojurescript "1.9.229"]]
:plugins [[lein-cljsbuild "1.1.4"]]
:cljsbuild {:builds
{:main {:source-paths ["src"]
:compiler {:main attendomat.core
:optimizations :advanced
:output-to "export/Code.gs"
:output-dir "target"
:pretty-print false
:externs ["resources/gas.ext.js"]
:foreign-libs [{:file "src/entry_points.js"
:provides ["attendomat.entry-points"]}]}}}})
</code></pre><p>Now my code goes in <code>src/attendomat/core.cljs</code>, and I compile it with <code>lein
cljsbuild once main</code>. The resulting <code>Code.gs</code> I need to manually copy over to
Google’s script editor.</p><p>Those last three lines of <code>project.clj</code> can use a bit of clarification. I’m using advanced compilation, so I need externs definitions for the
GAS API stuff.(If you’re not sure what those externs are for, check out <a shape="rect" href="https://lambdaisland.com/episodes/javascript-libraries-clojurescript">Using JS libraries from ClojureScript</a>) Luckily someone has already gone ahead and scraped the API docs
to generate this externs file. You can grab it at
<a shape="rect" href="github.com/tyskdm/gas.ext.js">https://github.com/tyskdm/gas.ext.js/blob/master/dist/0.7.2/gas.ext.js</a>.</p><p>We also need to tell the compiler to leave the top-level functions alone, using
the <code>^:externs</code> metadata annotation.</p><pre><code class="language-clojure">(ns attendomat.core
(:require [attendomat.entry-points]))
(defn ^:export create-menu []
(.. js/SpreadsheetApp
getUi
(createMenu "ClojureScript")
(addItem "Select attendee", "popup_attendee_selector")
(addToUi)))
</code></pre><p>Now that function will be available in JavaScript as
<code>attendomat.core.create_menu</code>, but Google expects to find simple non-namespaced
function names, that’s where the “entry-points” stuff comes in.</p><p>Create a simple JavaScript file under <code>src/entry_points.js</code>, for example</p><pre><code class="language-javascript">function onOpen(e) {
attendomat.core.create_menu();
}
</code></pre><p>By adding it under <code>:foreign-libs</code> this code will be prepended unchanged to our
final build artifact (see <code>project.clj</code> above).</p><h2 id="importing--exporting-code">Importing / exporting Code</h2><p>You may be wondering if there’s really no better way than to copy-paste that
code over after every compilation. One small trick is to use a command line
clipboard interface. On linux there’s <code>xclip</code> or <code>xsel</code>, so my build command
looks like this.</p><pre><code>lein cljsbuild once main ; cat export/Code.gs | xclip -selection clipboard
</code></pre><p>At least that takes part of half of the copying and pasting.</p><p>Now there are tools like
<a shape="rect" href="https://github.com/danthareja/node-google-apps-script">this one that runs on Node.js</a>,
that will use the Google Drive API to push and pull your project files. Neat!
But… it only works for standalone scripts, not for those bound to a document.
To get them onto a document you need to jump through some other hoops, so in the
end I decided it wasn’t worth it for me. However for developing commercial
add-ons this could be a great workflow. Combine Clojure’s productivity with what
this platform has to offer and you could have a winner on your hands.</p><h2 id="reflections">Reflections</h2><p>I definitely have mixed feeling developing for this platform, it brings back
memories of doing Microsoft-only stuff back in the 90’s. Not that I’m not
already up to my neck in Google’s muck, but yeah…</p><p>One thing that’s really still missing in an interactive workflow. Forget about a
REPL, your feedback cycle is now: compile, copy, paste, click, click, click,
plus a lot of <code>println</code> debugging (or <code>Logger.log</code> debugging, to be precise). It doesn’t help that the code is basically obfuscated. I tried with <code>:optimizations :whitespace</code>, and the result is too big, my browser wouldn’t let me copy it into the text field. What you can do is enable pretty-printing. When an error occurs all you get is a line number, so having expressions each on their own line really helps then.</p><p>Testing locally is not an option unless you mock out all of Google’s API, and
what would be the point of that? I don’t immediately see a solution upcoming,
given how the code needs to be uploaded and executed on Google’s servers, and
given the limits of the import/export API.</p><h2 id="related-projects">Related projects</h2><p><a shape="rect" href="https://github.com/johnmn3/clgs">johnmn3/clgs</a> uses bootstrapped ClojureScript
so you can use ClojureScript expressions in cell formulas. There’s a bit more
info on
<a shape="rect" href="https://groups.google.com/forum/#!topic/clojurescript/d3fRb5_rkeA">this thread</a>
on the Clojure Google Group.</p></<>>Union Types with Clojure.Spechttps://lambdaisland.com/blog/2016-09-25-union-types2016-09-27T14:32:12+00:00<<>><p>Elm and other statically typed languages have a great feature called Union Types
(also called Sum Types or Algebraic Data Types).</p><p>Here’s an example taken from Elm. Suppose your system used to represent users as
integers, maybe just an auto-incrementing primary key, but then switched to
UUIDs represented as strings.</p><p>To correctly model this situation, you need a way to create a type that can be
either an integer or a string, that’s what union types give you.</p><pre><code>type UserID = OldID Int | NewID String
</code></pre><p>Now each time you want to do something with a user id, you’ll have to add code
to cover all these cases. This you can do with Elm’s <code>case ... of</code>. The
interesting thing is that if you miss one of the cases, then the compiler will
complain. Say in the future you decide to use a reified UUID type, now the code
won’t compile until you’ve handled that case everywhere you’re dealing with user
ids.</p><pre><code>toNewID : UserID -> String
toNewID userID =
case userID of
OldID number -> toString number
NewID string -> string
</code></pre><p>With clojure.spec we get the same power of expression, here’s how a <code>::user-id</code> spec would look like.</p><pre><code class="language-clojure">(require [clojure.spec :as s])
(s/def ::user-id (s/or :old-id int?
:new-id string?))
</code></pre><p>Consuming this is straightforward, especially with <a shape="rect" href="https://github.com/clojure/core.match">core.match</a>.</p><pre><code class="language-clojure">(require [core.match :refer [match]])
(defn format-id [user-id]
(match [(s/conform ::user-id user-id)]
[:old-id id] (str "Old id: " id)
[:new-id id] (str "New id: " id)
:else "Invalid id"))
</code></pre><p>So we get pretty much the same functionality, even the syntax is still pretty
clean, but there’s one crucial difference: we don’t get compile time, or even
runtime checks to make sure we’re covering all cases.</p><p>I thought it would be a fun excercise to implement a <code>case-of</code> macro, that
during macro-expansion checks to see if all cases are covered. This is what it
looks like in action.</p><pre><code class="language-clojure">(require '[lambdaisland.uniontypes :refer [case-of])
(s/def ::availability (s/or :sold-out #{:sold-out}
:in-stock pos-int?
:reordered (s/tuple pos-int? pos-int?)
:announced string?))
(defn format-availability [a]
(case-of ::availability a
:sold-out _
"Sold out."
:in-stock amount
(str "In stock: " amount " items left.")
:reordered [min max]
(str "Available again in " min " to " max "days" )
:announced date
(str "Will be available on: " date)))
</code></pre><p>The first argument is the name of a spec, the second is the value to check.
After that you give the name of a case in the <code>s/or</code> spec, a binding form that
will receive the conformed value, you can also use destructuring, as in the
<code>:reordered</code> case, and finally the code that handles the given case.</p><p>If you have too many or too few cases, your code will not compile, and you’ll get an error message that looks like this.</p><pre><code>The cases in this `case-of` are different from the ones in the spec:
(s/def :lambdaisland.uniontypes-test/availability
(s/or :sold-out #{:sold-out}
:in-stock pos-int?
:reordered (tuple pos-int? pos-int?)
:announced string?))
Add a case for :announced.
Remove the case :foo.
</code></pre><p>If the given value does not conform to the spec, it’ll throw an exception. You can override that behavior by adding a special <code>:spec/invalid</code> case. (Note: that’s <em>not</em><code>:clojure.spec/invalid</code> for… reasons).</p><p>The given variable will in this case be bound to the result of
<code>clojure.spec/explain-data</code>.</p><p>The code is <a shape="rect" href="https://github.com/lambdaisland/uniontypes">on Github</a> as well as <a shape="rect" href="https://clojars.org/lambdaisland/uniontypes">Clojars</a>.</p><pre><code class="language-clojure">[lambdaisland/uniontypes "0.1.0"]
</code></pre><h2 id="use-cases">Use cases</h2><p>It might not be immediately obvious why this compile time check is so important, so let me give another example.</p><p>You have a chat application, each channel contains messages, and notifications
like people coming and going. In your application state this is modeled as a
single heterogeneous sequence. While rendering you need to loop over this
sequence, and render the right component depending on the type of each entry.</p><p>The same kind of data is used to display notifications, to update a search
index, and to aggregate statistics.</p><p>The new product manager decides you really need picture uploads, so that becomes
a new type of entry. Now you need to update all of the places in your code where
you deal with entries. Normally you would try to <code>grep</code>, or just do it from
memory, and hope for the best. If you were using <code>uniontypes</code>, you could be sure
you weren’t forgetting any of these cases.</p><h2 id="clojurescript-support">ClojureScript support</h2><p>So does it work on ClojureScript you ask… good question! Yes and no. I tried
hard to make it compatible with ClojureScript, and that threw more than one
spanner in the works. I got it working though, in the sense that <code>lein doo
phantom</code> tells me all the tests pass on ClojureScript (there aren’t many, but
enough to test at least a happy path and some error conditions).</p><p>Any attempt to actually use it on a ClojureScript project went awry,
unfortunately.</p><p>One of the reasons it’s hard to do this on ClojureScript is that the spec is
needed both at Compile time, to check that you have all the branches, and at
runtime, to validate and conform the value. That means the spec provided by the
end user needs to be available both on Clojure and ClojureScript, in other
words, it’s necessary to put your specs in <code>cljc</code> files.</p><p>That alone isn’t always enough to also make sure the spec is loaded during
macroexpansion, so the library will try to load the namespace based on the name
of the spec. So if you do <code>(case-of :foo.bar/baz)</code>, then during macroexpansion
this will try to load <code>foo.bar</code>. This worked to get the tests going, but might
also be what’s causing issues. It’s not the greatest hack as far as hacks go.</p><p>Another “interesting” issue I ran into was the difference between <code>clojure.spec</code>
and <code>cljs.spec</code>. ClojureScript has gotten smarter about this, in the sense that
when you <code>(require '[clojure.spec :as s])</code> it will actually require <code>cljs.spec</code>,
but when writing macros you need to make sure manually that the symbols you’re
generating are in the right namespace.</p><p>What I ended up doing was detecting if the macro was being expanded for use in ClojureScript or not using the <code>(:ns &env)</code><a shape="rect" href="https://groups.google.com/forum/#!topic/clojurescript/iBY5HaQda4A">trick</a>, and then doing a <code>clojure.walk/prewalk</code> over the generated code, to replace all symbols and keywords in the <code>clojure.spec</code> namespace with their cousins in <code>cljs.spec</code>.</p><h2 id="reflections">Reflections</h2><p>I thought this would be one evening hack, it turned out to be a bit more
involved than that. I’m a bit bummed that ClojureScript support isn’t fully
there yet, as I think this could be especially useful in UI programming. The
fact that it’s working for the restricted case of running tests on Phantom.js
does mean that it’s compatible in principle, if anyone cares to do the final
stretch of troubleshooting.</p><p>I need to dogfood this some more to see how useful it will be in practice, but I
think it does show that there are still a lot of imaginative uses of
clojure.spec left to be uncovered.</p></<>>