Introduction to Luminus, part 1

To give you a starting point for building your own app, we’ll go through the process of building an application from scratch with Luminus. In this first part in a series of three, you’ll learn to set up the database, write migrations to create tables with Migratus, create SQL queries with HugSQL, and tie it all together with Compojure routes and Selmer templates.

Transcript

One of the challenges when starting out with Clojure is piecing together a web development stack. There is no single dominant framework like Rails in the Ruby world, or Django for Python. Instead the focus is on small composable libraries. Figuring out which ones you need, what they do, and how to put them together can be a real challenge.

The Luminus project seeks to remedy this. It gives you everything you need to get started in a single package. Luminus comes with comprehensive documentation, including information on security, testing, and how to deploy your app to production.

In this episode, and the next one, I’ll take you on a quick tour through Luminus by building a simple wiki application.

lein new luminus lambwiki +postgres

Luminus comes as a template for leiningen, so I’m generating my application skeleton with the lein new command. I’ll be using PostgreSQL to store the data. It’s a tried and trusted database that I’m confident to use both during development and in production.

cd lambwiki
l
env
log
resources
src
target
test
Procfile
profiles.clj
project.clj
README.md

The directory layout is pretty standard. Most code lives under the src directory. Configuration that only relates to a specific environment, such as development, or production, goes under env. The resources directory contains everything that isn’t clojure code, like SQL queries, HTML templates, CSS, and Javascript.

createdb lambwiki_dev
emacsclient -n profiles.clj

;; profiles.clj
{:profiles/dev  {:env {:database-url "jdbc:postgresql://localhost/lambwiki_dev?user=arne&password=arne"}}
 :profiles/test {:env {:database-url "jdbc:postgresql://localhost/lambwiki_test?user=arne&password=arne"}}}

I will create a database to store our data using createdb, a command line utility provided by Postgres.

The app needs to know how to connect to the database. In production you would typically pass the database URL in as an environment variable. During development you can use leiningen’s profiles.clj. This file is not committed to git, so it’s perfect for setup that only pertains to your local environment.

cd lambwiki
git init
git add -A
git commit -m 'lein new luminus lambwiki +postgres'

I like to already commit this generated code to Git, so I can easily see later on which parts I added myself. Just initialize the repository, add all files, and commit it with the leiningen command as a comment.

cider-jack-in
(start)

I’ll jack in CIDER to have a REPL to work with. Yeah I did speed that part up a bit. Start the app and browse to port 3000. You should see this Luminus welcome page. So far so good. I’ll get rid of this boilerplate content, and clean up the template a bit.

<!-- resource/templates/base.html -->
    <meta charset="utf-8">
    <meta http-equiv="x-ua-compatible" content="ie=edge">

    {% style "https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.1.8/semantic.css" %}

...

<nav class="ui inverted menu">
  <div class="ui container">
    <a href="/" class="item">Home</a>
  </div>
</nav>

<div class="container">
  {% block content %}
  {% endblock %}
</div>

The base.html template is what some web frameworks call a layout, it contains the things that needs to be on every page. Other templates then “extend” the base template to add their specific content.

The templating system used by Luminus is called Selmer, and its syntax is inspired by the Django templating system.

I’m not a huge fan of Bootstrap, so I’ll replace it with Semantic UI. Nice, reduced to its essence.

<!-- home.html -->
  <div class="ui container">
    <h1 class="ui header">Coming soon: wiki!</h1>
  </div>

In the home.html template just put some HTML as a sentinel. There, that looks better. A blank slate! It looks a bit too empty. I’ll add a navigation bar at the top so it looks a bit more like an actual site.

After committing these changes to git, it’s time to move on to the database.

lein migratus create add-pages-table

Migratus is a migration library bundled with Luminus. Each change to the database schema is stored in a migration file, so that the database on each developer’s machine, as well as the database in production, can all stay in sync.

A database without tables is like a namespace without vars. With lein migratus create you can generate a new empty migration file. Inside it create tables for pages and page revisions.

-- 20160426172359-add-pages-table.up.sql
CREATE TABLE pages (
    id SERIAL PRIMARY KEY,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
    uri_slug TEXT NOT NULL UNIQUE,
    title TEXT NOT NULL
);
--;;
CREATE TABLE revisions (
    id SERIAL PRIMARY KEY,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
    page_id INTEGER NOT NULL REFERENCES pages(id),
    body TEXT
);

It’s a good practice to have a numeric id as a primary key, and to timestamp columns to each table to track when exactly a row was inserted. This will also be useful later on for retreiving the most recent revision of a page. A page will have a title, and also a URI slug, the path of its URI.

The pages table doesn’t have a body field, instead I’ll create a new revision with the new body whenever a page changes. Immutability for the win!

Defining both tables in the same file like this is fine, but you need to separate the statements with dash-dash-semicolon-semicolon so migratus makes sure to split them up.

-- 20160426172359-add-pages-table.down.sql
DROP TABLE revisions;
--;;
DROP TABLE pages;

For completeness I’ll add the “down” migration as well, so it can be rolled back in case things go wrong.

lein run migrate

You can run the app with the argument “migrate” to run the migrations. You could use the Migratus leiningen plugin to do the same thing. The benefit of having this built-in to the app itself is that you can easily run your migrations in production as well, even if you don’t have leiningen available there.

git add -A resource/migrations
git commit

Also don’t forget to commit these changes.

-- resources/sql/queries.sql
-- :name create-page! :insert :raw
INSERT INTO pages (uri_slug, title) VALUES (:uri_slug, :title)

Now that the tables are there, it’s time to put some data in them. Open the queries.sql and add a new INSERT statement.

Database access in Luminus is based on HugSQL. You write queries in plain SQL, and based on the metadata provided in comments these get turned into Clojure functions. You use Clojure keyword syntax to define placeholders in the query that can be filled in later. HugSQL does not support positional arguments, so placeholders are always named.

(conman/bind-connection *db* "sql/queries.sql")

This line in db.core namespace is responsible for turning the SQL into Clojure. Make sure to evaluate it again whenever you make changes to the queries.sql file. Conman is just a thin wrapper around HugSQL providing some extra convenience. Without it you would have to pass in the database connection to each query function.

(lambwiki.db.core/create-page! {:uri_slug "home", :title "Home"})
IllegalArgumentException db-spec mount.core.DerefableState@19bbc000 is missing a required parameter  clojure.java.jdbc/get-connection (jdbc.clj:292)

Now try to create a new page from the REPL. Oops, what happened there? This error message is coming from Mount. Luminus uses mount to manage application state, like starting and stopping the web server, or connecting to the database. In this case our application isn’t using the database yet, so Mount didn’t bother to start it. You can start it manually now. Once you start using the query functions in code this problem will simply go away.

(mount/start #'lambwiki.db.core/*db*)
(lambwiki.db.core/create-page! {:uri_slug "home", :title "Home"})
;;=> {:id 1, :created_at #inst "2016-04-28T12:43:00.521-00:00", :uri_slug "home", :title "Home"}

Great, you’ve created your first page! But the body of the page is stored in a revision, so you need one of those as well.

-- :name create-revision! :insert :raw
INSERT INTO revisions (page_id, body) VALUES (:page_id, :body)

(lambwiki.db.core/create-revision! {:page_id 1, :body "Welcome Home"})
;;=> {:id 1, :created_at #inst "2016-04-28T12:43:17.366-00:00", :body "Welcome Home", :page_id 1}

Add the query to queries.sql, call bind-connection again, and now you should be able to create a revision.

{% extends "base.html" %}
{% block content %}
<div class="ui container">
  <h1 class="ui header">{{title}}</h1>
  {{body|markdown}}
  <hr>
  <a class="ui button" href="/{{uri_slug}}/edit">Edit page</a>
</div>
{% endblock %}

Now you can finally start showing some stuff. Rename home.html to wiki-page.html. Render the page title, as well as the body. The body uses Markdown syntax, so use the markdown filter ot convert it to html.

(defn render-wiki-page [page]
  (layout/render
   "wiki/page.html" page))

(defroutes home-routes
  (GET "/" []
    (found "/home"))
  (GET "/:slug" [slug]
    (render-wiki-page {:title "The title" :body "The body" :uri-slug slug})))

Finally in the routes is where it all comes together. You can repurpose the home-routes to implement the wiki functionality.

The wiki-page template will receive a map containing the page title, body, and uri_slug. The “home” page you already created from the REPL will be the landing page, so I’m just going to put in a redirect from the root of the domain to “home”.

Uh right, there’s no query yet to retreive the page from the database. Ok, fine. Just start with some dummy data to see that things are working.

Oh and one more thing, this found function here isn’t available in this namespace yet. It needs to be added to the namespace declaration. I know it’s defined somewhere in Ring, and I could search the documentation, but I’m lazy, so I’m going to use clj-refactor. This will automatically add the missing declaration. Honestly I don’t know how I would survive without it.

Let’s check the browser…. uhhh seems it’s not returning HTML…. because I forgot to render the template. And… there. Magic!

-- :name find-page-by-uri-slug :query :one
SELECT pages.*, revisions.body
FROM pages, revisions
WHERE pages.id = page_id
AND uri_slug=:uri_slug
ORDER BY revisions.created_at DESC
LIMIT 1

This final query is the most complicated. It uses a join to retreive the page data together with the body of the most recent revision. See how that timestamp comes in handy now. Because of the annotations in the comment above the query, HugSQL knows this is a regular select query that returns one row.

Does that work? Looks like it does.

(defn render-wiki-page [page]
  (layout/render
   "wiki/page.html" page))

(defroutes home-routes
  (GET "/" []
    (found "/home"))
  (GET "/:slug" [slug]
    (when-let [page (find-page-by-uri-slug {:uri_slug slug})]
      (render-wiki-page page))))

All that’s left is to update the route to use the query. And of course to require it in the namespace declaration.

That’s it for now. In the next episode of Lambda Island you’ll find out how to edit and create new wiki pages. Till next time!

Show Notes

browse source code

!!WARNING!! High pacing

The pace of this video is very high. Pause often, keep the transcript handy, and refer to the code on GitHub if necessary.

This is less of an issue in following videos, and should no longer be an issue starting episode 10. I’ll redo these Luminus episodes at some point. Until then either skip to the later episodes or try to make the best of it.

Docs

The Luminus docs should be your first port of call, they have information on all the main pieces of your application.

Migrations are powered by Migratus, and database queries by HugSQL. Check their respective project pages if you want the full low-down.

The CSS system I’m using is Semantic UI. It’s not particularly special, Bootstrap or Foundation would do the trick just as well. I just like it’s clear and modular approach.

Versions

The software used in this episode

$ lein --version
Leiningen 2.6.1 on Java 1.8.0_91 Java HotSpot(TM) 64-Bit Server VM

  :dependencies [[org.clojure/clojure "1.8.0"]
                 [selmer "1.0.4"]
                 [markdown-clj "0.9.88"]
                 [ring-middleware-format "0.7.0"]
                 [metosin/ring-http-response "0.6.5"]
                 [bouncer "1.0.0"]
                 [org.webjars/bootstrap "4.0.0-alpha.2"]
                 [org.webjars/font-awesome "4.6.1"]
                 [org.webjars.bower/tether "1.1.1"]
                 [org.webjars/jquery "2.2.2"]
                 [org.clojure/tools.logging "0.3.1"]
                 [compojure "1.5.0"]
                 [ring-webjars "0.1.1"]
                 [ring/ring-defaults "0.2.0"]
                 [mount "0.1.10"]
                 [cprop "0.1.7"]
                 [org.clojure/tools.cli "0.3.3"]
                 [luminus-nrepl "0.1.4"]
                 [org.webjars/webjars-locator-jboss-vfs "0.1.0"]
                 [luminus-immutant "0.2.0"]
                 [luminus-migrations "0.1.2"]
                 [conman "0.5.1"]
                 [org.postgresql/postgresql "9.4-1206-jdbc4"]
                 [luminus-log4j "0.1.3"]
                 [hiccup "1.0.5"]]

GNU Emacs 25.0.92.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7) of 2016-03-30
CIDER 0.13.0snapshot (package: 20160510.1330)