Lambda Island

Datomic is a database based on the same principles that underly the design of Clojure itself. Learn what makes it different, and how to start using from Clojure immediately.

This second part dives deeper into transacting entities and querying them.

Transcript

In the previous episode you created this transaction which adds two entities to Datomic, each with a single :user/name attribute. I showed you that once you dereference the result you get a map back with four keys, the first two of which represent the database before and after the transaction.

The transaction result also contains :tx-data, a list of datoms that were created as part of this tranaction. It can provide some great insights into what Datomic is doing.

Notice that these are exactly the 5-tuples we were talking about earlier.

You inserted two new entities, each with one attribute, so you’d expect to see two new datoms, but Datomic actually created three datoms, for three separate entities.

The first entity is the transaction. Each time you call d/transact, Datomic will create a new transaction entity, and all datoms created in that same transaction will point to it with their transaction field. The transaction stores the timestamp of when this data was inserted. It’s also possible to add other meta-information to this transaction entity, for instance to store a reference to the user who performed this operation.

(:tx-data @tx-result)
;;=>
[#datom[13194139534316 50 #inst "2017-10-12T10:16:31" 13194139534316 true]
 #datom[17592186045421 63 "jillosaurus"               13194139534316 true]
 #datom[17592186045422 63 "jonnyboy"                  13194139534316 true]]
;;            🡑        🡑               🡑                    🡑         🡑
;;          entity   attribute        value             transaction  added?

Another thing you’ll notice is that the attributes are just numbers, but before we always referred to them as keywords. For Datomic entities are just numbers, and since attributes are entities it refers to those by number as well. But you can give an entity a name by assiging it a :db/ident attribute.

The ident function will tell you the name of an entity.

(d/ident (d/db conn) 50)            ;;=> :db/txInstant
(d/ident (d/db conn) 63)            ;;=> :user/name

Idents or entity ids can be used interchangably. The entity function returns a map representation of an entity, notice how it works with keywords or numeric ids just the same.

(d/entity (d/db conn) :user/name)
;;=> #:db{:id 63}

(d/entity (d/db conn) 63)
;;=> #:db{:id 63}

The result of entity is a lazy map, that’s why in a REPL you just see a map with a :db/id, the other attributes only show up when you ask for them.

To fully realize an entity map, use touch, which will load all the attributes. touch is also able to recursively realize entities that this entity references, if they are marked in the schema as components of this entity. I won’t cover components in this episode, but there’s a post on the datomic blog that covers them in depth. I’ll link to it in the show notes.

(:db/valueType (d/entity (d/db conn) :user/name))
;;=> :db.type/string

(d/touch (d/entity (d/db conn) :user/name))
;;=> #:db{:id 63, :ident :user/name, :valueType :db.type/string, :cardinality :db.cardinality/one}

(d/touch (d/entity (d/db conn) 17592186045421))
;;=> {:db/id 17592186045421, :user/name "jillosaurus"}

entity is a great API for certain use cases, but the real powerhouse of Datomic querying is the q function. Before we get there let’s add some more stuff to the database.

First create two more attributes. The :user/friends attribute is of type :ref, meaning it forms a reference to another entity. It has cardinality/many, meaning a user can have multiple friends. This way you can start to model a social graph.

The second attribute, :user/admin?, is a boolean flag to indicate that a user has administrative privileges.

(d/transact conn [{:db/ident       :user/friends
                   :db/valueType   :db.type/ref
                   :db/cardinality :db.cardinality/many}

                  {:db/ident       :user/admin?
                   :db/valueType   :db.type/boolean
                   :db/cardinality :db.cardinality/one}])

Next let’s use these attributes to create two more user entities. Now that users have friends, you’ll need a way to refer to other entities when creating a transaction.

You already learned two ways to refer to entities: directly with their numeric id, or if the entity has a :db/ident attribute, then you can refer to it with a keyword.

This example shows two more ways to refer to other entities, by string, and by “lookup ref”.

“user1” is friends with “user2”, but until user2 has been transacted, it doesn’t have an entity id, and so we have a chicken and egg situation where we need the id of an entity inside the transaction that creates the entity.

The solution is to use strings as temporary ids. By using the same string multiple times in the same transaction, you are now able to reference entities. The transactor will swap out the strings for actual entity ids as it assigns them.

“user2” is friends with “jillosaurus”, this is a user that’s already in the database. You could look up its entity id first and use that in the transaction, but since we told Datomic that usernames are unique, you can also use its username to look it up. This is done with a two element vector as shown here, and is known as a “lookup ref”.

(d/transact conn [{:db/id "user1"
                   :user/name "hugabug"
                   :user/friends #{"user2"}}

                  {:db/id "user2"
                   :user/name "plexus"
                   :user/admin? true
                   :user/friends #{[:user/name "jillosaurus"]}}])

q is a pretty powerful function, and I encourage you to read the documentation on querying in Datomic, and to practice and play around with it. I’m just going to show you the tip of the iceberg: unification of logic variables, and passing in arguments.

Let’s start with a simple example, this query lists the entity id and username of each user, meaning each entity that has a :user/name assigned to it. The first argument to q is the query, it’s just a Clojure vector, but make sure to quote it since it contains the symbols ?e and ?name which should be passed to q directly, instead of being resolved as vars.

If you’ve done logic programming before, for instance with Prolog or core.logic, then things will look familiar. After all, Datomic’s queries are based on Datalog, which is a variant of Prolog that’s designed for querying data.

If you haven’t then no worries, things should become clear soon enough.

The :where keyword is used to define one or more “clauses”. These are basically patterns that can match datoms. This clause will match any datom that has :user/name as attribute.

?e and ?name are logic variables, you can use these in the :find part to specify what should be in the result set.

So this query can be read as “find all values ?e and ?name, such that there is a datom in the database with entity id ?e, attribute :user/name, and value ?name.

(d/q '[:find ?e ?name
       :where [?e :user/name ?name]]
     (d/db conn))
;;=>
#{[17592186045418 "jillosaurus"]
  [17592186045419 "jonnyboy"]
  [17592186045424 "hugabug"]
  [17592186045423 "plexus"]}

If your query has multiple clauses, then all of them need to match. So to find the names of all the admins, you can use this query. The ?e variable now occurs twice in the :where part, so Datomic will try to find values of ?e that match all the clauses.

?e now no longer appears in the :find part, it’s just being used to make it clear to Datomic that the two :where clauses are about the same entity. So this is like an AND operator, it finds entities with a :user/name, AND with :user/admin? being true.

(d/q '[:find ?name
       :where
       [?e :user/name ?name]
       [?e :user/admin? true]]
     (d/db conn))
;;=> #{["plexus"]}

Notice again that these clauses have the same five-tuple structure, except that the transaction and added? flag are missing. You can specify them, but if you don’t care about them then you don’t have to.

(d/q '[:find ?name
       :where
       [?e :user/name ?name]
       [?e :user/admin? true]]
     ;; 🡑        🡑      🡑           🡑         🡑
     ;; entity   attr   value   transaction  added?
     (d/db conn))

Here’s a more advanced example that makes use of the transaction id, see if you can figure out what’s going on. This query asks the question: in the same transaction that made the user “plexus” admin, which other datoms were created.

Whenever a logic variable occurs in more than one clause, you can think of it in SQL terms as a JOIN on that variable.

So the first two clauses have ?u in common, they find each entity that has username “plexus”, and admin? true.

The second and third clause have ?t in common, so it will take all entities that the first two clauses returned, and then find any datoms that have the same transaction id.

The final two clauses join on ?a, the attribute id. This is done to find the :ident, in other words the name of the attribute, instead of just its number.

I’m showing you this example to demonstrate how you can use the transaction id in a query, but this is already a bit of a “meta” example, so don’t worry if it doesn’t fully make sense yet.

(d/q '[:find ?e ?a-ident ?v
       :where
       [?u :user/name   "plexus"]
       [?u :user/admin? true        ?t]
       [?e ?a           ?v          ?t]
       [?a :db/ident    ?a-ident]]
     ;; 🡑        🡑      🡑           🡑         🡑
     ;; entity   attr   value   transaction  added?
     (d/db conn))
;;=>
#{[17592186045423 :user/name "plexus"] 
  [17592186045424 :user/name "hugabug"]
  [17592186045423 :user/admin? true]
  [17592186045424 :user/friends 17592186045423]
  [17592186045423 :user/friends 17592186045418]
  [13194139534318 :db/txInstant #inst "2017-10-12T11:12:38.655-00:00"]}

Just as the transaction id and added? flag can be omitted most of the time, so you can also omit the “value”. This will find all entities with a certain attribute, no matter what the actual value of that attribute is.

This can be useful because in Datomic entities don’t have a fixed type, they are simply a bag of attributes and values, and any attribute can be assigned to any entity.

We tend to think of our data as falling in distinct categories like “users”, “posts”, or “products”, but Datomic allows you to mix and match attributes freely within the same entity, so these type-like semantics are up to you as application developer to decide upon.

In practice the presence of a certain attribute is used to indicate that an entity is of a certain type. For example, your application could treat any entity with a :user/name attribute as “being a user”, and to query for all users means querying for all entities that have a :user/name, regardless of its value.

(d/q '[:find ?e 
       :where
       [?e :user/name]]
     (d/db conn))
;;=> #{[17592186045418] [17592186045419] [17592186045423] [17592186045424]}

Let’s look at an easier example of a “join”. SQL has a special syntax for these, but with Datalog this just falls out of the pattern matching we’ve been doing all along.

This query finds the pairs of usernames such that the first user is friends with the second. The first clause finds all pairs of friends, you could stop there, but you would get entity id numbers back, instead of usernames.

So the second clause goes from the ?user entity to its name, and the third does the same for ?friend.

The process where Datomic tries to find values for these variables that match all clauses is called “unification”.

(d/q '[:find ?username ?friendname
       :where
       [?user   :user/friends ?friend]
       [?user   :user/name    ?username]
       [?friend :user/name    ?friendname]]
     (d/db conn))
;;=> #{["plexus" "jillosaurus"] ["hugabug" "plexus"]}

Finally let’s look at an example that takes arguments. Let’s turn this query into a function called friends-of that, given a database value and a username, finds all the friends of a given user.

Queries are just vectors, meaning they are data you can manipulate, so you might be tempted to build up a query vector dynamically that contains the given username. This is tedious however, but worse it prevents datomic from caching this query.

Instead friends-of will make use of the capabilities that q provides for parameterizing queries.

This query will look very similar to the previous one, but besides the database, it now also takes the username as an extra input argument.

If you pass q only one argument, it will assume it is the database, and so you don’t have to name it, but when you have multiple input arguments your query vector needs to have an :in part that names the different inputs.

$ is a special name given to the default database that the query engine will work with, so you must put that one first. After that you give names to other incoming arguments. So here ?username will now get the value “plexus”.

The rest of the query is largely unchanged, but because the ?username variable has a pre-determined value the results are much more constrained.

(defn friends-of [db username]
  (d/q '[:find ?friendname
         :in $ ?username
         :where
         [?user   :user/friends ?friend]
         [?user   :user/name    ?username]
         [?friend :user/name    ?friendname]]
       db username))

(friends-of (d/db conn) "plexus")
;;=> #{["jillosaurus"]}

Datomic Quickstart, part 2