Skip to main content

Planner

caution

This page is a draft. Feedback is welcome.

Welcome to the heart of Pathom. The planner is the part of Pathom that figures which resolvers to call, in which order to fulfill the data demand.

What is a plan?​

The Pathom plan is a DAG, and always has a single root.

Now we are going to a series of examples to understand how the planner does its job, and what it outputs.

To understand the planner, we going to walk though a series of examples:

The simplest plan​

To compute the plan, Pathom requires:

In case you already have some data available, you can tell the planner about it using the Shape Descriptor, so it will consider that when formulating the plan.

For our first example, it's going to be a simple request that calls a single resolver:

(ns com.wsscode.pathom3.docs.planner
  (:require [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.planner :as pcp]
            [edn-query-language.core :as eql]))

; start with a simple resolver
(pco/defresolver answer []
  {::answer-to-everything 42})

; create an environment with the indexes
(def env (pci/register answer))

; generate execution DAG
(let [ast (eql/query->ast [::answer-to-everything])]
  (pcp/compute-run-graph
    (assoc env
      :edn-query-language.ast/node ast)))

Pathom represents the plan DAG using plain Clojure maps. Our previous example generate this graph:

{::pcp/nodes
 {1
  {::pco/op-name
   com.wsscode.pathom3.docs.planner/answer

   ::pcp/node-id
   1

   ::pcp/requires
   {:com.wsscode.pathom3.docs.planner/answer-to-everything {}}

   ::pcp/input
   {}

   ::pcp/source-for-attrs
   #{:com.wsscode.pathom3.docs.planner/answer-to-everything}}}

 ::pcp/index-resolver->nodes
 {com.wsscode.pathom3.docs.planner/answer #{1}}

 ::pcp/unreachable-resolvers
 #{}

 ::pcp/index-ast
 {:com.wsscode.pathom3.docs.planner/answer-to-everything
  {:type         :prop
   :dispatch-key :com.wsscode.pathom3.docs.planner/answer-to-everything
   :key          :com.wsscode.pathom3.docs.planner/answer-to-everything}}

 ::pcp/root
 1

 ::pcp/index-attrs
 {:com.wsscode.pathom3.docs.planner/answer-to-everything 1}}

To generate the plan, Pathom starts scanning the AST. For each attribute it sees, it looks it up in the index. In case the attribute requires other dependencies, Pathom will recursively walk the index-oir until it finds the matches or reach a dead end.

Now it's time to introduce the planner, viz. This is a tool that lets you visualize a Pathom execution plan visually. This is also a tool to understand and debug how Pathom builds up a plan. Using the items on the left, you can navigate through the graph building process:

note

This tool still in the early stages. Expect changes to it.

tip

You can click the nodes to see details.

Our current case is simple, a single resolver node.

Time to break down some of the attributes here:

::pcp/nodes​

It contains a map with all the nodes from this graph, indexed by node-id. The node-id is a monotonic incremental value, the first node gets the ID 1.

In the planner graph, there are three types of possible nodes:

Resolver nodes​

Represent a call to a resolver

AND nodes​

Represent the call of multiple resolvers, also, being a sibling node in a AND node means the runner can call those nodes in parallel.

OR nodes​

Represent that one of the nodes must get the required data.

::pcp/index-resolver->nodes​

Index pointing to the nodes for a given resolver.

::pcp/root​

The start node of the plan. In the visualization, the root node has a black border.

::pcp/index-attrs​

This index knows which node is responsible for a given attribute. This is important for internal optimizations, when some dependency appears repeated, Pathom can use this index to connect straight to a node, avoiding re-processing.

Other attributes​

You can find spec definitions with descriptions for every property of the planner at the planner source code.

Dependencies​

The next case we are going to look at is what happens when there are attribute dependencies.

(ns com.wsscode.pathom3.docs.planner
  (:require [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.planner :as pcp]
            [edn-query-language.core :as eql]))

(pco/defresolver pi []
  {::pi Math/PI})

; TAU depends on PI
(pco/defresolver tau [{::keys [pi]}]
  {::tau (* pi 2)})

(def env (pci/register [pi tau]))

; generate execution DAG
(let [ast (eql/query->ast [::tau])]
  (pcp/compute-run-graph
    (assoc env
      :edn-query-language.ast/node ast)))

To make these examples shorter, from now on, I'm going to use the index-oir representation of the graph instead of writing the full resolvers. To start, let's look at our previous example using the index-oir format:

note

In the index-oir I'm going to use short names for the convenience of the demonstration, but keep in mind for real systems, its recommended to stick with long names.

; index-oir
{:pi  {{} #{pi}}
 :tau {{:pi {}} #{tau}}}

; query
[:tau]

The black arrow means the node tau will run after the node for pi.

AND nodes​

AND nodes represent a set of nodes that all need to run. Also, the planner designs the graph in a way that the runner can run all the nodes from an AND node in parallel if desired. To see an example of that here is a plan for a query that asks for three different attributes: :a, :b: and :c:

; index-oir

{:a {{} #{a}}
 :b {{} #{b}}
 :c {{} #{c}}}

; query

[:a :b :c]

Now, to see a combination of AND node and chaining, let's ask for a single attribute :d that requires :a, :b and :c:

; index-oir

{:a {{} #{a}}
 :b {{} #{b}}
 :c {{} #{c}}
 :d {{:a {} :b {} :c {}} #{d}}}

; query

[:d]

Note there are orange arrows and black arrows. The orange arrows run first (possibly in parallel), the black arrow only runs after all the orange arrows finish.

OR nodes​

For cases when there is more than one option to fetch an attribute. In those cases, Pathom uses the OR node to express that. For this example, we will have three different resolvers to the attribute :a:

; index-oir

{:a {{} #{a1 a2 a3}}}

; query

[:a]

Note the OR also uses orange arrows, but this time they mean at least once of the edges must run, instead of all of them.

In the next example, we can see how Pathom deals with graphs that require different input paths:

; index-oir

{:a {{:b {}} #{a1 a2 a3}
     {:c {} :d {}} #{a9}}
 :b {{} #{b}}
 :c {{:d {}} #{c}}
 :d {{} #{d}}}

; query

[:a]

Other examples​

Impossible paths​

Here you can see what happens when some request tries to find dependencies and reach a dead-end:

; index-oir

; :a depends on :b, there is no path for :b
{:a {{:b {}} #{a}}}

; query

[:a]

Now a partial dead-end, note how it discards the dead cluster on ab and ab2:

; index-oir

; :a depends on :b, there is no path for :b
{:a {{:c {}} #{ac}
     {:b {}} #{ab ab2}}
 :c {{} #{c}}}

; query

[:a]

Combined OR and AND​

When combining those, you will see the OR nodes get processed before the AND nodes.

; index-oir

{:a {{} #{a1 a2}}
 :b {{} #{b1 b2}}}

; query

[:a :b]

And a more complex example:

; index-oir

{:a {{} #{a}}
 :b {{:g {}} #{b}}
 :c {{} #{c}}
 :e {{} #{e e1}}
 :f {{:e {}} #{f}}
 :g {{:c {} :f {}} #{g}}
 :h {{:a {} :b {}} #{h}}}

; query

[:h]

Explorer​

You can use the following tool to play around with indexes and queries, and see how the planner will compute the plan for it:

tip

You can pull the Index OIR from some environment:

(def env (pci/register [some resolvers here]))

; print index oir
(clojure.pprint/pprint (::pci/index-oir env))