Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 40 ยท Random Numbers and Pytrees

Pytrees Basics: Nested Data JAX Understands

Understand JAX pytrees: nested lists, tuples and dicts of arrays. Learn tree_map, leaves, structure, flatten and unflatten, tree reduce and common pitfalls.

Neural-network parameters are not one array. A small model has a weight matrix and a bias per layer, each layer has a name, and the whole thing is nested: a dict of dicts of arrays. JAX functions such as grad, jit and vmap need to accept and return structures like this, not just single arrays.

JAX's answer is the pytree: any nested combination of Python containers (lists, tuples, dicts, None, and a few more) whose leaves are arrays or numbers. JAX can flatten any pytree into a list of leaves plus a description of its shape, work on the leaves, and rebuild the same structure. This lesson explains that machinery and the handful of tree functions you will use constantly.

The plain blocks use the real jax.tree API. The runnable blocks implement a small, honest version of the same functions in pure Python (tree_map, leaves) so you can run them here. They support lists, tuples and dicts, which is enough to show every idea. Real JAX handles more container types.

What counts as a pytree

  • Leaves: arrays and numbers (and anything JAX does not recognize as a container).
  • Containers (nodes): list, tuple, dict, collections.OrderedDict, collections.namedtuple, None (an empty node with no leaves), and any class you register (Custom Pytrees).
Python
import jax, jax.numpy as jnp

params = {
    "layer1": {"w": jnp.ones((3, 4)), "b": jnp.zeros(4)},
    "layer2": {"w": jnp.ones((4, 2)), "b": jnp.zeros(2)},
    "scale": 2.0,
}

print(jax.tree.leaves(params))        # five leaves: 4 arrays and the float 2.0
print(jax.tree.structure(params))     # PyTreeDef({'layer1': {'b': *, 'w': *}, ...})

Dictionaries are flattened in sorted key order, so {"w":..., "b":...} always flattens as b then w, no matter how you wrote it. That makes the structure deterministic.

flowchart TB
    P["params (dict)"] --> L1["layer1 (dict)"]
    P --> L2["layer2 (dict)"]
    P --> S["scale: 2.0 (leaf)"]
    L1 --> W1["w: array (3, 4) leaf"]
    L1 --> B1["b: array (4,) leaf"]
    L2 --> W2["w: array (4, 2) leaf"]
    L2 --> B2["b: array (2,) leaf"]

The boxes are nodes or leaves. A function that works on leaves works on the whole tree.

tree.map: apply a function to every leaf

jax.tree.map(f, tree) applies f to each leaf and returns a tree with the same structure. (Older code spells it jax.tree_util.tree_map; it still works.) This is how you scale all parameters, cast dtypes, or take a step of gradient descent in one line:

Python
sgd_step = lambda p, g: p - 0.1 * g
new_params = jax.tree.map(sgd_step, params, grads)    # params and grads: same structure

With several trees, map walks them together, leaf by leaf, so f receives one leaf from each. The trees must have the same structure, otherwise JAX raises an error that names the mismatch.

Here is a runnable version of the same idea.

Python A pure-Python tree_map (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result has the same nesting as params, with 0.1 * grad subtracted from every leaf. That one-line gradient step is exactly how plain SGD is written in JAX.

Leaves, structure, flatten, unflatten

Three functions describe what map is built from:

Function Returns
jax.tree.leaves(tree) a flat list of the leaves
jax.tree.structure(tree) a PyTreeDef: the skeleton without values
jax.tree.flatten(tree) both: (leaves, treedef)
jax.tree.unflatten(treedef, leaves) rebuilds a tree from a skeleton and new leaves

Flatten then unflatten is the heart of everything: map is just "flatten, apply to each leaf, unflatten".

Python
leaves, treedef = jax.tree.flatten(params)
doubled = [leaf * 2 for leaf in leaves]
rebuilt = jax.tree.unflatten(treedef, doubled)       # same structure as params

Working with a flat list is handy for operations that need all leaves together, such as computing a global norm or concatenating every parameter into one vector.

Python Leaves, global norm and parameter count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

jax.tree.reduce(fn, tree, initial) folds all leaves with a two-argument function, so a parameter count is jax.tree.reduce(lambda n, leaf: n + leaf.size, params, 0).

Trees and transformations

This is where pytrees pay off. Every JAX transformation accepts and returns pytrees:

  • jax.grad(loss)(params, x, y) returns gradients with exactly the structure of params. No flattening code.
  • jax.jit(f) takes tree arguments and caches by tree structure plus leaf shapes and dtypes.
  • jax.vmap(f, in_axes=...) can map some leaves and not others (see vmap Axes).

The structure you pass in decides the structure you get back. If params is a dict of dicts, grads is a dict of dicts with identical keys, which is why tree.map(update, params, grads) just works.

None, empty containers and strings

  • None is a node with no leaves. jax.tree.leaves({"a": None, "b": 1}) is [1]. This is how optional pieces of a state are expressed.
  • Empty lists and dicts are nodes with zero leaves.
  • A string or a Python object that is not a recognized container is a leaf. Passing it into a jitted function fails, because a string is not an array JAX can trace. Keep configuration values static (see static_argnames) rather than inside the arguments.
Python None and empty containers have no leaves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Paths: knowing which leaf is which

jax.tree_util.tree_map_with_path(f, tree) passes a key path to f, so you can treat leaves differently by name. A typical use is weight decay applied to weights but not to biases.

Python
def decay_mask(path, leaf):
    last = path[-1].key            # DictKey("w") or DictKey("b")
    return last == "w"

mask = jax.tree_util.tree_map_with_path(decay_mask, params)

jax.tree_util.keystr(path) turns a path into a readable string such as ['layer1']['w'].

Common pitfalls

  1. Mismatched structures. tree.map(f, a, b) requires b to have a's structure. A missing key or a list where a tuple was expected is an error.
  2. Lists versus tuples. They are different node types; a list does not match a tuple of the same contents.
  3. Hidden non-array leaves. A stray integer or string in your params tree becomes a leaf that grad tries to differentiate (integers cause an error).
  4. Key order. Dicts flatten in sorted key order; relying on insertion order for flat lists will bite.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a pytree in JAX?

    Pytrees are ordinary nested Python containers. JAX knows how to flatten them to leaves and rebuild them.
  2. 2. What does jax.tree.map(f, tree_a, tree_b) require of its inputs?

    map walks the trees in step, calling f on matching leaves, so the structures must agree.
  3. 3. What does jax.grad(loss)(params, ...) return when params is a dict of dicts?

    Gradients mirror the structure of the differentiated argument.
  4. 4. How many leaves does {'a': None, 'b': [1, 2]} have?

    None is a node with no leaves, so only 1 and 2 are leaves.

Summary

  • A pytree is any nesting of lists, tuples, dicts (and registered classes) with arrays or numbers as leaves.
  • tree.map applies a function across leaves; leaves, structure, flatten and unflatten are the underlying parts.
  • Every JAX transformation takes and returns pytrees; gradients mirror the parameter structure.
  • None and empty containers have no leaves; strings and arbitrary objects are leaves and cannot be traced.
  • Use path-aware maps to treat leaves differently by name.

Next: Custom Pytrees: Registering Your Own Classes