Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 40 ยท vmap and Vectorization

Per-Example Gradients with vmap and grad

Combine jax.grad with jax.vmap to get one gradient per training example, then use them for gradient clipping, noise estimates and influence scores.

Training code usually asks for one gradient: the gradient of the average loss over a batch. Some methods need the gradient of every example separately. Differentially private training clips each example's gradient before averaging. Influence analysis asks which training example pulled the weights in which direction. Gradient-noise estimates compare the spread of individual gradients with their mean.

In most frameworks, per-example gradients are awkward: you run the backward pass once per example, or you hack the framework. In JAX they are one line, because grad and vmap are both function transformations and transformations compose.

Plain blocks are real JAX. The runnable blocks compute the same quantities with NumPy, using gradients derived by hand, because jax.grad cannot run in the browser. They are stand-ins for the meaning, not the machinery.

The one-line recipe

Write the loss for a single example. Take its gradient with grad. Map that gradient function over the batch with vmap, sharing the parameters.

Python
import jax, jax.numpy as jnp

def loss(params, x, y):                 # ONE example: x (3,), y scalar
    pred = jnp.dot(params["w"], x) + params["b"]
    return (pred - y) ** 2

grad_one = jax.grad(loss)               # gradient w.r.t. params (argument 0)
per_example = jax.vmap(grad_one, in_axes=(None, 0, 0))

params = {"w": jnp.array([0.5, -1.0, 2.0]), "b": 0.1}
X = jnp.ones((8, 3))
Y = jnp.zeros(8)

g = per_example(params, X, Y)
print(g["w"].shape, g["b"].shape)       # (8, 3) (8,)

The result has the same structure as params (a dict with w and b), but every leaf gained a leading axis of length 8, one gradient per example. The mean of those over axis 0 equals the ordinary batch gradient of the mean loss.

flowchart LR
    L["loss(params, x, y)<br/>one example"] --> G["grad<br/>d loss / d params"]
    G --> V["vmap, in_axes=(None, 0, 0)<br/>params shared, data mapped"]
    V --> R["tree of gradients<br/>each leaf has a batch axis"]

Order matters: vmap(grad(f)) maps the gradient. grad(vmap(f)) would try to differentiate a vector-valued function and fail.

A runnable stand-in

For the squared-error loss (w.x + b - y)^2 the gradients are 2 * err * x for w and 2 * err for b. Computing them by hand lets us see the same table of per-example gradients and check the averaging property.

Python Per-example gradients by hand (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The row g_w[0] is the gradient that example 0 alone would apply. In JAX, vmap(grad(loss)) produces exactly this table without you deriving anything.

Use 1: per-example gradient clipping

Differentially private SGD clips each example's gradient to a maximum norm, then adds noise to the sum. The clip needs the individual gradients, which is precisely what we have. With pytrees, the global norm is computed over every leaf of one example's gradient.

Python
def clip_tree(g, max_norm):
    norm = jnp.sqrt(sum(jnp.sum(leaf ** 2) for leaf in jax.tree.leaves(g)))
    scale = jnp.minimum(1.0, max_norm / (norm + 1e-12))
    return jax.tree.map(lambda leaf: leaf * scale, g)

clipped = jax.vmap(clip_tree, in_axes=(0, None))(g, 1.0)
mean_clipped = jax.tree.map(lambda leaf: leaf.mean(axis=0), clipped)

Note how clip_tree is written for one example's gradient tree and then vmapped again. Nothing in it knows about batches.

Python Clip every example's gradient (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gradients with norm above 1 are scaled down to exactly 1; small ones are untouched. That is the whole mechanism.

Use 2: how noisy is my batch gradient?

If per-example gradients all point the same way, small batches are fine. If they disagree wildly, you need larger batches or a smaller learning rate. A common summary is the signal-to-noise ratio: the squared norm of the mean gradient divided by the average squared deviation from it.

Python Gradient signal-to-noise (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Near the start of training the ratio is high (the examples agree on a direction). As the model approaches a minimum the mean gradient shrinks while individual gradients stay large, and the ratio drops. That drop is one reason learning rates are decayed.

Higher-order and Jacobian variations

The same composition idea extends further:

  • jax.vmap(jax.value_and_grad(loss), ...) returns per-example losses and gradients together.
  • jax.vmap(jax.jacrev(f)) gives a Jacobian per example.
  • jax.vmap(jax.hessian(f)) gives a per-example Hessian (small models only).
  • jax.grad(lambda p: jnp.mean(jax.vmap(loss, (None, 0, 0))(p, X, Y))) is the ordinary batch gradient, written by mapping the loss inside grad. Both orders are valid; they answer different questions.
Python
def batch_loss(params, X, Y):
    return jnp.mean(jax.vmap(loss, in_axes=(None, 0, 0))(params, X, Y))

batch_grad = jax.grad(batch_loss)(params, X, Y)   # one gradient for the mean loss

grad of the mapped-then-averaged loss and the mean of vmap(grad(loss)) agree, as the first stand-in demonstrated.

Memory and speed notes

A table of per-example gradients costs batch_size * parameter_count numbers. For a million-parameter model and a batch of 256 that is 256 million floats, about 1 GB in float32. Options when it is too large:

  1. Reduce the batch, or process micro-batches and accumulate.
  2. Use lax.map instead of vmap for the gradient step (sequential, low memory).
  3. Reduce inside the mapped function when you only need a summary such as the norm, so the full table is never kept.

Per-example gradients in JAX

Write the loss for ONE example
Apply grad gradient w.r.t. params
Apply vmap in_axes=(None, 0, 0)

Parameters are shared, the data is mapped.

1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which expression gives one gradient per example for loss(params, x, y)?

    grad is applied to the single-example loss first, then vmap maps that gradient function over the data while sharing params.
  2. 2. What is the structure of the per-example gradient result when params is a dict {'w': ..., 'b': ...}?

    vmap preserves the pytree structure and adds the batch axis to each leaf.
  3. 3. Why does per-example clipping need per-example gradients?

    Clipping the mean gradient would not bound any single example's influence, so each gradient is clipped first.
  4. 4. What is the main cost of storing per-example gradients?

    Every example gets its own copy of a gradient the size of the parameters.

Summary

  • jax.vmap(jax.grad(loss), in_axes=(None, 0, 0)) returns one gradient per example in a single line.
  • The result keeps the parameter pytree structure, with a batch axis added to every leaf.
  • The mean of the per-example gradients equals the gradient of the mean loss.
  • Typical uses: gradient clipping, gradient-noise estimates, influence analysis.
  • Memory grows with batch size times parameter count; use micro-batches or lax.map when needed.

Next: vmap with Conditionals and Its Limits