Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 40 ยท Autodiff with grad

value_and_grad, argnums and Auxiliary Data

Get the loss and its gradient in one pass with jax.value_and_grad, differentiate with respect to several arguments using argnums, and return extra metrics with has_aux.

The previous lesson, grad Basics, showed jax.grad returning the derivative of a scalar function with respect to its first argument. Real training code needs more: you usually want the loss value as well as the gradient, you often differentiate with respect to something other than the first argument, and you want to carry out extra numbers such as accuracy without differentiating them.

Three small features cover all of that: jax.value_and_grad, the argnums option, and has_aux. JAX cannot run in the in-browser runtime (see What Is JAX?), so the JAX code below is read-only and the editors use NumPy stand-ins that give the same numbers.

value_and_grad: the loss and its gradient together

Calling loss(w) and then jax.grad(loss)(w) runs the forward computation twice. jax.value_and_grad does it once and returns both results as a pair.

Python
import jax
import jax.numpy as jnp

def loss(w, x, y):
    return jnp.mean((x @ w - y) ** 2)

value, g = jax.value_and_grad(loss)(w, x, y)
print(value, g.shape)      # a scalar and an array shaped like w

The transformed function has the same signature as loss, but it returns (value, gradient). This is the form you will see inside nearly every training step.

Python Stand-in: value and gradient in one pass

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

argnums: choose what to differentiate

By default grad differentiates with respect to argument 0. Pass argnums to pick another position, or a tuple to get several gradients at once.

Python
def f(a, b, c):
    return a * b + jnp.sin(c)

jax.grad(f)(1.0, 2.0, 3.0)                       # d f / d a = b = 2.0
jax.grad(f, argnums=1)(1.0, 2.0, 3.0)            # d f / d b = a = 1.0
jax.grad(f, argnums=(0, 2))(1.0, 2.0, 3.0)       # (2.0, cos(3.0))

With a tuple, the result is a tuple with one gradient per selected argument, in the order you listed them. Arguments that are not selected can be integers, strings or anything else, because only the selected ones must be floating-point.

Python Stand-in: argnums with dual numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The stand-in differentiates one argument per forward pass (forward mode); real JAX uses reverse mode and gets all selected gradients from one backward pass. The numbers agree.

Gradients with respect to containers

An argument may be a dictionary, a list or any nested structure of arrays (a pytree). The gradient has the same structure:

Python
params = {"w": jnp.ones(3), "b": 0.0}

def loss(params, x, y):
    pred = x @ params["w"] + params["b"]
    return jnp.mean((pred - y) ** 2)

grads = jax.grad(loss)(params, x, y)
# grads == {"w": array of shape (3,), "b": scalar}

This is why you rarely need argnums for model parameters: put them in one container, keep it as argument 0, and the gradient comes back in the same shape.

has_aux: carry extra outputs

A loss function often computes more than the loss: accuracy, intermediate activations, updated statistics. grad insists that the function returns a scalar, so these extras would break it. Mark the function with has_aux=True and return a pair (loss, aux). Only the first element is differentiated; the second passes through untouched.

Python
def loss_fn(params, x, y):
    logits = x @ params["w"]
    loss = jnp.mean((logits - y) ** 2)
    metrics = {"mae": jnp.mean(jnp.abs(logits - y))}
    return loss, metrics                      # (differentiated, auxiliary)

(loss, metrics), grads = jax.value_and_grad(loss_fn, has_aux=True)(params, x, y)

Note the nesting of the result: value_and_grad returns ((loss, aux), grads). With plain grad and has_aux=True you get (grads, aux).

Python Stand-in: returning metrics next to the loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Putting it together: one gradient step

Python
@jax.jit
def step(params, x, y, lr=0.1):
    (loss, metrics), grads = jax.value_and_grad(loss_fn, has_aux=True)(params, x, y)
    params = jax.tree.map(lambda p, g: p - lr * g, params, grads)
    return params, loss, metrics

This is the entire inner loop of gradient descent. Here jit compiles it (jit Basics) and jax.tree.map applies the update to every leaf of the parameter container. The Flax training loop is this same step with a library-provided optimizer.

Name the three results clearly ((loss, metrics), grads). Mixing up the nesting of has_aux is the most common error with this API.

Common mistakes

  • Returning a non-scalar loss (for example, forgetting jnp.mean): grad raises an error about scalar outputs.
  • Passing an integer where a float is differentiated: write 1.0, not 1.
  • Putting extra outputs in the return value without has_aux=True.
  • Differentiating the wrong argument because the parameters are not argument 0.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does jax.value_and_grad(f)(x) return?

    It computes the value and the gradient in a single pass and returns them as a pair.
  2. 2. How do you get gradients with respect to arguments 0 and 2?

    argnums accepts an int or a tuple of ints; with a tuple you get a tuple of gradients.
  3. 3. What is has_aux=True for?

    The auxiliary value is passed through unchanged, which is how you return metrics next to the loss.
  4. 4. With value_and_grad and has_aux=True, what is the shape of the result?

    The value part is itself the pair (loss, aux), and the gradient follows.
  5. 5. A parameters dictionary is argument 0. What structure does grad return?

    Gradients mirror the structure of the differentiated argument.

Summary

  • jax.value_and_grad(f) returns (value, grad) without running the forward pass twice.
  • argnums selects which argument(s) to differentiate; a tuple gives a tuple of gradients.
  • Gradients of dictionaries and other pytrees have the same structure as the input.
  • has_aux=True lets the function return (loss, aux); aux is not differentiated.
  • A training step is value_and_grad, a tree-wide parameter update, and jit.

Next: Jacobians and Hessians