Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 40 ยท Autodiff with grad

Forward and Reverse Mode: jvp and vjp

Understand the two building blocks of JAX autodiff: jax.jvp for forward mode and jax.vjp for reverse mode, how they relate to grad and Jacobians, and when each is cheaper.

Every derivative function in JAX, grad, jacfwd, jacrev and hessian, is built from two primitives: jax.jvp and jax.vjp. Knowing them explains why grad is cheap for machine learning, why Jacobians come in two flavours, and how to compute things like Hessian-vector products without building giant matrices.

JAX itself cannot run in the in-browser runtime (see What Is JAX?), so the JAX snippets are read-only. The editors use NumPy stand-ins that perform the same linear-algebra operations on an explicit Jacobian, which makes the ideas concrete.

The Jacobian as a linear map

Take f: R^n -> R^m, and a point x. Its Jacobian J is an m x n matrix, but you rarely need the whole matrix. Two products with J are enough:

Product Meaning Primitive
J @ v (v in R^n) how the output changes when the input moves along v jvp (forward mode)
u @ J (u in R^m) how a weighted sum of outputs changes per input vjp (reverse mode)

JVP stands for Jacobian-vector product; VJP for vector-Jacobian product. Neither builds J. Each costs only a small multiple of evaluating f once.

jvp: pushing a tangent forward

Python
import jax
import jax.numpy as jnp

def f(x):
    return jnp.array([x[0] * x[1], jnp.sin(x[0])])

x = jnp.array([1.0, 2.0])
v = jnp.array([1.0, 0.0])           # move x along the first axis

y, tangent = jax.jvp(f, (x,), (v,))
# y       = f(x)           = [2.0, 0.8414...]
# tangent = J @ v          = [x1 * 1, cos(x0) * 1] = [2.0, 0.5403...]

jvp takes the function, a tuple of inputs (the "primals") and a tuple of tangent vectors with matching shapes. It returns the output and the output tangent in a single forward sweep: the derivative information travels with the computation, just like the dual numbers used in the stand-ins of earlier lessons.

Python Stand-in: J @ v without forming J

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

To assemble a full Jacobian with jvp you call it once per input coordinate (n times) and stack the columns. That is exactly jacfwd.

vjp: pulling a cotangent back

jax.vjp does the opposite. It runs f once forward, remembers what it needs, and returns a function that maps an output-sized vector (a cotangent) back to an input-sized vector:

Python
y, f_vjp = jax.vjp(f, x)          # forward pass, saves residuals
(grad_x,) = f_vjp(jnp.array([1.0, 0.0]))     # u @ J with u = [1, 0]
# grad_x = first row of J = [x1, x0] = [2.0, 1.0]

f_vjp returns a tuple with one entry per input of f. To build a full Jacobian with vjp, apply it to each unit vector of the output (m times) to get the rows; that is jacrev.

Notice what happens for a scalar function, m = 1: a single call f_vjp(1.0) produces the entire gradient, all n partial derivatives at once. That is precisely jax.grad:

Python
def grad(f):
    def wrapped(x):
        y, f_vjp = jax.vjp(f, x)
        return f_vjp(jnp.ones_like(y))[0]
    return wrapped
Python Stand-in: gradient = one vjp with the vector [1]

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why reverse mode wins for machine learning

Compare costs for f: R^n -> R^m:

  • A full Jacobian by forward mode: n JVPs.
  • A full Jacobian by reverse mode: m VJPs.

A neural-network loss has millions of inputs (the parameters) and one output. Reverse mode needs one backward pass and gets all million derivatives. Forward mode would need a million passes. The price of reverse mode is memory: the forward pass must store intermediate values (the residuals) for the backward pass. This is why training memory grows with network depth and why tricks such as rematerialization (jax.checkpoint) exist.

flowchart LR
    subgraph Forward["jvp: forward mode"]
    A["input tangent v"] --> B["f"] --> C["output tangent J v"]
    end
    subgraph Reverse["vjp: reverse mode"]
    D["output cotangent u"] --> E["f, run backwards"] --> F["input cotangent u J"]
    end

Forward mode pushes input perturbations through; reverse mode pulls output sensitivities back.

Combining them: Hessian-vector products

Second derivatives show how the two combine. The gradient of f is a function from R^n to R^n. The Hessian-vector product H @ v is the JVP of that gradient function along v:

Python
def hvp(f, x, v):
    return jax.jvp(jax.grad(f), (x,), (v,))[1]      # forward-over-reverse

It costs a small multiple of one gradient and never builds the n x n matrix. Jacobians and Hessians uses this idea for the full Hessian.

Other uses of jvp and vjp

  • Sensitivity analysis: "if input v changes slightly, how does the output move?" is a single jvp.
  • Implicit differentiation and optimization layers: solve a linear system whose matrix is given only as a jvp/vjp function.
  • Custom derivative rules: custom_jvp and custom_vjp are written in exactly these terms.
  • Linearization: jax.linearize(f, x) returns f(x) and a function equivalent to the JVP, evaluated cheaply for many tangents.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does jax.jvp(f, (x,), (v,)) return?

    It returns the output and the output tangent, the Jacobian multiplied by the input tangent v.
  2. 2. Which primitive underlies jax.grad?

    grad is a vjp of a scalar function applied to the cotangent 1.0.
  3. 3. A loss has 1,000,000 parameters and one output. Which mode computes the full gradient cheaper?

    Reverse mode needs one pass per output; forward mode needs one per input.
  4. 4. What is the price of reverse mode?

    Intermediate values are saved for the backward pass; jax.checkpoint trades memory for recomputation.
  5. 5. How do you compute a Hessian-vector product efficiently?

    Forward-over-reverse gives H @ v at the cost of a few gradients with O(n) memory.

Summary

  • jvp computes J @ v going forward; vjp computes u @ J going backward.
  • jacfwd is n jvps; jacrev is m vjps; grad is one vjp.
  • Reverse mode is the right default for many inputs and one output, but needs memory for residuals.
  • Forward-over-reverse gives Hessian-vector products without the matrix.
  • Custom derivative rules are written in terms of these primitives.

Next: jit Basics: Compiling Functions with XLA