Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 40 ยท Autodiff with grad

Jacobians, Hessians and Higher-Order Derivatives

Differentiate vector-valued functions with jax.jacfwd and jax.jacrev, build Hessians, and understand when forward or reverse mode is cheaper. Includes runnable stand-ins.

jax.grad needs a scalar output. Many functions return vectors: a neural network layer maps a vector to a vector, a physics simulation maps a state to the next state. The derivative of such a function is a matrix of partial derivatives called the Jacobian. Differentiate the gradient once more and you get the Hessian, the matrix of second derivatives that describes curvature.

JAX provides jacfwd, jacrev and hessian for these, and because every transformation returns an ordinary function, they compose freely. JAX code here is read-only (see What Is JAX?); the editors use NumPy stand-ins that compute the same matrices.

The Jacobian

For f: R^n -> R^m, the Jacobian J is an m x n matrix with J[i, j] = d f_i / d x_j.

Python
import jax
import jax.numpy as jnp

def f(x):
    return jnp.array([x[0] * x[1], jnp.sin(x[0]), x[1] ** 2])   # R^2 -> R^3

x = jnp.array([1.0, 2.0])
J = jax.jacobian(f)(x)        # shape (3, 2)
# [[ x1,  x0 ],      [[2.        , 1.        ],
#  [cos x0, 0 ],  =   [0.5403023 , 0.        ],
#  [ 0 , 2 x1 ]]      [0.        , 4.        ]]

The output shape rule is simple: the Jacobian has shape output.shape + input.shape. A function from a (3,) vector to a (2, 2) matrix has a Jacobian of shape (2, 2, 3).

Python Stand-in: a numerical Jacobian

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Forward mode and reverse mode

JAX offers two algorithms that produce the same matrix at different cost:

  • jax.jacfwd builds the Jacobian one column at a time (one pass per input). Cheap when there are few inputs and many outputs.
  • jax.jacrev builds it one row at a time (one pass per output). Cheap when there are many inputs and few outputs, which is the machine-learning situation.
Python
J1 = jax.jacfwd(f)(x)
J2 = jax.jacrev(f)(x)      # same values

jax.jacobian is an alias for jacrev. The rule of thumb for a tall Jacobian (more outputs than inputs) is jacfwd; for a wide one (more inputs than outputs) it is jacrev. A scalar-output function is the extreme wide case, which is why grad uses reverse mode. Forward and Reverse Mode: jvp and vjp explains the machinery underneath.

flowchart LR
    A["f: R^n to R^m"] --> B{"n small, m large?"}
    B -- "yes" --> C["jacfwd: n passes"]
    B -- "no" --> D["jacrev: m passes"]

Choose the mode with fewer passes.

The Hessian

The Hessian of a scalar function is the Jacobian of its gradient. JAX has it built in, and it is simply a composition:

Python
def loss(w):
    return jnp.sum(w ** 3) + w[0] * w[1]

H1 = jax.hessian(loss)(jnp.array([1.0, 2.0]))
H2 = jax.jacfwd(jax.jacrev(loss))(jnp.array([1.0, 2.0]))     # same thing
# [[6 w0, 1 ],      [[ 6., 1.],
#  [ 1  , 6 w1]]  =  [ 1., 12.]]

Forward-over-reverse (jacfwd(jacrev(...))) is the efficient order, because the inner reverse pass produces a vector-valued gradient of which you take a forward-mode Jacobian. The Hessian is square and symmetric for smooth functions.

Python Stand-in: Hessian from the gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What are Hessians used for?

  • Newton's method. Solve H p = -g and step by p; it converges in few iterations near a minimum.
  • Curvature checks. All eigenvalues positive means a local minimum; mixed signs mean a saddle point.
  • Uncertainty. The inverse Hessian at a fitted optimum approximates the covariance of the parameters.
  • Sharpness analysis of neural-network minima.

For a model with p parameters the Hessian has p^2 entries, so a full Hessian is only practical for small p. For large models you use Hessian-vector products instead, which need no matrix at all:

Python
def hvp(f, x, v):
    return jax.jvp(jax.grad(f), (x,), (v,))[1]     # forward-over-reverse

Higher-order derivatives of scalar functions

Since grad returns a function, you can nest it as many times as you like:

Python
f = lambda x: jnp.sin(x)
d1 = jax.grad(f)
d2 = jax.grad(d1)
d3 = jax.grad(d2)
d3(0.0)      # -cos(0) = -1.0

The cost grows with each nesting, but nothing else changes. This lets you differentiate through an optimization step (for meta-learning) or compute derivatives of a physics model with respect to its inputs.

Integer inputs are rejected here too: a Jacobian is only defined for floating-point inputs. Convert with x.astype(jnp.float32) first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A function maps a vector of 5 numbers to a vector of 3 numbers. What is the Jacobian's shape?

    The shape is output shape + input shape, here (3,) + (5,) = (3, 5).
  2. 2. Which transformation is usually cheaper for a function with many inputs and few outputs?

    Reverse mode needs one pass per output, so few outputs means few passes.
  3. 3. How is the Hessian related to the gradient?

    Differentiating the gradient function again produces the matrix of second derivatives.
  4. 4. Why use Hessian-vector products for big models?

    A hessian-vector product costs about as much as a couple of gradients and uses O(p) memory.

Summary

  • jacfwd and jacrev return the matrix of partial derivatives; the result has shape output.shape + input.shape.
  • Use jacfwd for tall Jacobians, jacrev for wide ones; jacobian is jacrev.
  • jax.hessian(f) equals jacfwd(jacrev(f)) and is symmetric for smooth scalar functions.
  • Hessian-vector products avoid building the full matrix.
  • grad can be nested for derivatives of any order.

Next: stop_gradient and Custom Derivatives