Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 40 ยท Autodiff with grad

grad Basics: Differentiating Python Functions

Learn jax.grad: differentiate ordinary Python functions, use loops and if statements, handle vectors, understand scalar outputs and float inputs, and check gradients numerically.

Derivatives drive almost everything in machine learning and scientific computing: training a model means following the slope of a loss function. Writing derivatives by hand is slow and error-prone; approximating them with finite differences is imprecise. Automatic differentiation (autodiff) gives exact derivatives of your code, automatically.

In JAX, jax.grad takes a Python function and returns a new function that computes its gradient. The original function can contain loops, branches and calls to other functions, as long as it is built from JAX operations. This lesson covers the basics. JAX itself cannot run in the in-browser runtime (see What Is JAX?), so JAX code is read-only and the editors use a small forward-mode autodiff stand-in written in plain Python that produces the same numbers.

Your first gradient

Python
import jax
import jax.numpy as jnp

def f(x):
    return x ** 3 + 2.0 * x          # f'(x) = 3x^2 + 2

df = jax.grad(f)                     # df is a function
print(df(2.0))                       # 14.0

d2f = jax.grad(df)                   # grad of grad: the second derivative
print(d2f(2.0))                      # 12.0   (f''(x) = 6x)

jax.grad(f)(x) evaluates the derivative of f at x. Because grad returns a function, you can pass it straight into other transformations, which is the key to composability.

flowchart LR
    F["f(x)"] -->|"jax.grad"| G["df(x)"]
    G -->|"jax.grad"| H["d2f(x)"]

Each application of grad returns another ordinary function.

A stand-in you can run

The class below carries a value together with its derivative (a "dual number"). Running a function on dual numbers computes the function and its derivative in one pass: that is forward-mode autodiff. JAX's grad uses reverse mode, which is better for many inputs and one output, but the numbers agree.

Python Stand-in: a tiny grad for plain Python

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ordinary Python control flow works

Because JAX records what your function actually does for a given input, if, for and while statements are fine in grad (they see concrete values; the restrictions arrive with jit, see jit basics).

Python
def g(x):
    if x > 0:
        return x ** 2
    return -x

jax.grad(g)(3.0)     # 6.0
jax.grad(g)(-3.0)    # -1.0

def power_loop(x, n):
    result = 1.0
    for _ in range(n):
        result = result * x
    return result

jax.grad(power_loop)(2.0, 3)    # 12.0  = 3 * 2^2
Python Stand-in: control flow, loops and recursion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rules for jax.grad

  1. The function must return a scalar (a Python float or a 0-d array). For a vector output you need a Jacobian (Jacobians and Hessians).
  2. The differentiated argument must be floating-point. jax.grad(f)(3) with an integer raises an error; use 3.0. Integers and booleans have no derivative.
  3. By default grad differentiates with respect to the first argument. Use argnums to pick others (next lesson).
  4. Arguments may be arrays, and the gradient then has the same shape.
Python
def loss(w):
    return jnp.sum(w ** 2)

jax.grad(loss)(jnp.array([1.0, 2.0, 3.0]))     # [2. 4. 6.]

def mse(w, x, y):
    return jnp.mean((x @ w - y) ** 2)          # scalar output

The gradient of a scalar function with respect to a vector is the vector of partial derivatives, with exactly the input's shape: this is what an optimizer needs to update parameters.

Python Stand-in: gradient of a vector function by coordinates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result True tells us the numerical gradient matches the analytic one. In real code you would simply call jax.grad(mse)(w0), and gradient checking like this is a handy way to validate a hand-written derivative or a custom rule.

Things to watch out for

  • Non-differentiable points. jnp.abs at 0 or relu at 0 have no unique derivative; JAX returns one fixed value by convention; check it with grad if it matters to you.
  • NaN and inf gradients. jnp.sqrt at 0 has an infinite derivative. A famous trap is jnp.where(x > 0, jnp.sqrt(x), 0.0), which still produces a NaN gradient for negative x because both branches are differentiated; use the "double where" trick, or guard the input of the sqrt.
  • Not everything is differentiable. Operations such as argmax, rounding or integer conversion have zero or undefined gradients.
  • Python scalars only as inputs. If you convert an array to a Python float with float(x) inside the function, the derivative is lost.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does jax.grad(f) return?

    grad is a transformation: it takes a function and returns a function.
  2. 2. Which call raises an error?

    Differentiation requires floating-point inputs; integers have no derivative.
  3. 3. What must the function passed to jax.grad return?

    grad is defined for scalar-valued functions; for vector outputs use jacobian-style transformations.
  4. 4. grad of a scalar loss with respect to an array w has which shape?

    There is one partial derivative per input element, arranged like the input.

Summary

  • jax.grad(f) returns the gradient function of a scalar-valued f; apply it repeatedly for higher derivatives.
  • Python if, loops and recursion work, because JAX differentiates what actually ran.
  • Inputs must be floats; the gradient has the shape of the differentiated argument.
  • Beware non-differentiable points and NaN gradients from guarded branches.
  • Gradient checking with finite differences is a good test.

Next: value_and_grad, argnums and Auxiliary Data