Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 40 ยท Neural Networks with Flax

Optax and TrainState: Optimizers as Pure Functions

Understand how Optax optimizers work as pure init and update functions, how to chain clipping and weight decay, add learning-rate schedules, and use Flax TrainState.

A model needs parameters, a loss, and an optimizer. In PyTorch the optimizer is an object that mutates parameters in place. JAX has no in-place mutation, so optimizers follow the same pure-function style as everything else: given the current parameters, gradients and optimizer state, they return the update and the new state.

The standard library for this is Optax. This lesson explains its two-function interface (init and update), shows how Adam keeps state, how to chain transformations such as gradient clipping and weight decay, how learning-rate schedules plug in, and how Flax's TrainState bundles parameters, optimizer and step counter into one pytree.

Optax and Flax need JAX, so real code is in plain blocks. The runnable blocks re-implement SGD and Adam in NumPy as (init, update) pairs with the same shape of API, so you can watch the state change. Optax's own numbers would match for these simple optimizers, but we do not print any JAX output.

The Optax interface

An Optax optimizer is a GradientTransformation: a pair of pure functions.

Python
import jax, jax.numpy as jnp
import optax

tx = optax.adam(learning_rate=1e-3)

params = {"w": jnp.zeros(3), "b": jnp.zeros(())}
opt_state = tx.init(params)                       # state shaped like params

grads = {"w": jnp.array([1., -2., 0.5]), "b": jnp.array(0.1)}
updates, opt_state = tx.update(grads, opt_state, params)
params = optax.apply_updates(params, updates)     # params + updates

Three steps, every time:

  1. tx.init(params) builds the optimizer state (for Adam: running means of gradients and squared gradients, plus a step count).
  2. tx.update(grads, state, params) returns updates (what to add to the parameters) and the new state. Nothing is modified in place.
  3. optax.apply_updates(params, updates) adds the updates to the parameters, leaf by leaf.

Because the updates and the state are pytrees shaped like the parameters, this works for a dict of dicts of any depth.

flowchart LR
    G["grads"] --> U{"tx.update"}
    S["opt_state"] --> U
    P["params"] --> U
    U --> UP["updates"]
    U --> NS["new opt_state"]
    UP --> A{"apply_updates"}
    P --> A
    A --> NP["new params"]

An optimizer step is a pure function from (params, grads, state) to (params', state').

A runnable optimizer pair

SGD with momentum and Adam, written with the same init/update shape. Because we use NumPy arrays directly for one-array params, the code stays short.

Python init/update optimizers: SGD and Adam (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both reach the minimum at zero. Notice that update never changes x or state: it returns new ones, and the loop reassigns. That pattern is the whole Optax design.

Chaining transformations

Real training rarely uses a bare optimizer. You clip gradients, apply weight decay, scale by a learning rate. Optax builds these from small pieces with optax.chain:

Python
tx = optax.chain(
    optax.clip_by_global_norm(1.0),        # 1. limit gradient norm
    optax.scale_by_adam(),                 # 2. Adam's direction
    optax.add_decayed_weights(1e-4),       # 3. weight decay
    optax.scale(-1e-3),                    # 4. learning rate (negative: descent)
)

Each piece is a GradientTransformation; chain runs them in order and keeps one state per piece. The ready-made optax.adamw(learning_rate, weight_decay=...) is a chain like the last three steps.

Python Gradient clipping as a transformation (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three-element gradient (norm 13) is scaled by 1/13, giving norm 1. Clipping stops a single bad batch from sending the weights far away, which is important for recurrent networks and transformers.

Learning-rate schedules

A schedule is just a function from step number to learning rate. Pass it wherever a learning rate is accepted:

Python
schedule = optax.warmup_cosine_decay_schedule(
    init_value=0.0, peak_value=1e-3,
    warmup_steps=500, decay_steps=10_000, end_value=1e-5)

tx = optax.adamw(learning_rate=schedule, weight_decay=1e-2)
print(schedule(0), schedule(500), schedule(10_000))

Common choices: optax.constant_schedule, optax.exponential_decay, optax.cosine_decay_schedule, optax.piecewise_constant_schedule, optax.linear_schedule. Optax keeps the step count in the optimizer state, so the schedule advances automatically.

Python Warmup then cosine decay (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rate climbs linearly for five steps, peaks, then eases down to the end value. Warmup avoids unstable early updates; the decay lets training settle.

TrainState: parameters, optimizer and step in one pytree

Passing params, opt_state and step separately through every function is tedious. Flax's TrainState bundles them, together with the apply_fn and the optimizer:

Python
from flax.training import train_state

state = train_state.TrainState.create(
    apply_fn=model.apply,
    params=variables["params"],
    tx=optax.adam(1e-3),
)

def loss_fn(params, x, y):
    logits = state.apply_fn({"params": params}, x)
    return optax.softmax_cross_entropy_with_integer_labels(logits, y).mean()

grads = jax.grad(loss_fn)(state.params, x, y)
state = state.apply_gradients(grads=grads)       # returns a NEW state; step is now 1
print(state.step)

state.apply_gradients(grads=...) runs tx.update, applies the updates and increments state.step, returning a fresh TrainState. The old state is untouched, so the whole thing can sit inside a jitted function that takes a state and returns a state. A custom TrainState subclass can add fields such as batch-norm statistics or an exponential moving average of the weights.

Since TrainState is a pytree dataclass, it is exactly the sort of class described in Custom Pytrees: apply_fn and tx are static fields, while params, opt_state and step are children.

Python A miniature TrainState (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The old state s0 is still intact after the step. That immutability is what lets JAX transform the update function safely.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does tx.update(grads, opt_state, params) return in Optax?

    Optax returns the updates (to be added to the parameters) and the new optimizer state. You then call optax.apply_updates.
  2. 2. What does optax.chain do?

    chain returns one optimizer that applies each transformation in sequence, keeping one state per piece.
  3. 3. Where does an Optax learning-rate schedule get its step number?

    Optax tracks the step count in the state, so the schedule advances automatically with each update.
  4. 4. What does TrainState.apply_gradients return?

    JAX arrays and pytrees are immutable by convention, so a new state is returned.

Summary

  • Optax optimizers are pure (init, update) pairs: update returns updates and a new state, then apply_updates adds them.
  • optax.chain composes clipping, Adam scaling, weight decay and the learning rate.
  • Schedules are step-to-rate functions passed in place of a learning rate.
  • Flax TrainState bundles params, optimizer state, step, apply_fn and the optimizer; apply_gradients returns a new state.
  • Immutability makes whole training steps safe to jit.

Next: Training an MLP: The Full Loop