Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 40 ยท Neural Networks with Flax

Saving Models, Learning-Rate Schedules and Gradient Clipping

Save and restore JAX parameter pytrees, add warmup and cosine learning-rate schedules with Optax, clip gradients by global norm, and chain optimizer transformations.

A model that trains but cannot be saved is a demo. A model that trains with a constant learning rate and no safety net is often unstable. This lesson covers three practical pieces of every serious training script: checkpointing parameters, schedules that change the learning rate over time, and gradient clipping that prevents one bad batch from wrecking the weights.

All three follow one JAX idea: parameters, optimizer state and schedules are plain data and pure functions, so saving and transforming them is straightforward.

Optax, Flax and JAX are not available in the live editors here. The Optax and Flax snippets are read-only. The runnable blocks reimplement the same formulas in NumPy, so their output comes from NumPy and demonstrates the math, not the libraries.

Checkpointing a pytree

Parameters are a pytree of arrays. To save one you flatten it to named arrays and write them; to restore you read them back into the same structure. The simplest approach is Flax's byte serialization:

Python
from flax import serialization

# save
raw = serialization.to_bytes(state.params)
with open("params.msgpack", "wb") as f:
    f.write(raw)

# restore: you need a template with the right structure
with open("params.msgpack", "rb") as f:
    params = serialization.from_bytes(template_params, f.read())

The template is needed because the file stores values, not Python structure; you create it by running model.init again. For large or sharded training runs, the Orbax library provides asynchronous, multi-host checkpoints with versioning (orbax.checkpoint.CheckpointManager), and it is the standard choice there.

Save the whole TrainState (parameters, optimizer state and step) if you want to resume training exactly; save only params for inference. Never use pickle for models from untrusted sources: unpickling can execute arbitrary code.

The same idea with plain arrays

To see what a checkpoint really is, flatten a nested parameter dict into path-named arrays, save them as one NumPy archive, then rebuild the tree.

Python Flatten a params tree, save, restore, compare (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This mirrors jax.tree_util.tree_flatten_with_path, which gives you the same path-and-leaf pairs for any pytree (see pytrees basics).

Learning-rate schedules

A constant learning rate is rarely best. Typical recipes:

  • Warmup: start near zero and ramp up over the first few hundred steps, because Adam's statistics are unreliable early on.
  • Cosine decay: after warmup, lower the rate smoothly toward zero. It needs no tuning of decay steps beyond the total length.
  • Step decay or exponential decay: multiply by a factor at fixed intervals.

In Optax a schedule is just a function from the step number to a learning rate. You pass it where a learning rate would go:

Python
import optax

schedule = optax.warmup_cosine_decay_schedule(
    init_value=0.0, peak_value=1e-3,
    warmup_steps=500, decay_steps=10_000, end_value=1e-5)

tx = optax.adamw(learning_rate=schedule, weight_decay=1e-4)

Because schedule(step) is a pure function, it is jit-friendly: the optimizer state carries a step counter and evaluates the schedule inside the compiled step.

Python Warmup + cosine decay, computed by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the shape: a ramp up to the peak, then a smooth bell-shaped fall. Logging the learning rate during a real run (schedule(state.step)) is a cheap way to confirm the schedule is wired in.

Gradient clipping

Occasionally one batch produces enormous gradients (a rare outlier, a bad initialization, an exploding recurrent network). A single huge update can undo hours of training. Clipping by global norm rescales all gradients together when their combined length exceeds a limit, preserving their direction:

Python
tx = optax.chain(
    optax.clip_by_global_norm(1.0),          # step 1: clip
    optax.adamw(learning_rate=schedule),     # step 2: update rule
)

optax.chain composes gradient transformations; the order matters, and clipping goes first so Adam sees sane inputs. The global norm is sqrt(sum of squares of every gradient leaf), taken over the whole pytree.

Python Clip by global norm over a gradient pytree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ratio of the two entries stays the same before and after: clipping shortens the update without turning it.

Putting it together

A common production setup combines everything:

Python
schedule = optax.warmup_cosine_decay_schedule(0.0, 3e-4, 1_000, total_steps, 1e-5)
tx = optax.chain(
    optax.clip_by_global_norm(1.0),
    optax.adamw(schedule, weight_decay=0.01),
)
state = train_state.TrainState.create(apply_fn=model.apply, params=params, tx=tx)

# every N steps
serialization.to_bytes(state)            # params + optimizer state + step

Other useful transformations in the same style: optax.apply_every(k) for gradient accumulation over several micro-batches, optax.add_decayed_weights for weight decay, and optax.masked to apply a transformation to a subset of parameters (for instance, no weight decay on biases). The mental model never changes: gradients flow through a chain of pure functions, each carrying its own state. This is the optimizer design from Optax and TrainState.

Common mistakes

  • Restoring into the wrong structure. If the model definition changed, the saved tree no longer matches the template.
  • Saving only params and then "resuming". Optimizer moments and the step counter are lost, so the schedule restarts and Adam warms up again.
  • Schedules that count the wrong thing. Optax schedules count optimizer steps, not epochs. Convert epochs to steps explicitly.
  • Clipping after Adam. The update has already been normalized; clip the raw gradients.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do you need a template when restoring parameters with flax.serialization.from_bytes?

    The saved bytes hold the leaf values; the target pytree supplies the structure.
  2. 2. What does clip_by_global_norm do when the norm is below the limit?

    The scale factor is min(1, max_norm / norm), so small gradients pass through untouched.
  3. 3. In optax.chain(clip_by_global_norm(1.0), adamw(...)), why is clipping first?

    Transformations are applied in order; clipping the raw gradients protects Adam's moment estimates.
  4. 4. What is lost if you checkpoint only params and resume training?

    Save the full TrainState to resume exactly.

Summary

  • A checkpoint is a serialized pytree; keep a template of the structure to restore into it.
  • Save the full train state to resume, only params for inference; use Orbax for large runs.
  • Schedules are pure functions of the step: warmup then cosine decay is a strong default.
  • Clip by global norm to preserve gradient direction while bounding step size, and put it first in optax.chain.
  • Avoid pickle for untrusted model files.

Next: Data Parallelism with pmap and Sharding