Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 40 ยท Neural Networks with Flax

Training an MLP: The Full Flax Loop

Build a complete JAX training loop: loss function, jitted train step, minibatches with epoch shuffling, evaluation, and the mistakes that make loops slow or wrong.

You now have all the parts: a model with init and apply, a pure optimizer, random keys and pytrees. This lesson assembles them into the loop every JAX training script contains: loss, gradient, update, repeat, plus minibatches, shuffling each epoch and evaluation.

The structure is the same whether the model has two layers or two hundred, so it is worth learning once, carefully. We first write the real Flax version, then run the identical algorithm in NumPy with hand-derived gradients so you can watch the loss fall.

The plain Flax code is real; run it on your machine after pip install jax flax optax. The runnable block is a stand-in: same data flow, same Adam update, but gradients derived by hand (jax.grad does that for you). Its printed numbers come from that NumPy program, not from JAX.

The shape of a training script

flowchart TB
    D["data: X, y"] --> E{"for each epoch"}
    E --> SH["shuffle with epoch key"]
    SH --> B{"for each minibatch"}
    B --> TS["train_step(state, batch)<br/>jit: loss, grad, update"]
    TS --> B
    B -->|epoch done| EV["evaluate on held-out data"]
    EV --> E

The inner step is a pure function from state to state, compiled once and reused thousands of times.

The real Flax version

Python
import jax, jax.numpy as jnp
import flax.linen as nn
import optax
from flax.training import train_state

class MLP(nn.Module):
    hidden: int = 64

    @nn.compact
    def __call__(self, x):
        x = nn.tanh(nn.Dense(self.hidden)(x))
        x = nn.tanh(nn.Dense(self.hidden)(x))
        return nn.Dense(1)(x)

def create_state(key, lr):
    model = MLP()
    params = model.init(key, jnp.ones((1, 1)))["params"]
    return train_state.TrainState.create(
        apply_fn=model.apply, params=params, tx=optax.adam(lr))

@jax.jit
def train_step(state, x, y):
    def loss_fn(params):
        pred = state.apply_fn({"params": params}, x)
        return jnp.mean((pred - y) ** 2)
    loss, grads = jax.value_and_grad(loss_fn)(state.params)
    return state.apply_gradients(grads=grads), loss

@jax.jit
def eval_loss(state, x, y):
    pred = state.apply_fn({"params": state.params}, x)
    return jnp.mean((pred - y) ** 2)

def train(key, x_train, y_train, x_val, y_val, epochs=200, batch_size=32):
    init_key, data_key = jax.random.split(key)
    state = create_state(init_key, 3e-3)
    n = x_train.shape[0]
    for epoch in range(epochs):
        perm = jax.random.permutation(jax.random.fold_in(data_key, epoch), n)
        for start in range(0, n - batch_size + 1, batch_size):
            idx = perm[start:start + batch_size]
            state, loss = train_step(state, x_train[idx], y_train[idx])
        if epoch % 20 == 0:
            print(epoch, float(loss), float(eval_loss(state, x_val, y_val)))
    return state

Read it top to bottom and notice the decisions:

  • Loss is a function of params (closed over the batch), so jax.value_and_grad differentiates the right argument and returns the loss for logging at no extra cost.
  • train_step returns a new state. It is jitted once; later calls reuse the compiled program as long as shapes stay the same.
  • Shuffling uses fold_in(data_key, epoch). Each epoch gets a different, reproducible order with no mutable generator.
  • Batches have a fixed size (the loop drops a final partial batch) so jit does not recompile for a smaller last batch. A changing shape is the most common reason a "fast" JAX loop is slow.
  • Evaluation is separate and jitted with no gradient and no state change.

The same algorithm, runnable

We fit y = sin(3x) with a two-hidden-layer tanh network trained by Adam on minibatches. Forward and backward passes are written out in NumPy; everything else (shuffle per epoch, minibatches, validation split) mirrors the Flax script above.

Python A complete training loop, hand-written gradients (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The validation error should drop sharply during the first hundred epochs and level off near the noise floor (the data noise has variance 0.0025). That is a healthy curve: both losses fall together, and the validation loss does not climb back up.

Reading the curves

  • Train and validation both high: under-fitting. Use more capacity, train longer, or raise the learning rate.
  • Train low, validation high and rising: over-fitting. Add data, dropout (see the next lesson), weight decay or early stopping.
  • Loss nan: learning rate too large, a log of zero, or a division by zero. Lower the rate, clip gradients, check for the where trap.
  • Loss flat from step one: gradients may be zero (a dead ReLU layer, a detached input), or the learning rate is far too small.

Making the real loop fast

  1. Jit the step, not the loop. Keep the Python for outside; compile only the pure step.
  2. Keep shapes fixed (drop the ragged last batch or pad it) so the step compiles once.
  3. Do not print every step. float(loss) forces the host to wait for the device (asynchronous dispatch). Log every N steps.
  4. Move the data to the device once. Indexing a NumPy array inside the loop copies each batch from host to device; for small datasets, jnp.asarray the whole array first.
  5. Consider lax.scan over steps for tiny models whose per-step Python overhead dominates.

Common mistakes

  • Differentiating the wrong argument (the loss must have params first, or use argnums).
  • Closing over a changing Python value in a jitted function (it is baked in at trace time).
  • Evaluating with training-only layers still active (dropout on, batch statistics instead of running averages).
  • Reusing the same key for the shuffle and for initialization.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the loop use jax.random.fold_in(data_key, epoch) for shuffling?

    fold_in derives a deterministic key per epoch, so runs are reproducible and resumable.
  2. 2. What is a common cause of a jitted training step recompiling repeatedly?

    jit caches by input shapes and dtypes; a new shape triggers a new compilation.
  3. 3. What does jax.value_and_grad give you compared to jax.grad?

    value_and_grad returns (value, grads), avoiding a second forward pass just to log the loss.
  4. 4. Training loss keeps falling while validation loss rises. What is happening?

    The model memorizes training data and generalizes worse; use regularization, more data or early stopping.

Summary

  • A training loop is: shuffle, minibatch, train_step(state, batch) returning a new state, periodic evaluation.
  • jax.value_and_grad over a loss of params, then state.apply_gradients(grads), inside one jitted function.
  • Derive per-epoch keys with fold_in; keep batch shapes fixed to avoid recompiles.
  • Read train versus validation curves to tell under-fitting, over-fitting and numeric problems apart.
  • Avoid host synchronization inside the loop; log sparingly.

Next: Dropout, BatchNorm and the NNX API