Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 40 ยท Neural Networks with Flax

Dropout, BatchNorm and the NNX API

Handle layers that behave differently in training and evaluation in Flax: dropout RNG keys, BatchNorm running statistics, mutable collections, and a first look at NNX.

Some layers act differently during training and during evaluation. Dropout randomly zeroes activations while training and does nothing at test time. BatchNorm normalizes with the statistics of the current batch while training, and with stored running averages at test time. In a language built on pure functions, both need a clear answer to the question: where does the extra state or randomness come from?

Flax's answer is explicit. Randomness arrives as a named key you pass to apply; running statistics live in a separate collection that apply can return as an updated copy. This lesson shows the real Flax code and then reproduces the mechanics in NumPy so you can see the numbers.

The Flax snippets are real but cannot run in the browser here (JAX is not available in the live editors). The runnable blocks are NumPy stand-ins of the same logic, so printed values come from NumPy, not from Flax.

Dropout needs a key and a flag

In Flax, a Dropout layer takes a deterministic flag. When it is False it draws a random mask, so it needs an RNG stream called "dropout".

Python
import jax, jax.numpy as jnp
import flax.linen as nn

class Net(nn.Module):
    @nn.compact
    def __call__(self, x, train: bool):
        x = nn.Dense(128)(x)
        x = nn.relu(x)
        x = nn.Dropout(rate=0.3, deterministic=not train)(x)
        return nn.Dense(10)(x)

model = Net()
key = jax.random.key(0)
init_key, dropout_key = jax.random.split(key)
params = model.init(init_key, jnp.ones((1, 784)), train=False)["params"]

# training: pass a key for the "dropout" stream
logits = model.apply({"params": params}, x, train=True,
                     rngs={"dropout": dropout_key})
# evaluation: no key needed, dropout is the identity
logits = model.apply({"params": params}, x, train=False)

Two rules follow from this design:

  • Use a fresh key every step. Reusing dropout_key would apply the same mask to every batch. Derive one per step with jax.random.fold_in(dropout_key, step), as explained in PRNG keys and split.
  • Forget train=False at test time and the output is random. Predictions will change between calls with different keys.

Why scale by 1 / (1 - rate)?

Dropping 30% of the units lowers the expected size of the next layer's input. Inverted dropout multiplies the survivors by 1 / (1 - rate) so the expected value stays the same, which means the test-time layer can simply pass data through.

Python Inverted dropout with an explicit seed (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The NumPy generator hides a mutable counter, which is why the two training calls differ. In JAX you make that explicit: the second call would need a different key.

BatchNorm and mutable collections

Batch normalization keeps two kinds of numbers. The scale and shift (gamma, beta) are learned parameters and live in params. The running mean and variance are not learned by gradient descent; they are updated by averaging and live in a separate collection called batch_stats.

Python
class ConvNet(nn.Module):
    @nn.compact
    def __call__(self, x, train: bool):
        x = nn.Dense(64)(x)
        x = nn.BatchNorm(use_running_average=not train)(x)
        x = nn.relu(x)
        return nn.Dense(10)(x)

model = ConvNet()
variables = model.init(jax.random.key(0), jnp.ones((8, 20)), train=False)
params, batch_stats = variables["params"], variables["batch_stats"]

# training step: ask apply to hand back the UPDATED batch_stats
logits, updates = model.apply(
    {"params": params, "batch_stats": batch_stats},
    x, train=True, mutable=["batch_stats"])
batch_stats = updates["batch_stats"]

# evaluation: use the stored averages, nothing is mutated
logits = model.apply({"params": params, "batch_stats": batch_stats},
                     x, train=False)

mutable=["batch_stats"] is the key idea. Nothing is mutated in place: apply runs the function, returns the outputs and a new batch_stats tree, and you thread it into the next step like any other state. The gradient is taken with respect to params only, because batch_stats is not differentiated.

When you use a TrainState (see Optax and TrainState), subclass it and add a batch_stats field, then replace it after each step.

Python BatchNorm: batch statistics in training, running averages in eval (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line shows why BatchNorm has an evaluation mode: with one sample the batch variance would be zero, but the running averages give a stable answer.

Mistakes to avoid

  • Updating statistics under jit without returning them. If apply runs with mutable but you discard updates, the running averages never change.
  • Evaluating with train=True. Predictions then depend on whatever else happens to be in the batch.
  • Taking gradients of batch_stats. Differentiate only params; use has_aux=True in jax.value_and_grad to carry the updated stats out of the loss function (see value_and_grad).
Python
def loss_fn(params, batch_stats, x, y, dropout_key):
    logits, updates = model.apply(
        {"params": params, "batch_stats": batch_stats}, x, train=True,
        mutable=["batch_stats"], rngs={"dropout": dropout_key})
    loss = optax.softmax_cross_entropy_with_integer_labels(logits, y).mean()
    return loss, updates["batch_stats"]

(loss, new_stats), grads = jax.value_and_grad(loss_fn, has_aux=True)(
    params, batch_stats, x, y, dropout_key)

A first look at NNX

Newer Flax versions include NNX, an API where modules are ordinary Python objects that hold their own arrays and RNG state, closer to PyTorch in feel. You write nnx.Linear(2, 3, rngs=nnx.Rngs(0)) and call it directly; there is no separate init/apply. Layers like dropout and batch norm switch with model.train() and model.eval(). NNX modules are converted to and from pytrees (nnx.split and nnx.merge) when you need JAX transformations such as jit and grad.

Python
from flax import nnx

class MLP(nnx.Module):
    def __init__(self, rngs: nnx.Rngs):
        self.fc1 = nnx.Linear(784, 128, rngs=rngs)
        self.drop = nnx.Dropout(0.3, rngs=rngs)
        self.fc2 = nnx.Linear(128, 10, rngs=rngs)

    def __call__(self, x):
        return self.fc2(self.drop(nnx.relu(self.fc1(x))))

model = MLP(nnx.Rngs(0))
model.train()      # dropout active
model.eval()       # dropout off

NNX trades a little of Linen's functional purity for convenience, and it is where the Flax project is heading. The functional Linen style you learned in this chapter still teaches the underlying model, because NNX transforms convert back to explicit state and keys. Check the current Flax documentation for the exact API of your installed version, since NNX evolves quickly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a Flax Dropout layer need when it is active (deterministic=False)?

    Randomness is explicit: pass rngs={'dropout': key}, and derive a fresh key each step.
  2. 2. Where do BatchNorm running means and variances live?

    They are not learned by gradients; apply returns an updated batch_stats when you pass mutable=['batch_stats'].
  3. 3. Why does inverted dropout divide survivors by (1 - rate)?

    Scaling preserves the expected value, so no correction is needed at test time.
  4. 4. How do you get updated BatchNorm statistics out of a loss function you differentiate?

    JAX functions are pure, so the new statistics travel out as an auxiliary return value.

Summary

  • Dropout takes deterministic and an "dropout" RNG key; use a fresh derived key per step.
  • BatchNorm keeps learned params apart from running batch_stats; training returns updated stats via mutable.
  • Differentiate params only and carry the new stats out with has_aux=True.
  • Always evaluate with train=False so outputs are deterministic and independent of the batch.
  • NNX offers a more object-style API (model.train(), model.eval()) built on the same ideas.

Next: Convolutional Networks in Flax