Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 73 ยท Training in Practice

Batch Norm and Layer Norm

Learn how batch norm and layer norm work: forward and backward passes in NumPy, running statistics, where to place them, and why they speed training.

Deep networks are sensitive to the scale of the numbers flowing through them. If one layer's outputs drift to huge or tiny values, the next layer sees inputs it was never tuned for, activations saturate, and gradients shrink or blow up. Weight initialization fixes this at the start of training, but the weights change at every step, so the scale can drift later.

Normalization layers attack the problem directly: inside the network, they rescale the activations to a steady mean and spread, then let the model learn what scale it actually wants. The two you will meet everywhere are batch normalization (batch norm) and layer normalization (layer norm). In this lesson you will write both in NumPy, including the backward pass, and see why networks that use them train faster.

Normalize, then let the model undo it

Measure Compute the mean and variance of the activations.
Standardize Subtract the mean, divide by the standard deviation.
Rescale Learned gamma and beta restore any scale that helps.

The layer can even learn to cancel the normalization if that is best.

1 / 2

Why bother? The covariate shift debate

The original batch norm paper explained its success with internal covariate shift: as earlier layers change, the distribution of inputs to later layers keeps moving, so later layers must keep re-adapting. Normalizing each layer's inputs, the argument went, keeps that distribution steady.

Later research challenged the story. Experiments showed that batch norm helps even when you deliberately add noise that makes the distribution shift, and the more convincing explanation is that normalization smooths the loss surface: gradients become more predictable, so you can use much larger learning rates without diverging. You do not need to settle the debate to use the tool. What matters in practice is the observed effect: faster, more stable training, less sensitivity to initialization, and a mild regularizing side effect.

Batch normalization

Take a batch of activations x with shape (N, D): N examples, D features. Batch norm treats each feature column on its own:

  1. mu = mean of the column over the batch
  2. var = variance of the column over the batch
  3. x_hat = (x - mu) / sqrt(var + eps), where eps (like 1e-5) avoids dividing by zero
  4. y = gamma * x_hat + beta

gamma and beta are learnable vectors of length D, initialized to 1 and 0. Without them, every layer would be forced to output zero-mean, unit-variance values, which would limit what it can represent (a sigmoid, for instance, would be stuck in its near-linear middle).

The backward pass is the interesting part, because every output depends on every example through mu and var. Working out the chain rule gives a compact formula for the input gradient. With dxhat = dout * gamma:

dx = (1 / N) / sqrt(var + eps) * (N * dxhat - sum(dxhat) - x_hat * sum(dxhat * x_hat))

The next example implements both passes and then proves the backward pass right with a finite-difference check (see Gradient Checking).

Python Batch norm forward and backward

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "after" rows show means of 0 and standard deviations of 1 (or very close) for all three features, no matter how different the inputs were. The third feature comes out at 0.95 instead of 1: its variance (about 0.0001) is so small that eps = 1e-5 is no longer negligible next to it. The three relative errors are tiny, which confirms the backward formula.

Running statistics: training versus inference

During training, batch norm uses the statistics of the current mini-batch. That is a problem at prediction time: you may classify a single example, and "the mean of one example" is meaningless. Worse, the output for one example would depend on whichever other examples happened to share its batch.

The fix is to keep running averages of the batch means and variances while training, using an exponential moving average with a momentum such as 0.9 or 0.99:

running_mean = momentum * running_mean + (1 - momentum) * batch_mean

At inference the layer switches to evaluation mode: it normalizes with the running averages, so each example is processed independently and the output is deterministic. This is the same train-versus-eval split you saw with dropout in Dropout and Data Augmentation, and forgetting to switch modes is one of the most common bugs in practice.

Layer norm, introduced next, needs no such bookkeeping. First, the batch norm class with running statistics:

Python A BatchNorm class with train and eval modes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The running mean and standard deviation settle close to the true values (3, -2 and 2, 0.5); they start at 0 and 1, which is why the first steps are far off. In evaluation mode a single example at the mean maps to roughly zero (not exactly, because the estimates are noisy). In training mode a batch of one normalizes to exactly zero in every feature, whatever the example is, because its own mean is itself. That is why batch norm misbehaves with very small batches.

Internal covariate shift, and batch renormalization

The original batch normalization paper explained its success with internal covariate shift. Covariate shift means that the distribution of a model's inputs changes between training and use. The internal version says that the same happens inside a network: every update to layer 1 changes the distribution of the inputs to layer 2, so layer 2 chases a moving target, and the effect compounds through depth. The consequences are the ones you already know: you need small learning rates, careful initialization and activations that do not saturate. Normalizing each layer's inputs to a stable mean and variance removes the moving target.

The program below shows the shift directly: one update to the first layer changes the mean and spread of what the second layer receives. (Later research suggests that smoother loss surfaces, rather than reduced shift, may be the main benefit of normalization; the practical advice does not change.)

Batch renormalization repairs a weakness of batch norm: with tiny or non-identical batches, the batch mean and variance are noisy and differ from the running statistics used at inference, so training and testing disagree. Batch renorm normalizes with the batch statistics as usual and then applies a correction towards the running ones:

TEXT
x_hat = (x - mu_batch) / sigma_batch * r + d
r = clip(sigma_batch / sigma_running, 1/r_max, r_max)
d = clip((mu_batch - mu_running) / sigma_running, -d_max, d_max)

r and d are treated as constants in the backward pass (no gradient flows through them). The result is that training behaves as if it used the running statistics, so the model trains the way it will be used. The clipping limits start at r = 1, d = 0 (ordinary batch norm) and are relaxed gradually. When the batch is small, group normalization or layer normalization are the other common answers.

Python Internal covariate shift and batch renormalization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Layer normalization

Layer norm flips the axis. For each example it computes the mean and variance across that example's own features, then normalizes and rescales with a learned gamma and beta:

Python
mu = x.mean(axis=1, keepdims=True)
var = x.var(axis=1, keepdims=True)
x_hat = (x - mu) / np.sqrt(var + eps)
y = gamma * x_hat + beta

Because the statistics come from a single example, layer norm behaves identically in training and evaluation, works with a batch of one, and handles variable-length sequences cleanly. That makes it the standard choice in recurrent networks and in transformers (see The Transformer Block). Batch norm remains popular in convolutional networks, where batches are large and the statistics are stable.

Batch norm Layer norm
Statistics over the batch, per feature the features, per example
Depends on other examples yes no
Needs running averages yes no
Train and eval differ yes no
Typical home CNNs, large batches transformers, RNNs, small batches
Python Batch norm versus layer norm on one array

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Batch norm gives zero-mean columns; layer norm gives zero-mean rows. And changing one example leaves every other example's layer-normed output untouched, while batch norm lets that example change everyone's output.

Where normalization goes

The classic placement is after the linear (or convolution) layer and before the activation: Linear -> BatchNorm -> ReLU. Some architectures put it after the activation instead, and transformers use layer norm either before the attention and MLP sublayers ("pre-norm") or after them ("post-norm"). Two practical details:

  • The layer before a batch norm does not need its own bias, since batch norm subtracts the mean and beta takes over that job.
  • Normalization layers have their own parameters (gamma, beta), which should usually be excluded from weight decay.

Seeing the speed-up

The last example trains the same four-hidden-layer sigmoid network on the two-moons data twice, with and without batch norm. Sigmoid layers saturate easily, so this is exactly the kind of network that benefits. Everything is written compactly; the backward pass reuses the batch norm functions from the first example.

Python Training with and without batch norm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain network sits near the "I know nothing" loss of about 0.69 for a long time, because the gradient has to travel back through four saturating sigmoids and arrives tiny. With batch norm every layer's input is re-centered, the sigmoids stay in their sensitive range, and the loss drops much sooner.

Batch norm adds a dependency between examples in a batch, so it needs reasonably large batches (a few dozen or more) to give stable statistics. With tiny batches, prefer layer norm or similar per-example methods such as group norm.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In batch norm, over which axis are the mean and variance computed for an input of shape (N, D)?

    Batch norm gives each of the D features its own mean and variance, computed across the N examples. Layer norm is the one that works along axis 1.
  2. 2. Why does batch norm keep running averages of the mean and variance?

    At prediction time there may be a single example, so batch statistics are unreliable. Running averages gathered during training are used in evaluation mode instead.
  3. 3. Which statement about layer norm is true?

    Layer norm computes its statistics per example, so it is independent of batch size and identical in both modes. It still has learnable gamma and beta.
  4. 4. What is the purpose of gamma and beta?

    Standardizing forces zero mean and unit variance. Gamma and beta restore flexibility, and the layer can even learn to undo the normalization.

Summary

  • Normalization layers re-center and re-scale activations inside the network, keeping signals and gradients in a healthy range.
  • Batch norm uses per-feature statistics across the batch; it needs running averages for inference and differs between train and eval modes.
  • Layer norm uses per-example statistics across features, so it works with any batch size and is the standard choice in transformers and RNNs.
  • Learned gamma and beta let each layer choose its own scale after standardizing.
  • The backward pass of batch norm has a compact closed form that you can verify with gradient checking.
  • Typical placement is linear layer, then normalization, then activation, and the speed-up comes mostly from a smoother, easier-to-optimize loss surface.

Next: Evaluating Models.