Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 45 ยท Building Models

Initialization, Normalization and Dropout

Learn how Xavier and Kaiming initialization, BatchNorm, LayerNorm and Dropout keep deep networks trainable, and how train and eval modes change them.

A deep network is a long chain of multiplications. If each layer makes the numbers a little bigger, the signal explodes by the last layer. If each layer makes them a little smaller, it fades to zero. In both cases the gradients on the way back suffer the same fate, and training stalls before it starts. Three families of tools keep the numbers in a healthy range: careful initialization, normalization layers and dropout (which fights a different problem, overfitting).

This lesson follows Activation Functions and Loss Functions. As in earlier lessons, the real PyTorch code is read-only because PyTorch cannot run in the browser. Every idea also has a live NumPy stand-in that you can run and change, so you can watch the variance of activations and the train/eval switch yourself.

Keep the signal alive

Initialization Pick starting weights with the right scale.
Normalization Re-center and re-scale activations as they flow.
Dropout Randomly silence units while training.

The first two help optimization; dropout helps generalization.

1 / 2

Why the starting weights matter

A linear layer computes y = W x. Suppose every input value has variance 1 and the weights are random with variance v. Each output is a sum of fan_in products, so its variance is roughly fan_in * v. With 256 inputs, picking v = 1 makes the variance 256 times larger after every layer. Picking v = 0.0001 makes it about 40 times smaller each time. After ten layers neither survives.

The cure is to tie the weight variance to the layer size. If v = 1 / fan_in, the output variance stays close to the input variance.

Xavier and Kaiming

  • Xavier (Glorot) initialization balances the forward and backward passes by using both fan_in and fan_out. It was designed for symmetric activations such as tanh and sigmoid.
  • Kaiming (He) initialization accounts for ReLU zeroing out about half of its inputs. It doubles the variance to 2 / fan_in to compensate. Use it with ReLU and its relatives.

In PyTorch you apply them with functions in torch.nn.init. The functions that end in an underscore change the tensor in place:

Python
import torch.nn as nn

layer = nn.Linear(256, 128)

nn.init.xavier_uniform_(layer.weight)                       # Xavier, uniform
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")  # Kaiming, normal
nn.init.zeros_(layer.bias)                                  # biases usually start at 0

You rarely have to do this by hand. nn.Linear and nn.Conv2d already start with a sensible Kaiming-style uniform scheme, so initialization becomes your job mainly for custom layers, very deep models or special architectures. To initialize a whole model, write a small function and pass it to model.apply, which calls it on every submodule:

Python
def init_weights(module):
    if isinstance(module, nn.Linear):
        nn.init.kaiming_normal_(module.weight, nonlinearity="relu")
        nn.init.zeros_(module.bias)

model.apply(init_weights)

Watching the variance live

The next example builds a ten-layer network in plain NumPy, feeds in random data and reports the standard deviation of the activations after layers 1, 3, 5, 7 and 10. This is a stand-in for a real PyTorch model, but the arithmetic of the forward pass is identical.

Python Activation spread through ten layers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the numbers:

  • With tiny weights (small), the tanh activations shrink from 0.1554 after the first layer to essentially zero by layer 7. Gradients would be just as dead.
  • With large weights (large), every unit saturates at the flat ends of tanh, and the spread sticks near 0.97. The values look healthy, but the derivative of tanh is almost zero there, so gradients vanish anyway.
  • Xavier keeps a tanh network alive: the spread declines slowly (0.6260 down to 0.2249) instead of collapsing.
  • In a ReLU network, Xavier shrinks the signal to 0.0233 by layer 10, while Kaiming holds it near 0.75. That is the factor 2 doing its job.

Try changing layers=10 to 40 and see which initialization still works.

Python Plot the spread layer by layer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the log scale the fixed-scale and Xavier-style lines fall steeply, while the Kaiming line stays roughly flat. Initialization is the cheapest way to make a deep network trainable.

Normalization layers

Good initialization only fixes the starting point. Once the weights change during training, the scale of the activations can drift again. Normalization layers repair this continuously: they subtract a mean and divide by a standard deviation, then apply a learned scale (gamma) and shift (beta) so the network can still represent whatever it needs.

The two layers you will meet most differ only in which numbers they average over:

Layer Normalizes over Depends on batch size? Typical home
nn.BatchNorm1d / BatchNorm2d each feature across the batch yes MLPs and CNNs
nn.LayerNorm all features of one sample no Transformers, RNNs

BatchNorm

For each feature, BatchNorm computes the mean and variance across the current batch, normalizes with them, and keeps running averages of those statistics. During training it uses the batch statistics. During evaluation it uses the stored running averages, because a prediction should not depend on which other samples happen to share its batch.

Python
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(20, 64),
    nn.BatchNorm1d(64),   # one mean/variance per each of the 64 features
    nn.ReLU(),
    nn.Linear(64, 10),
)

Typical placement is between the linear (or convolution) layer and the activation, though putting it after the activation also works. Because BatchNorm has its own learned shift, the bias of the layer before it is redundant; nn.Linear(20, 64, bias=False) saves a few parameters.

BatchNorm has two practical weaknesses. A batch of one cannot give a meaningful variance, and a very small batch gives noisy statistics. That is the main reason Transformers use LayerNorm instead.

LayerNorm

LayerNorm ignores the other samples. For each sample it computes the mean and variance over the feature dimension and normalizes with those. Training and evaluation behave identically, and there are no running averages to maintain.

Python
norm = nn.LayerNorm(512)        # normalizes the last dimension, size 512
y = norm(x)                     # x has shape (batch, seq_len, 512)

Both layers are tiny compared with the rest of the model, but they often allow a larger learning rate and make results less sensitive to initialization.

Both layers in NumPy

The stand-in below implements the same arithmetic. It also shows the thing beginners find surprising: the layer holds state (the running mean and variance) and behaves differently in each mode.

Python BatchNorm, LayerNorm and Dropout by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What the output shows:

  • Every LayerNorm row has mean 0 and standard deviation 1 (the -0. entries are tiny negative rounding errors), and so does every BatchNorm column.
  • After 60 batches the running mean has moved from its starting value of 0 to roughly 4.4 to 5.0 for each feature, close to the true mean of 5 but still noisy, because momentum 0.1 gives each batch a small say.
  • In eval mode the layer normalizes a single sample using the stored statistics. Without running averages this would be impossible, since one row has no meaningful batch variance.

Internal covariate shift and why BatchNorm helps

A covariate shift is a change in the distribution of a model's inputs between training and use (a spam filter trained on last year's emails reading this year's). Internal covariate shift is the same problem inside the network: while the weights of layer 1 change during training, the distribution of the values arriving at layer 2 keeps moving, so layer 2 is chasing a target that never stands still. In a deep stack the drift compounds. Typical consequences are saturated activations (a sigmoid or tanh fed values that are too large gives a near-zero gradient), vanishing or exploding gradients, a need for small learning rates and a high sensitivity to initialization.

Batch normalization was introduced to attack exactly this: re-center and re-scale each layer's inputs for every mini-batch, so the next layer sees a stable distribution whatever the previous layer is doing. Whether stabilizing distributions is the real reason it works is debated (later work argues it mainly smooths the loss landscape), but the practical benefits are well established:

  • Faster, more stable training and the ability to use larger learning rates.
  • Less sensitivity to initialization.
  • Mild regularization, because each sample is normalized with statistics of a random batch, which adds a little noise (sometimes allowing less dropout).
  • Less saturation of sigmoid and tanh units, and healthier gradients in deep networks.

The computation step by step

For a mini-batch of m values of one feature:

  1. Mean: mu = mean(x)
  2. Variance: var = mean((x - mu)**2)
  3. Normalize: x_hat = (x - mu) / sqrt(var + eps) (the small eps, default 1e-5, avoids dividing by zero)
  4. Scale and shift: y = gamma * x_hat + beta, with gamma and beta learned
  5. Update running statistics: running_mean = (1 - momentum) * running_mean + momentum * mu (and the same for the variance), used at evaluation time.

Step 4 matters: if normalization hurt, the network can learn gamma = sqrt(var) and beta = mu and undo it.

nn.BatchNormXd and where to put it

PyTorch has one class per input shape: nn.BatchNorm1d(num_features) for (N, C) or (N, C, L), nn.BatchNorm2d(C) for images (N, C, H, W) and nn.BatchNorm3d(C) for volumes. num_features is the number of channels (or features), because each channel gets its own statistics, computed over the batch and the spatial positions. Putting it between a convolution and its activation is the classic arrangement:

Python
import torch.nn as nn

block = nn.Sequential(
    nn.Conv2d(3, 32, kernel_size=3, padding=1, bias=False),   # bias is redundant before BN
    nn.BatchNorm2d(32),
    nn.ReLU(inplace=True),
)
print(block[1].weight.shape, block[1].bias.shape)    # gamma and beta: torch.Size([32]) each
print(block[1].running_mean.shape)                    # a buffer, not a parameter

running_mean and running_var are buffers: saved in the state_dict but not trained by the optimizer. Remember model.eval() before inference so they are used instead of batch statistics.

Python Why re-normalizing keeps sigmoids alive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without normalization the more the previous layer drifts, the smaller the gradient the sigmoid passes back; with it, the slope stays the same. That is the "internal covariate shift" story in numbers.

Batch Renormalization and other normalizers

BatchNorm degrades with tiny batches (noisy statistics) and when batch statistics differ from the running averages. Batch Renormalization adds correction factors that tie the batch statistics to the running ones, so training and evaluation agree more closely. It is not part of core PyTorch, but the idea explains why alternatives exist. The common ones in PyTorch:

Layer Statistics over Good for
nn.BatchNorm2d batch and spatial positions, per channel CNNs with reasonable batch sizes
nn.LayerNorm all features of one sample Transformers, RNNs
nn.GroupNorm(groups, C) one sample, groups of channels CNNs with small batches (detection, segmentation)
nn.InstanceNorm2d one sample, one channel Style transfer, image generation
nn.RMSNorm root mean square of one sample (no mean subtraction) Recent large language models
nn.utils.parametrizations.weight_norm normalizes the weights, not the activations Reparameterizing layers

Pick the layer by asking: do I have large batches (BatchNorm), variable-length sequences or tiny batches (LayerNorm, GroupNorm), or per-image styles (InstanceNorm)?

Dropout

Large networks can memorize their training data. Dropout discourages that by randomly setting a fraction p of activations to zero on every training step, so no unit can rely on one specific neighbor always being present. It acts like training a large collection of thinner networks that share weights.

PyTorch uses inverted dropout: during training the surviving values are divided by 1 - p, so the expected value of each activation is unchanged. That means nothing needs rescaling at evaluation time, where dropout simply passes values through.

Python
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(p=0.3),     # silence 30% of the 256 activations while training
    nn.Linear(256, 10),
)

In the NumPy output above, the training call zeroes some entries and doubles the others (p = 0.5, so the survivors are scaled by 2), while the evaluation call returns the input unchanged. Over many values the mean stays near 1, which is exactly the property inverted dropout is built to have. Typical values are p = 0.1 to 0.5; Transformers often use around 0.1.

Dropout is a regularizer, so it usually slows training a little and only pays off when the model would otherwise overfit. Hold it back until you see the gap between training and validation loss. The next chapters return to this in Overfitting and Regularization.

Train mode versus eval mode

Dropout and BatchNorm are the two standard layers whose behavior depends on a flag stored inside the module. You flip the flag for the whole model with two methods:

Python
model.train()   # dropout active, BatchNorm uses batch statistics and updates running ones
model.eval()    # dropout off,    BatchNorm uses the stored running statistics

A training loop therefore starts each epoch with model.train() and runs validation after model.eval(). model.eval() does not stop gradient tracking; to save memory during evaluation you additionally wrap the code in torch.no_grad(), a topic covered in Autograd in Practice. The two are independent switches, and you normally want both during validation.

Python
model.eval()
with torch.no_grad():
    predictions = model(batch)
model.train()               # do not forget to switch back

Signs that you forgot the mode: validation accuracy that jumps around from run to run (dropout is still active), or predictions that change depending on the other items in the batch (BatchNorm is still using batch statistics).

Choosing among them

  • Use PyTorch's default initialization unless you have a reason not to. Reach for Kaiming with ReLU and Xavier with tanh when you write custom layers or very deep stacks.
  • Add BatchNorm to CNNs and MLPs trained with reasonable batch sizes. Use LayerNorm for sequence models and when batches are tiny.
  • Add Dropout only when you observe overfitting.
  • Always pair model.train() and model.eval() with the right part of your loop.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does a ReLU network use Kaiming initialization (variance 2 / fan_in) rather than Xavier-style 1 / fan_in?

    ReLU discards the negative half of the values, which roughly halves the variance at each layer. Doubling the weight variance cancels that loss and keeps the activations from fading.
  2. 2. What does nn.LayerNorm normalize over?

    LayerNorm computes a mean and variance for each sample over its feature dimension, so the result does not depend on the batch and behaves the same in train and eval mode.
  3. 3. A model with BatchNorm gives different predictions for the same input depending on the other items in the batch. What is the most likely cause?

    In train mode BatchNorm normalizes with the current batch's statistics. Calling model.eval() makes it use the stored running averages, which do not depend on batch companions.
  4. 4. In PyTorch's inverted dropout with p = 0.5, what happens to the values that survive during training?

    Dividing the survivors by 1 - p keeps the expected value of each activation unchanged, so evaluation mode can simply pass values through with no rescaling.
  5. 5. Does model.eval() turn off gradient tracking?

    eval() switches the mode of layers like Dropout and BatchNorm. Gradient tracking is a separate switch controlled by torch.no_grad() or torch.inference_mode().

Summary

  • Deep networks multiply many matrices, so activations and gradients can explode or vanish; the weight scale decides which.
  • Xavier initialization suits tanh and sigmoid; Kaiming (2 / fan_in) suits ReLU. PyTorch layers already start with a sensible default; nn.init.*_ functions and model.apply let you override it.
  • BatchNorm normalizes each feature over the batch and keeps running statistics; LayerNorm normalizes each sample over its features and has no batch dependence.
  • Dropout randomly zeroes activations during training and rescales the survivors by 1 / (1 - p); it is a regularizer for models that overfit.
  • model.train() and model.eval() switch dropout and BatchNorm behavior; torch.no_grad() separately turns off gradient tracking.

Next, put the pieces to work with Optimizers: SGD, Momentum, Adam.