Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 73 ยท Neural Networks From Scratch

Weight Initialization

Learn why zero weights fail, how signals vanish or explode in deep networks, and how Xavier and He initialization keep training stable, with NumPy code.

Before the first training step, every weight in your network needs a starting value. It sounds like a detail, but it can decide whether the network learns at all. Start every weight at zero and the network can never break its own symmetry. Start with values that are a little too big or a little too small and, ten layers later, the signal has either exploded into huge numbers or faded to almost nothing.

In Your First Trained Network we used small random numbers and it worked, because the network was shallow. This lesson explains what "small random numbers" should really mean, and gives you the two standard recipes, Xavier (Glorot) and He initialization, that every framework uses by default.

Three ways to start badly

All zeros Every unit computes and learns the same thing.
Too small The signal shrinks layer by layer until it vanishes.
Too large The signal grows or saturates and gradients die.

A good start keeps the size of the signal steady through every layer.

1 / 2

Why zeros fail: the symmetry problem

Imagine a hidden layer with eight units whose weights all start at the same value. Every unit sees the same inputs and applies the same weights, so every unit produces the same output. During backpropagation every unit receives the same gradient, so every unit is updated in the same way. After a million steps they are still identical. Eight units behave like one unit, and you have paid for eight.

Zero is just the most extreme case of this. The next example trains the same small network three times on the two-moons data: with all-zero weights, with all weights equal to 0.5, and with random weights. It then checks how many different columns the first weight matrix has afterwards, since each column belongs to one hidden unit.

Python Identical weights stay identical

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With zero weights the accuracy is exactly 0.5 and nothing learns: the output weights are zero, so no gradient reaches the hidden layer at all. With all weights equal to 0.5 the network does learn something, but all eight hidden units remain copies of each other, so it is effectively a network with one hidden unit and it scores worse. Random weights give every unit a different starting point (eight distinct units out of eight), and the network reaches 0.955.

Biases are different. Starting biases at zero is perfectly fine, because the random weights already break the symmetry. Frameworks initialize biases to zero by default.

So the first rule is: weights must be random. The second question is how large the random numbers should be.

Signals that vanish or explode

Consider one layer that computes h = activation(x @ W). If the input x has n features, each output is a sum of n products. A sum of many independent terms has a variance that adds up: with unit-variance inputs and weights of variance v, the output variance is about n * v. For a layer with 256 inputs and weights drawn from a normal distribution with standard deviation 1, the spread of the outputs is about 16 times that of the inputs, in a single layer. Stack ten layers and the numbers multiply.

The opposite happens with tiny weights. With a standard deviation of 0.01, each layer multiplies the spread by about 0.16, and ten layers shrink it by a factor of a hundred billion.

Why does this matter for learning? Because backpropagation reuses the same weights and activations. If the forward signal vanishes, so do the gradients, and the early layers get no useful updates. If the signal explodes, gradients explode, or, with squashing activations like tanh, the units saturate at plus or minus one where the slope is nearly zero (see Activation Functions).

The next example pushes a batch of random data through ten layers of 256 units and prints the standard deviation of the activations at several depths, for four initialization schemes. Two of them are the recipes we are about to derive.

Python Watching the signal through ten layers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table row by row:

  • N(0, 0.01) collapses with both activations. By layer 10 the tanh signal is around 1e-8 and the ReLU signal around 1e-10.
  • N(0, 1) with tanh stays near 0.97 forever, which looks stable but is a trap: the units are pinned at the extremes of tanh, so they output almost only +1 or -1 and their slopes are close to zero. With ReLU the same weights explode to the order of 1e10.
  • Xavier keeps tanh healthy and decays only slowly. With ReLU it shrinks steadily to about 0.025 by layer 10.
  • He holds ReLU steady near 0.8 through all ten layers.

The idea behind the recipes

The goal is simple: choose the weight variance so that the activations have about the same variance in every layer, in the forward pass, and ideally the gradients do too in the backward pass.

For a layer with fan_in inputs, the variance of the pre-activation is roughly fan_in * Var(w) * E[x^2]. To keep it equal to E[x^2] you need

TEXT
Var(w) = 1 / fan_in

The backward pass sends gradients through the transposed matrix, which sums over fan_out terms, so it would prefer 1 / fan_out. Xavier (Glorot) initialization compromises with the average of the two:

TEXT
Var(w) = 2 / (fan_in + fan_out)

It assumes the activation is roughly linear around zero, which is true for tanh and (near the origin) for sigmoid. There is also a uniform version: draw weights from U(-a, a) with a = sqrt(6 / (fan_in + fan_out)), which has the same variance.

ReLU breaks that assumption. It throws away every negative pre-activation, so on average only half of the signal's energy survives each layer. To compensate, He initialization (also called Kaiming) doubles the variance:

TEXT
Var(w) = 2 / fan_in

In code, the standard deviation is the square root of the variance:

Python
def xavier_normal(n_in, n_out, rng):
    return rng.normal(0, np.sqrt(2 / (n_in + n_out)), (n_in, n_out))

def he_normal(n_in, n_out, rng):
    return rng.normal(0, np.sqrt(2 / n_in), (n_in, n_out))

Our layers above were square (256 by 256), where Xavier's variance equals 1 / fan_in. For layers with different widths the two terms differ, and Xavier's compromise matters more.

Rule of thumb: ReLU and its relatives (leaky ReLU, GELU variants) get He. Tanh, sigmoid and softmax output layers get Xavier. Many frameworks apply a sensible default per layer type, but when you build a network by hand you pick.

Seeing it: histograms and a decay curve

Numbers in a table are easy to forget, but a picture of the activations makes the saturation problem obvious. The next example plots the tanh activations at layer 5 for the bad N(0, 1) start and for Xavier, then compares how the spread of ReLU activations evolves for Xavier and He over 10 layers on a log scale.

Python Saturated tanh versus healthy tanh

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The red histogram has two tall spikes at the edges: about 86% of the outputs sit beyond 0.99 in magnitude, so the units are saturated and learning will crawl. With Xavier, none do. The green one is a smooth bump, so most units are in the sensitive middle region. On the right, the Xavier line for ReLU slopes down while the He line stays flat.

Practical notes

  • Fan in and fan out. For a dense layer with weights of shape (n_in, n_out), fan_in is n_in and fan_out is n_out. For convolution layers, both include the kernel area (you will meet this in Convolution, Pooling and Feature Maps).
  • Normal or uniform? Either works as long as the variance is right. Normal is the more common choice for He, uniform for Xavier in older libraries.
  • Output layers. For a final softmax or sigmoid layer, smaller weights are common so the first predictions are not overconfident. Starting near 50/50 gives the loss a sensible value to begin with.
  • Normalization layers such as batch norm (Batch Norm and Layer Norm) make training much less sensitive to initialization, but they do not make it irrelevant.
  • Residual networks often scale the last layer of each block down (even to zero) so that each block starts as an identity function.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does initializing all weights of a layer to the same value prevent the layer from learning useful features?

    Identical weights give identical outputs and identical gradients, so the units never differ. Random initialization breaks this symmetry.
  2. 2. A 10-layer ReLU network is initialized with N(0, 1) weights and 256 units per layer. What happens to the activations?

    Each layer multiplies the spread by about sqrt(fan_in) = 16, so after ten layers the values are astronomically large.
  3. 3. Which standard deviation does He initialization use for a dense layer with fan_in inputs?

    He initialization uses variance 2 / fan_in. The extra factor of 2 compensates for ReLU zeroing out about half of its inputs.
  4. 4. Which pairing is the usual recommendation?

    Xavier assumes an activation that is roughly linear near zero, as tanh is. He accounts for ReLU discarding the negative half.
  5. 5. Is it a problem to start all bias values at zero?

    Symmetry between units is broken by their different weight vectors, so zero biases are the normal default.

Summary

  • Weights must start random: identical starting weights make units that stay identical forever.
  • If the weight variance is too small the signal and gradients vanish; too large and they explode or saturate the activations.
  • The goal is to keep the activation variance roughly constant from layer to layer.
  • Xavier (Glorot) uses variance 2 / (fan_in + fan_out) and suits tanh and sigmoid.
  • He uses variance 2 / fan_in and suits ReLU-family activations.
  • Biases normally start at zero.

Next: Momentum, RMSProp and Adam.