Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 40 ยท Building Models with Keras

Initializers and Regularizers

Learn how weight initialization (Glorot, He) keeps signals stable, how L1 and L2 regularizers shrink weights, and how to set both on Keras layers.

Two small arguments on every Keras layer have an outsized effect on training: kernel_initializer, which decides the starting weights, and kernel_regularizer, which adds a penalty that discourages large weights. Starting values decide whether a deep network trains at all; the penalty decides how much it overfits. This lesson explains both, with experiments you can run.

Why initialization matters

Weights cannot all start at zero. If every weight in a layer is identical, every unit computes the same output and receives the same gradient, so they stay identical forever: the layer behaves like one unit. Random initialization breaks the symmetry.

But random is not enough; the scale matters. As a signal passes through a layer, its size is multiplied by something like the weight scale and the square root of the layer's width. Over many layers:

  • weights too small: the signal shrinks toward zero (vanishing activations and gradients);
  • weights too large: the signal grows until activations saturate or overflow (exploding);
  • weights scaled to the layer's size: the signal stays at a steady size from layer to layer.

Let us watch this happen. We push random data through 20 ReLU layers of width 256 and print the typical size (standard deviation) of the activations.

Python Stand-in: signal size through 20 layers for different scales (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Zeros give nothing; too-small weights make the signal vanish; too-big weights blow it up by orders of magnitude; the He scale keeps it steady. The same reasoning applies to the gradients flowing backward.

The standard initializers

Keras bundles schemes that choose the scale from the layer size (fan_in = number of inputs, fan_out = number of outputs).

Initializer Rule Typically used with
glorot_uniform (default) uniform limit sqrt(6 / (fan_in + fan_out)) tanh, sigmoid, softmax, general default
glorot_normal normal, std sqrt(2 / (fan_in + fan_out)) same
he_normal normal, std sqrt(2 / fan_in) ReLU and its variants
he_uniform uniform, limit sqrt(6 / fan_in) ReLU and its variants
lecun_normal normal, std sqrt(1 / fan_in) SELU
zeros, ones constant biases, special cases
random_normal, random_uniform you set stddev or range experiments
orthogonal orthogonal matrix recurrent layers

Biases default to zeros, which is fine because the weights already break the symmetry.

Python
from keras import layers, initializers

layers.Dense(64, activation="relu", kernel_initializer="he_normal")

layers.Dense(64, kernel_initializer=initializers.RandomNormal(stddev=0.05, seed=1),
             bias_initializer=initializers.Constant(0.1))

In practice: leave the default for most layers, and pick he_normal when you use ReLU in a deep network. Modern normalization layers also make training much less sensitive to initialization.

What the Glorot rule produces

The numbers are small and depend on the layer size. Compute a few and sanity check them:

Python Stand-in: Glorot and He scales for some layer sizes (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Regularization: penalizing large weights

An overfit network often has large weights that make it react wildly to small input changes. A regularizer adds a penalty to the loss for large weights, nudging the optimizer toward simpler functions:

total loss = data loss + lambda * penalty(weights)

  • L2 (weight decay): penalty = sum(w**2). Shrinks all weights smoothly toward zero.
  • L1: penalty = sum(|w|). Pushes many weights exactly to zero, producing sparse models.
  • L1L2 ("elastic net"): both.

lambda is the regularization strength; typical values are 1e-5 to 1e-2. Keras attaches regularizers to a layer's kernel, bias or output:

Python
from keras import regularizers

layers.Dense(
    64,
    activation="relu",
    kernel_regularizer=regularizers.L2(1e-4),
    bias_regularizer=None,
    activity_regularizer=regularizers.L1(1e-5),   # penalizes the outputs
)

regularizers.L1L2(l1=1e-5, l2=1e-4)

Keras adds these penalty terms to the loss automatically during training. They are collected in model.losses, and the loss you see in fit output includes them, which is why the training loss can look higher than the loss computed without regularization.

L2 in action

The gradient of the L2 term lambda * sum(w**2) is 2 * lambda * w, so every step multiplies each weight by a factor slightly under 1 ("decay") on top of the usual update. Here we fit a model with 30 features to only 40 noisy examples, so it can overfit easily, with and without L2. Compare the weight sizes and test error.

Python Stand-in: L2 regularization on an overfit-prone problem (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see the weight norm shrink as l2 grows. A moderate value usually lowers the test error compared with none, while a value that is too large underfits (the test error rises again). That U-shaped behavior is why lambda is a hyperparameter you tune on validation data.

L1 and sparsity

L1's gradient is lambda * sign(w): a constant-size pull toward zero, however small the weight. L2's pull fades as the weight shrinks, so it never quite reaches zero. This is why L1 produces sparse weights and L2 does not. In deep learning, L2 (often as weight decay inside the optimizer, such as AdamW) is far more common than L1.

Other ways to regularize (preview)

Weight penalties are one tool. Chapter 4 adds dropout, early stopping and data augmentation, which usually matter more for large networks. Often you combine several.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can't all weights be initialized to zero?

    Identical weights stay identical under gradient descent; random initialization breaks the symmetry.
  2. 2. Which initializer is usually recommended with ReLU activations?

    He initialization uses std sqrt(2 / fan_in), which compensates for ReLU zeroing half of the signal.
  3. 3. What does an L2 regularizer add to the loss?

    L2 adds lambda * sum(w**2); L1 uses absolute values.
  4. 4. Which regularizer tends to produce exactly-zero (sparse) weights?

    L1's constant pull toward zero can drive small weights exactly to zero.
  5. 5. A very large regularization strength typically causes what?

    Too strong a penalty prevents the model from fitting real patterns.

Summary

  • Initialization sets the starting weights; zeros fail, bad scales vanish or explode the signal, He and Glorot scale by layer size.
  • Default is glorot_uniform; prefer he_normal for deep ReLU networks.
  • Regularizers add lambda * penalty(weights) to the loss: L2 shrinks weights, L1 makes them sparse.
  • Set them with kernel_initializer= and kernel_regularizer=; Keras collects the penalties in model.losses.
  • The strength is a hyperparameter: too weak overfits, too strong underfits.

Next: Loss Functions