Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 73 ยท Training in Practice

Overfitting, Underfitting and Regularization

Understand train/validation/test splits, bias and variance, learning curves, L1 and L2 weight decay and early stopping, with NumPy examples on noisy data.

A model that scores 100% on the data it trained on has not necessarily learned anything useful. It may simply have memorized the training examples, noise and all, and will stumble on data it has never seen. This failure is called overfitting, and it is the central problem of machine learning: we do not want a model that is good on the training set, we want one that generalizes to new data.

This lesson shows how to detect overfitting (with a proper data split and learning curves), explains the bias-variance idea behind it, and then covers the three regularization tools you will use constantly: weight decay (L2 and L1 penalties) and early stopping. Dropout and Data Augmentation completes the toolbox in the next lesson.

Underfit, good fit, overfit

Underfitting Too simple. Bad on training and new data.
Good fit Captures the pattern, ignores the noise.
Overfitting Great on training, bad on new data.

The gap between training and validation error is the warning sign.

1 / 2

Train, validation and test sets

You cannot judge generalization on the data you trained on, so you hold some data back:

  • The training set is the only data used to compute gradients.
  • The validation set is used while you build the model: to compare architectures, pick the learning rate, and decide when to stop training. Because you make decisions based on it, it slowly "leaks" into your choices.
  • The test set is used once, at the end, to report the final number. If you look at it repeatedly and adjust the model accordingly, it becomes another validation set and your estimate turns optimistic.

Common ratios are 60/20/20 or 80/10/10, but with millions of samples a few percent is enough for validation and test. Split before any preprocessing that learns from data (such as computing a mean for normalization), and shuffle first unless the data is a time series, where you split by time to avoid training on the future.

Bias, variance and model capacity

Two kinds of error compete:

  • Bias is error from a model that is too rigid to capture the true pattern. A straight line fitted to a curve has high bias. This is underfitting: the error is high on training and validation data alike.
  • Variance is error from a model that is too flexible and reacts to the random quirks of the particular training sample. It would change a lot if you drew a different sample. This is overfitting: training error is low, validation error is high.

Increasing a model's capacity (more polynomial degree, more layers, more units) lowers bias and raises variance. The sweet spot is in between. We can see it with a small experiment: fit polynomials of increasing degree to 25 noisy points of a sine curve, then measure the error on 200 fresh points.

Python Capacity, bias and variance with polynomials

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table shows the classic shape. Training error only goes down as the degree rises (0.225, 0.045, 0.037, 0.036, 0.018). Validation error falls to a minimum at degree 5 (0.078) and then climbs: 0.122 at degree 9, and an absurd 398 at degree 15, where the curve whips wildly between the training points. In the plot, the dashed gray line is the true sine; the degree-1 line misses its bend, the degree-3 curve follows it, and the degree-15 curve chases the noise.

Learning curves: what to look at

A learning curve plots training and validation loss as training proceeds. Reading the two lines together tells you what to do next:

What you see Diagnosis What to try
Both curves high and close together Underfitting (high bias) Bigger model, train longer, better features, less regularization
Training low, validation much higher, gap growing Overfitting (high variance) More data, regularization, dropout, augmentation, smaller model, early stopping
Both low and close Good Ship it, or squeeze out small gains
Validation falls, then turns upward while training keeps falling Overfitting began at the turning point Stop there (early stopping)

Weight decay: L2 and L1 penalties

Overfit networks typically have large weights that bend the function sharply to hit each training point. Regularization adds a cost for large weights, nudging the network toward smoother functions.

L2 regularization adds lambda * sum(w^2) to the loss. The gradient of that term is 2 * lambda * w, so each update becomes

TEXT
w = w - lr * (gradient_of_loss + wd * w)

with wd standing for the decay constant (some libraries fold the factor 2 into it). Every step pulls each weight a little toward zero in proportion to its size, which is why this is called weight decay. Biases are usually left out.

L1 regularization adds lambda * sum(|w|) instead. Its pull toward zero has constant strength regardless of size, so it drives many weights to exactly zero and produces sparse models. In practice L1 is applied with a soft-thresholding step: shrink each weight by lr * lambda, and clamp at zero if it crosses.

The next example compares no penalty, L2 and L1 on a linear model with 20 features, of which only the first four matter, trained on just 24 noisy samples. This is a deliberately easy-to-overfit setup.

Python L2 and L1 penalties on a small dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without a penalty the model fits the 24 training points almost perfectly (training error 0.69) but does badly on new data (validation error 17.4). L2 accepts a worse training fit (2.05) and gets a much better validation error (9.2) by shrinking all weights. L1 does better still here (training 3.24, validation 5.96) and sets 12 of the 20 weights to exactly zero, mostly the 16 features that carry no signal. L1 is a built-in feature selector, while L2 spreads the shrinkage over everything.

In the Adam family, use AdamW so the decay is applied directly to the weights rather than mixed into the adaptive gradient. See Momentum, RMSProp and Adam.

Early stopping

The cheapest regularizer is to stop training at the right moment. Networks tend to learn the broad pattern first and memorize details later, so validation loss typically falls, bottoms out and then rises. Early stopping watches the validation loss after every epoch, remembers the weights from the best epoch, and stops once there has been no improvement for patience epochs. Afterwards you restore the best weights.

The final example puts everything together on a classification problem built to overfit: a 64-unit network trained on only 60 very noisy points (moons with noise 0.45), with 300 validation points and 640 test points. It trains three ways: with no protection, with L2 weight decay of 0.01, and with early stopping (patience 50) and no penalty.

Python Learning curves with and without protection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the plot, the blue training loss sinks toward zero while the red validation loss bottoms out near epoch 107 and then climbs steeply: the textbook overfitting picture. The green L2 validation curve stays flat and low. In the table, the unprotected model ends with a training loss of 0.012 but a validation loss of 1.48, which is poor. Early stopping restores the epoch-107 weights and halts after 158 epochs instead of 1,500; its validation loss (0.390) and test accuracy (0.844) are the best of the three, against 0.789 for the unprotected model. L2 gives a healthy test accuracy of 0.816 and a training loss that stays honest (0.356).

One dataset and one split give only a rough comparison; differences of a few percentage points on 640 test samples can change with another seed. Trust the pattern (a widening gap and a rising validation loss), not the exact numbers.

Putting it together

  • Get more data first. Nothing beats it, and data augmentation is a cheap way to approximate it.
  • Start simple, confirm you can overfit a small sample (Debugging Training Runs explains this sanity check), then add capacity or regularization.
  • Combine tools: weight decay, dropout, early stopping and augmentation work together. Typical values: weight decay 1e-4 to 1e-2, patience 5 to 20 epochs.
  • Tune on validation, report on test, once.
  • Check for leaks: if validation accuracy looks too good, make sure no duplicates or future information crossed from train into validation.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A model has 99% training accuracy but 70% validation accuracy. What is the most likely problem?

    A large gap between training and validation performance means the model memorized training data instead of generalizing.
  2. 2. Why should the test set be used only once, at the end?

    Any set you tune against stops being an unbiased estimate of performance on truly new data.
  3. 3. How does L2 weight decay change the update rule?

    The update becomes w = w - lr * (gradient + wd * w), so each step shrinks the weight in proportion to its size.
  4. 4. What is a distinctive effect of L1 regularization compared with L2?

    L1's pull toward zero does not weaken as weights get small, so many weights reach zero exactly.
  5. 5. In early stopping with patience 20, when does training stop?

    Patience counts epochs since the best validation result; afterwards you restore the best weights.

Summary

  • Overfitting means low training error but high error on new data; underfitting means high error everywhere.
  • Split data into train, validation and test sets; tune on validation and touch the test set once.
  • Bias (too rigid) and variance (too flexible) trade off as model capacity grows.
  • Learning curves of training and validation loss show which problem you have and when overfitting begins.
  • L2 weight decay shrinks all weights (w -= lr * (g + wd * w)); L1 produces sparse weights.
  • Early stopping halts when validation loss stops improving and restores the best weights.

Next: Dropout and Data Augmentation.