Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 45 ยท Training

Overfitting and Regularization

Understand bias, variance and overfitting with live examples, then fix them with weight decay, dropout, data augmentation and early stopping in PyTorch.

A model that scores perfectly on its training data and badly on new data has not learned the task: it has memorized the examples. This is overfitting, and it is the most common reason a model that looks great in the training log disappoints in the real world. Its opposite, underfitting, is a model too simple or too briefly trained to capture the pattern at all. Good machine learning is mostly the art of landing between the two.

This lesson makes the problem visible with small, fully runnable NumPy experiments (polynomials and a hand-written neural network) and then shows the PyTorch tools that control it: weight decay, dropout, data augmentation and early stopping. PyTorch itself cannot run in the browser, so its code is read-only; the live examples are stand-ins that use the same ideas and real arithmetic. It follows The Complete Training Loop, where you saw the training and validation losses drift apart.

Three possible outcomes

Underfit Too simple. Both errors are high.
Good fit Both errors are low and close.
Overfit Train error tiny, validation error large.

Always compare training and validation numbers together.

1 / 2

Bias, variance and the noise floor

Every dataset contains noise: measurement error, randomness, things the inputs do not tell you. No model can predict noise, so there is a floor under the achievable error. The rest of the error splits into two parts.

  • Bias is error from a model that is too rigid. A straight line fitted to a curve is wrong however much data you give it. High bias means underfitting: training error and validation error are both high and close together.
  • Variance is error from a model that is too flexible and reacts to the noise in its particular training sample. A different sample would produce a very different model. High variance means overfitting: training error is tiny, validation error is large, and there is a big gap.

Model flexibility is the dial between them. The easiest place to see the dial is polynomial regression, where flexibility is the degree. The example fits the same 20 noisy points of a sine wave with degrees 1 to 15 and scores each fit on 300 fresh points the model never saw.

Python Polynomial degree: underfit, fit, overfit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table. A straight line (degree 1) is too simple: both errors are around 0.25 and the plot shows a line cutting through a wave. Degrees 3 to 9 reach a validation error of about 0.077, close to the noise floor of 0.0625 (the noise standard deviation was 0.25, so its variance is 0.0625). Beyond that the training error keeps shrinking, down to 0.0204 at degree 15, yet the validation error shoots up to 2.3537. The degree-15 curve threads through every training point and swings wildly between them. Falling training error is not evidence of a good model.

Learning curves

A learning curve plots training and validation error against the amount of training data. Its shape diagnoses what is wrong:

  • If both curves are high and converge on each other, the model has high bias. More data will not help; you need a more expressive model, better features or longer training.
  • If training error is far below validation error, the model has high variance. More data, a simpler model or regularization will help.

The next stand-in computes both curves for a rigid degree-1 model and a flexible degree-9 model by retraining on many random datasets of each size (taking the median over 40 trials).

Python Learning curves: high bias versus high variance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For degree 1, the training error rises from 0.221 to 0.259 while validation falls from 0.300 to 0.269: the two meet well above the noise floor of 0.0625, which is the signature of bias. Degree 9 behaves very differently. With only 14 examples the validation error is a staggering 36.408 while the training error is a tiny 0.015, a classic overfit. With 300 examples the model has enough data to pin down its coefficients, and the two errors become 0.060 and 0.064, almost at the floor. For flexible models the cure for overfitting is often simply more data.

In deep learning you plot the same story against epochs instead: training loss keeps falling, validation loss falls and then turns upward. The turning point is where overfitting begins.

Weight decay (L2 regularization)

Overfit models often have huge weights, as the degree-15 fit shows, because large opposing coefficients let a curve wiggle through every point. Weight decay adds a penalty lambda * sum(w**2) to the objective, or equivalently shrinks each weight a little every step, so the optimizer prefers small weights and smoother functions.

The ridge-regression stand-in below fits the degree-15 polynomial again, this time with increasing lambda:

Python Weight decay as ridge regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With lambda = 0 the largest coefficient is about 39,634 and the validation error 2.3537. Even a tiny penalty of 0.0001 brings the largest weight to 7.38 and validation error to 0.0807, and lambda = 0.01 reaches 0.0793, nearly as good as the best polynomial. Push lambda to 10 and the model is squashed into underfitting (0.3732). The penalty strength is a hyperparameter you tune on the validation set.

In PyTorch you set it on the optimizer:

Python
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
# or, with plain SGD (classic L2):
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)

Prefer AdamW over Adam(weight_decay=...) for the decoupled behavior explained in Optimizers: SGD, Momentum, Adam, and consider leaving biases and normalization parameters out of the decay.

Dropout

Dropout randomly zeroes a fraction of activations on each training step (see Initialization, Normalization and Dropout). Because any unit may vanish, the network cannot rely on a fragile combination of specific units, and it behaves like an average of many thinner networks.

Python
model = nn.Sequential(
    nn.Linear(784, 512), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(512, 512), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(512, 10),
)

Remember model.eval() when validating, or dropout stays on and your validation numbers get noisy. Typical rates are 0.1 to 0.5: higher for big fully connected layers, lower (or none) for convolutions and Transformers.

Data augmentation

The surest way to reduce overfitting is more data. When you cannot collect more, you can augment: apply random, label-preserving changes to each training example so the model sees a slightly different version every epoch. For images that means flips, small shifts, rotations, crops, color jitter and noise; for audio, time stretching and added noise; for text, careful synonym or word dropout.

In PyTorch, augmentation lives in the Dataset's transform, so it runs fresh on every access (the vision lessons use torchvision.transforms for this):

Python
from torchvision import transforms

train_tf = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
])
val_tf = transforms.ToTensor()        # never augment the validation data

The two key rules are: augment only the training data, and only apply changes that do not alter the label. The tiny-image demo makes the second rule concrete.

Python Data augmentation on a tiny image

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One labeled picture became five training pictures: the original, a shifted copy, a vertically moved copy, a mirrored copy and a noisy copy. Shifts and small noise keep a "7" a 7. The mirrored image, however, is no longer a valid digit, so a horizontal flip would be a mistake for digit recognition even though it is routine for photographs of animals. Always ask whether the label survives the transformation.

Early stopping

You already built early stopping in the previous lesson: track the best validation loss, stop after patience epochs without improvement, and restore the best weights. It regularizes by limiting how long the model has to memorize the training set. It costs nothing, so it is a good default in almost every project.

Putting them together

The final experiment trains a hand-written neural network (two hidden layers of 64 units, ReLU, Adam, all backpropagation done by hand) on only 30 noisy points of a sine wave. The noise has standard deviation 0.3, so the validation error cannot go below about 0.09. The code compares the unregularized network with each remedy and with all of them together.

Python Regularizing a small neural network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unregularized network drives its training error to 0.0400 but reaches only 0.1186 on validation, well above the noise floor. Weight decay (0.1052) and early stopping (0.1064, stopping after only 291 epochs) clearly help. Look at what else the table says, honestly: dropout at 0.2 made this particular small network a little worse (0.1306), and input jitter of 0.05 did essentially nothing (0.1193). Regularizers are not magic. Each has a strength to tune, and dropout in particular needs wide layers and plenty of training to pay off; it can hurt a small model on tiny data. Use the validation set to decide, one change at a time, and do not add every technique by habit.

A practical order of attack

  1. Make sure the model can overfit a small batch (if not, there is a bug or it is too small).
  2. Watch the gap between training and validation loss.
  3. If the gap is large: get more data or augment, add weight decay, try dropout, shrink the model, and use early stopping.
  4. If both losses are high: train longer, increase capacity, raise the learning rate or improve the features.
  5. Evaluate once on the untouched test set when you are done.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Training loss is 0.02 and validation loss is 1.9. What is the most likely diagnosis?

    A tiny training error with a much larger validation error is the signature of overfitting (high variance).
  2. 2. A learning curve shows training and validation error both high and almost equal as data grows. What helps most?

    Both errors being high and converged means high bias. The model cannot capture the pattern, so give it more capacity or better inputs; regularization would make it even more rigid.
  3. 3. Which statement about data augmentation is correct?

    Augmentation creates varied training examples. The validation data must stay fixed to give a comparable, honest measurement, and the changes must preserve the label.
  4. 4. What does weight decay do?

    Weight decay adds an L2 penalty (or shrinks weights each step), which discourages the huge coefficients typical of overfit models. Random zeroing is dropout, and lowering the learning rate is a schedule.
  5. 5. In the final experiment, dropout made a small network slightly worse. What is the right conclusion?

    Every regularizer has a strength and a situation where it helps. Small models on tiny datasets can be hurt by too much noise. Measure on validation data and adjust.

Summary

  • Overfitting means low training error and high validation error; underfitting means both are high.
  • Bias comes from rigid models, variance from flexible ones; there is also an unavoidable noise floor.
  • Learning curves tell the two apart: converged-and-high means bias, a big gap means variance.
  • Weight decay (AdamW(weight_decay=...)), dropout, data augmentation and early stopping all reduce overfitting; each has settings to tune against the validation set.
  • Augment only the training data, and only with label-preserving transformations.
  • Compare one change at a time, and keep a test set untouched until the end.

Next, move from tabular data to images with Convolutions and CNNs.