Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 73 ยท Training in Practice

Dropout and Data Augmentation

Implement inverted dropout in NumPy, understand train versus eval mode and the ensemble view, then augment arrays with flips, shifts and noise and see when each helps.

Weight decay and early stopping, from the previous lesson, restrain a network after the fact. Two more regularizers go straight to the cause of overfitting. Dropout prevents units from relying on each other by randomly switching them off during training. Data augmentation gives the network more varied training data by transforming the examples you already have.

Both tricks are cheap, widely used, and easy to get subtly wrong: dropout must behave differently while training and while predicting, and augmentation must only change examples in ways that keep the label true. This lesson builds both in NumPy and measures what they do.

Two ways to fight overfitting

Dropout Randomly silence units while training.
Augmentation Create new, label-preserving variations of your data.

One changes the model's behavior; the other changes the data.

1 / 2

Dropout

During every training step, dropout picks each hidden unit independently and, with probability p, sets its output to zero. A different random subset is dropped on every step. The effect is that no unit can count on a particular neighbor being present, so units cannot form fragile co-dependencies, a behavior called co-adaptation that is a major source of overfitting. Each unit has to be useful on its own and in many different company.

Inverted dropout

There is a bookkeeping problem. If a layer's outputs are zeroed with probability p, their sum drops by a factor 1 - p, so the next layer would receive smaller inputs in training than at prediction time. The standard fix is inverted dropout: scale the surviving units up by 1 / (1 - p) during training. Then the expected value of every unit is unchanged, and at prediction time you do nothing at all: dropout is simply switched off.

Python
def dropout(a, rate, training, rng):
    if not training or rate == 0.0:
        return a
    keep = 1.0 - rate
    mask = (rng.random(a.shape) < keep) / keep   # 0 or 1/keep
    return a * mask

Run the next example to see the mask, the scaling, and the fact that the average is preserved.

Python Inverted dropout and its expected value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each training pass zeroes about half of the numbers and doubles the survivors. Averaged over 20,000 passes, the result lands back on [1, 2, ..., 8] (up to small sampling error), while the version without scaling averages to about half of that. That is why training and prediction agree on average.

Train mode versus eval mode

Dropout is the first layer you meet that behaves differently depending on what you are doing, which is why frameworks have model.train() and model.eval(). During prediction, validation or testing you must switch dropout off, otherwise your predictions would be random. Forgetting to do so is a classic bug: the accuracy looks mysteriously low and changes from run to run. (Batch normalization, in Batch Norm and Layer Norm, is the other layer with two modes.)

Only the training step applies the mask, and only the forward pass of training: the backward pass must reuse the same mask, multiplying the gradient of a dropped unit by zero, since a unit that did not contribute to the output cannot be blamed for the error. In code, the gradient flowing back through the dropout layer is grad * mask.

Dropout as an ensemble

Each training step trains a different "thinned" sub-network that shares weights with all the others. A network with 64 hidden units has 2^64 possible sub-networks. Training with dropout is like training a huge ensemble, and prediction with all units on and the scaling already applied approximates averaging all of their predictions. We can check this claim directly with Monte Carlo dropout: keep dropout switched on at prediction time, average many random predictions, and compare with the normal switched-off prediction.

The next example trains the small overfitting-prone network from the previous lesson (64 hidden tanh units, 60 noisy training points) three times: without dropout, with dropout 0.3 and with dropout 0.5. Dropout is applied to the hidden layer, and the backward pass reuses the mask.

Python Dropout on a network that likes to overfit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without dropout the network reaches a training accuracy of 1.000 and a validation loss of 1.477 and a test accuracy of 0.789: it memorized the 60 points. With dropout the validation curves stay low, and the test accuracy rises to 0.855 (rate 0.3) and 0.853 (rate 0.5), at the price of a worse fit to the training data (training accuracy 0.867 and 0.833). That "price" is what you want: the model has stopped memorizing the noise.

The ensemble check is the neat part. Single random sub-networks score only between 0.78 and 0.805, since each is a damaged network. Their average of 200 predictions scores 0.850, and the normal evaluation mode, which uses all units with no randomness at all, scores 0.853. One cheap forward pass gives you what averaging hundreds of sub-networks would.

Typical settings: p = 0.1 to 0.3 in transformers and convolutional layers, up to 0.5 in large fully connected layers. Too high underfits. Dropout is usually not needed when you have a very large dataset or heavy augmentation. If you combine it with batch norm, test the combination, because they can interfere.

Dropout relatives: DropConnect and spatial dropout

Plain dropout zeroes activations. Two variants change what is dropped.

  • DropConnect zeroes individual weights instead. Each output unit sees a random subset of its incoming connections, which is a finer-grained noise than removing whole units. It regularizes dense layers well but costs more to implement efficiently, so ordinary dropout is far more common.
  • Spatial dropout (nn.Dropout2d / nn.Dropout3d, Keras SpatialDropout2D) zeroes entire channels of a feature map. In a convolutional layer, neighboring pixels in one channel are strongly correlated, so dropping single pixels barely hides anything: the missing value can be guessed from its neighbors and the noise is too weak to regularize. Dropping the whole channel forces the network not to depend on any single feature map.

Both keep the same rescaling by 1/(1 โˆ’ p) and both are switched off at inference.

Python Dropout, DropConnect and spatial dropout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Data augmentation

The surest way to cut overfitting is more data. When you cannot collect more, you can manufacture plausible variations of what you have. A picture of a cat is still a picture of a cat if it is shifted a few pixels, mirrored, slightly rotated, a bit brighter, or a little noisy. Applying such label-preserving transformations randomly while training shows the network a different version of each example every epoch.

The golden rule: augment only in ways that keep the label true and that reflect the variation your model will meet in real use. Common transformations by data type:

Data Typical augmentations Be careful
Photos Flips, small shifts, crops, rotations, color jitter, noise Vertical flips of street scenes; flips change text
Digits and text images Small shifts, small rotations, noise, slight scaling Flips turn a 2 into something that is not a 2
Audio Time shift, added background noise, speed change Large pitch changes can alter meaning
Tabular and sensor data Noise, jitter, scaling Keep values physically valid
Text Synonym swap, back-translation, word dropout Easy to change the meaning

Augmentation is applied only to the training data, fresh each time a batch is built, and never to the validation or test data (unless you deliberately want test-time augmentation).

Augmenting arrays

On images stored as NumPy arrays, the transformations are array operations. A horizontal flip reverses the last axis: img[:, ::-1]. A shift moves the array and fills the gap with zeros. Noise is just img + rng.normal(...), clipped back to the valid range.

Let us try them on the 8 by 8 digit images that ship with scikit-learn. The first example only draws the transformations, so you can see what each does to a handwritten 7, including a flip that makes the picture something a reader would not call a 7.

Python Augmenting one digit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Shifting by one pixel on an 8 by 8 image is a big move (one eighth of the width): the shift changes 35 of the 64 pixels. These digits are augmented much more aggressively than a 28 by 28 photo would be for the same shift.

Does it help? An honest experiment

Now measure. We train a plain logistic-regression classifier on just 150 digit images, then test on 1,197 held-out digits in two situations: the digits as they are (aligned), and the same digits all moved one pixel to the right. We compare three training sets: the 150 originals, the originals plus four noisy copies of each, and the originals plus four randomly shifted-and-noisy copies.

Python Augmentation helps only when the variation is real

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is more nuanced than "augmentation always helps". Look at the columns:

  • Noisy copies change almost nothing: a linear classifier on 64 pixels gains little from jitter.
  • Shifted copies lower the accuracy on the aligned test set (0.921 to 0.856), because the classifier now has to accept many positions and spends capacity on that. But on the moved test set the accuracy jumps from 0.358 to 0.647. The augmented model is robust to the variation it was shown.

So augmentation is a trade: it buys robustness to the variations you simulate, and it can cost a little accuracy on perfectly clean data, especially for a weak model. If your real data is always centered, shifts waste effort; if cameras and users are sloppy, shifts are essential. A bigger model (like the convolutional networks in Training a CNN From Scratch) has the capacity to learn the variations without losing accuracy on clean data, which is why augmentation is nearly universal in image training.

Do not apply random augmentation to the validation or test sets. You would be measuring the model on data that changes with the random seed, and your numbers would no longer be comparable between runs.

When to use what

Situation Reach for
Small dataset, images or audio Augmentation first; it multiplies your data
Big fully connected layers overfit Dropout 0.2 to 0.5
Transformer or modern CNN Small dropout (0.1), weight decay, augmentation
Gap between train and validation is small Neither; you may be underfitting
Noisy labels Stronger regularization, early stopping, more data

Check the learning curves from Overfitting, Underfitting and Regularization before and after to confirm that a technique actually closed the gap.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does inverted dropout multiply the surviving units by 1 / (1 - p) during training?

    Zeroing a fraction p shrinks the sum by (1 - p). Scaling the survivors up compensates, so no change is needed at prediction time.
  2. 2. What should happen to dropout when you evaluate the model on the test set?

    Evaluation mode disables dropout so that predictions are deterministic.
  3. 3. During backpropagation through a dropout layer, what happens to the gradient?

    A dropped unit did not affect the output, so its gradient is zero; surviving units carry the same scaling as in the forward pass.
  4. 4. Which augmentation is a bad idea for classifying handwritten digits?

    A mirrored digit is no longer a valid example of its class. Augmentations must preserve the label.
  5. 5. Where should random augmentation be applied?

    Validation and test sets should stay fixed so that scores are comparable between runs.

Summary

  • Dropout zeroes each hidden unit with probability p during training and scales the survivors by 1 / (1 - p) (inverted dropout), so nothing changes at prediction time.
  • Switch dropout off for evaluation; the backward pass reuses the forward mask.
  • Dropout works like training an ensemble of thinned sub-networks; the normal evaluation pass approximates their average.
  • In the experiment, dropout lifted test accuracy from 0.789 to about 0.85 on a small, noisy problem.
  • Data augmentation applies label-preserving transformations (shifts, flips, noise, and more) to training data only.
  • Augment in ways that match the variation you expect, and expect a small cost on perfectly clean data.

Next: Batch Norm and Layer Norm.