Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 40 ยท Training and Evaluation

Overfitting and Callbacks

Detect and fight overfitting with dropout, early stopping, checkpoints and learning-rate callbacks in Keras, with runnable NumPy stand-ins for each idea.

A model that scores brilliantly on its training data and poorly on new data has overfit: it memorized examples, noise included, instead of learning the pattern. Overfitting is the central problem of practical machine learning, because big networks have more than enough capacity to memorize. This lesson shows how to see it, the main remedies (more data, simpler models, regularization, dropout, early stopping), and the Keras callbacks that automate them. Every idea has a runnable NumPy stand-in.

Seeing overfitting

Fit polynomials of increasing degree to 15 noisy points from a smooth curve. A low degree underfits, a very high degree passes through every training point and wiggles wildly in between. Compare the error on the training points with the error on fresh points.

Python Stand-in: model capacity and overfitting (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training error keeps falling as the degree rises, but validation error falls and then climbs steeply: the signature of overfitting. In a neural network, the equivalent knobs are the number of layers and units, and the number of epochs.

flowchart LR
    A["Underfit<br/>train high, val high"] --> B["Good fit<br/>train low, val low"] --> C["Overfit<br/>train very low, val rising"]

As capacity or training time grows, a model passes through all three stages.

The remedies

Fighting overfitting

More data The best cure. Augment if you cannot collect.
Smaller model Fewer layers or units.
Weight penalties L1 and L2 regularizers.
Dropout Randomly silence units while training.

Combine several; check validation loss after each change.

1 / 2

Dropout

Dropout randomly sets a fraction of a layer's outputs to zero on each training step. Units cannot rely on specific partners being present, so the network learns redundant, robust features, like a team that trains with random members absent. At prediction time dropout is off and every unit participates.

To keep the expected activation size the same between training and prediction, surviving outputs are scaled up by 1 / (1 - rate) during training ("inverted dropout"). Keras does this automatically:

Python
model = keras.Sequential([
    keras.Input(shape=(784,)),
    layers.Dense(256, activation="relu"),
    layers.Dropout(0.3),            # drop 30% of units during training
    layers.Dense(128, activation="relu"),
    layers.Dropout(0.3),
    layers.Dense(10),
])

Dropout has no weights and is only active when training=True, which fit sets and predict/evaluate do not. Typical rates are 0.1 to 0.5; the layer is placed after the activation of the layer it regularizes. Too much dropout underfits.

Python Stand-in: inverted dropout (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two training calls drop different units; the averages stay near 1, which is exactly why no rescaling is needed at inference.

Early stopping

Validation loss typically falls, bottoms out, then rises as the model begins to overfit. Early stopping watches it and halts when it has not improved for patience epochs, then restores the best weights. It is the cheapest and most reliable regularizer; the only hyperparameter is patience.

Python
early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=5,                  # tolerate 5 epochs without improvement
    min_delta=1e-4,              # what counts as improvement
    restore_best_weights=True,   # go back to the best epoch
)

model.fit(x_train, y_train, epochs=200, validation_split=0.2, callbacks=[early])

With early stopping you set epochs generously high and let the callback decide.

Here is the logic on an overfitting problem: a 12-degree polynomial trained by gradient descent. Training error keeps shrinking while validation error turns upward, and the stopper keeps the best epoch.

Python Stand-in: an early-stopping loop with patience (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The restored weights give the best validation score seen, which is also the version you should evaluate and save.

Other callbacks

Callbacks are objects that Keras calls at set moments: start/end of training, each epoch, each batch. Pass a list to fit.

Python
callbacks = [
    keras.callbacks.EarlyStopping(monitor="val_loss", patience=5,
                                  restore_best_weights=True),

    keras.callbacks.ModelCheckpoint("best.keras", monitor="val_loss",
                                    save_best_only=True),

    keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5,
                                      patience=2, min_lr=1e-6),

    keras.callbacks.CSVLogger("training_log.csv"),

    keras.callbacks.TensorBoard(log_dir="logs"),
]
  • ModelCheckpoint saves the model to disk (here the native .keras format) after epochs, optionally only when the monitored value improves. It protects against crashes and lets you resume.
  • ReduceLROnPlateau multiplies the learning rate by factor when the monitored value stalls for patience epochs, so training squeezes out extra gains.
  • LearningRateScheduler applies a function epoch -> learning rate.
  • TensorBoard logs curves for the TensorBoard dashboard.
  • CSVLogger writes metrics to a CSV file.
  • TerminateOnNaN stops training if the loss becomes nan.

Write your own callback

Subclass keras.callbacks.Callback and override the hooks you need. Inside, self.model is the model and logs is a dictionary with the epoch's metrics.

Python
class PrintBest(keras.callbacks.Callback):
    def on_train_begin(self, logs=None):
        self.best = float("inf")

    def on_epoch_end(self, epoch, logs=None):
        val = logs.get("val_loss")
        if val is not None and val < self.best:
            self.best = val
            print(f"epoch {epoch + 1}: new best val_loss {val:.4f}")

model.fit(x, y, epochs=30, validation_split=0.2, callbacks=[PrintBest()])

Other hooks: on_epoch_begin, on_train_batch_end, on_test_end, on_predict_begin and so on. Setting self.model.stop_training = True inside a hook ends training, which is exactly how EarlyStopping works.

A callback runner (stand-in)

To see that callbacks are just function calls at the right moments, here is a tiny training loop that supports them, including an early-stopping callback that sets a stop_training flag.

Python Stand-in: a training loop with callbacks (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A practical recipe

  1. Train a baseline and look at both curves.
  2. If training and validation loss are both high, make the model bigger or train longer (underfitting).
  3. If validation loss rises while training loss falls: add data or augmentation, add Dropout(0.2-0.5) and a small L2, shrink the model, and always use EarlyStopping(restore_best_weights=True).
  4. Add ModelCheckpoint(save_best_only=True) for long runs.
  5. Evaluate once on the test set at the end.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which pattern in the learning curves signals overfitting?

    The model keeps improving on training data while generalizing worse.
  2. 2. Why does inverted dropout scale the surviving activations by 1 / (1 - rate)?

    Dropping units lowers the average activation, so survivors are scaled up; inference then needs no change.
  3. 3. What does restore_best_weights=True do in EarlyStopping?

    After stopping, the model reverts to the best epoch rather than keeping the last, possibly overfit, weights.
  4. 4. When is Dropout active in Keras?

    Dropout acts when training=True, as in fit, and is off in evaluate and predict.
  5. 5. Which callback saves the best model to disk?

    ModelCheckpoint writes the model when the monitored metric improves.

Summary

  • Overfitting means low training error but poor generalization; learning curves reveal it.
  • Cures: more data, smaller models, L1/L2 penalties, dropout and early stopping.
  • Dropout silences random units during training only; EarlyStopping(restore_best_weights=True) is the cheapest regularizer.
  • Callbacks (ModelCheckpoint, ReduceLROnPlateau, TensorBoard, custom subclasses) hook into training; setting stop_training ends it.
  • Use the test set once, after all tuning is done.

Next: How Convolution Works