Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 73 ยท Training in Practice

Debugging Training Runs

Debug a network that will not learn: overfit one batch, fix NaNs, tame exploding gradients with clipping, find dead units, and read loss curves.

Neural networks fail silently. A bug in ordinary code usually crashes or gives an obviously wrong answer; a bug in a training loop usually gives a network that trains, prints a loss, and is quietly mediocre. The loss may go down slowly, stay flat, or turn into nan, and the cause could be the data, the model, the loss, the gradients, the optimizer or just one wrong sign.

The way out is to debug in a fixed order, from the simplest checks to the most subtle, and to make the model prove itself on tiny cases before trusting it on big ones. This lesson gives you that routine, with runnable tools: the overfit-one-batch test, a learning-rate sweep, NaN hunting, gradient clipping, a per-layer health report, and a guide to reading loss curves.

Debug in this order

Tiny first Overfit one small batch before anything else.
Watch the loss Check the starting value and the trend.
Inspect layers Activations and gradients, layer by layer.
Then tune Only now touch learning rate and regularization.

Change one thing at a time, and write down what you tried.

1 / 2

Know the starting loss

Before training, compute the loss of the untrained model. For a classifier with K classes and sensible initialization, the predictions are close to uniform, so the cross-entropy loss should be about ln(K): 0.693 for two classes, 2.303 for ten. If your first loss is 40 or 0.001, something is off before any learning happens: a wrong loss function, unnormalized inputs, a huge initialization, or labels in the wrong format. This one-line check catches a surprising number of bugs. See Loss Functions and Weight Initialization for why.

Overfit one batch

The single most useful test: take 8 to 32 training examples and train on only those for a few hundred steps. A correct network with enough capacity must be able to memorize them, driving the loss close to zero. If it cannot, the bug is in the code or the optimization, not in the dataset or the generalization.

The next example defines a small ReLU network with softmax cross-entropy and runs the test three ways: correct, with a learning rate far too small, and with a sign error in the update (adding the gradient instead of subtracting it).

Python The overfit-one-batch test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The healthy run falls from about 1.5 (a little above ln(3) = 1.099, because random weights are never exactly uniform) to roughly 0.1: the network has memorized 16 points. The tiny learning rate has barely moved after 1,000 steps, and the sign bug sends the weights the wrong way until the loss overflows to nan (the plot shows the curve ending where it does). Real bugs are subtler, but the test works the same way: it tells you in seconds whether the whole pipeline (forward pass, loss, backward pass, update) can learn at all. If the backward pass is the suspect, verify it with Gradient Checking.

Sweeping the learning rate and hunting NaNs

The learning rate is the setting most likely to be wrong. Too small and the loss crawls; too large and it bounces or explodes. A quick sweep over several orders of magnitude shows where the healthy range is. A loss that becomes nan or inf almost always means numbers overflowed.

Common sources of nan:

  • A huge learning rate: weights grow until exp overflows.
  • log(0): a probability that underflows to exactly 0, then log gives -inf. Use a numerically stable log-softmax, or add a tiny epsilon.
  • Division by zero: a normalization with no epsilon, or a standard deviation of zero.
  • Bad data: a nan or inf already in the inputs or labels. Check with np.isfinite(X).all().
Python Learning-rate sweep and a naive cross-entropy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sweep shows a working band (0.01 to 1 here), a crawl at 0.0001, and a loss that ends higher than it started at 30 and 300. The last lines show the classic overflow: exp(2000) is inf, inf / inf is nan, and the loss is inf, while subtracting the maximum logit (the stable form) stays finite. A good first guess for the learning rate is the largest value that does not make the loss bounce; schedules are covered in Learning Rate Schedules.

Exploding gradients and clipping

If the loss suddenly spikes or becomes nan partway through training, the gradients probably exploded: a few unlucky batches or an unstable layer produce a gradient norm thousands of times larger than usual, and one update throws the weights far away. The mirror problem, vanishing gradients, makes the early layers learn nothing.

The standard defense is gradient clipping: before the update, compute the global norm of all gradients together, and if it exceeds a threshold, scale every gradient by threshold / norm. This keeps the direction but bounds the step. It is routine in recurrent networks and transformers.

A per-layer health report

When things go wrong, look inside. For each layer, three numbers tell most of the story:

  • the fraction of dead ReLU units, meaning units that output zero for every example in the batch (they pass no gradient and may never recover);
  • the mean and standard deviation of activations: shrinking to zero or growing without bound across depth signals a bad initialization (see Weight Initialization);
  • the gradient norm per layer: wildly different values between layers point at vanishing or exploding gradients.
Python Health report and gradient clipping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With too-large weights, the activation scale and gradient norms grow layer after layer, and clipping scales the global gradient norm (about 660 here) down to exactly 5, bounding the update step. The negative-bias run shows another failure: most units are dead from the start, so the network has almost nothing to learn with. Causes of dead units in practice are a too-high learning rate, a bad initialization or an input that was never normalized. Leaky ReLU variants and lower learning rates help; see Activation Functions.

Reading loss curves

Always plot training and validation loss together. Shape tells you what to do next:

What you see Likely cause Try
Flat from the first step Learning rate far too small, dead units, broken update, frozen parameters Overfit-one-batch test, increase learning rate, check the health report
Falls, then spikes upward Learning rate too high, an exploding batch Lower the rate, add clipping, add a warmup
Very noisy, no clear trend Batch too small or step too big Larger batch, smaller rate, average over a window
Train falls, validation rises Overfitting More data, regularization, early stopping
Both high and flat Underfitting Bigger model, train longer, better features
Validation better than train Dropout active in training only, or train loss averaged over the epoch while the weights improve Normal; compare with both in eval mode

A sanity checklist

Run through this before blaming the architecture:

  1. Inputs are finite, scaled to a sensible range, and labels are correct (look at a few examples by eye).
  2. The first loss is about ln(K) for K classes.
  3. The model can overfit one small batch to near-zero loss.
  4. The backward pass passes a gradient check on a tiny network.
  5. Train and eval modes are set correctly (dropout and batch norm behave differently).
  6. Data is shuffled, and training and validation come from the same distribution without overlap.
  7. Only then: sweep the learning rate, then adjust model size and regularization, changing one thing at a time and fixing a random seed so runs are comparable.

Keep a log of every experiment: what you changed and what happened. A run you cannot reproduce is a run you cannot learn from.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A 10-class classifier with a correct loss and sensible initialization should start with a loss near what value?

    Untrained predictions are close to uniform, so each true class gets probability near 1/10 and the cross-entropy is about -ln(1/10) = ln(10).
  2. 2. What does the overfit-one-batch test tell you?

    If a model with enough capacity cannot memorize a handful of examples, the problem is a bug or optimization failure, not generalization.
  3. 3. How does gradient clipping by global norm work?

    The direction of the update is kept and only its length is bounded, so one huge gradient cannot throw the weights far away.
  4. 4. Training loss keeps falling while validation loss starts rising. What is the most likely diagnosis?

    The model is fitting the training set at the expense of new data. Regularization, more data or early stopping address this.
  5. 5. Which is a common cause of a sudden nan loss?

    exp of a very large number is inf, and inf / inf is nan. Subtracting the maximum logit and keeping the learning rate sane avoids it.

Summary

  • Compute the loss before training: for K balanced classes it should be close to ln(K).
  • Make the model overfit one small batch; failing that test points to a bug in the pipeline, not in the data.
  • Sweep the learning rate over orders of magnitude and watch for nan from overflow, log(0) and division by zero.
  • Clip gradients by global norm to survive spikes; inspect dead units, activation scales and per-layer gradient norms with a health report.
  • Read train and validation curves together to tell overfitting, underfitting and instability apart.
  • Change one thing at a time, fix seeds, and keep an experiment log.

Next: Building a Mini Deep-Learning Framework.