Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 40 ยท Projects and Practice

Practice and Troubleshooting

A debugging playbook for TensorFlow and Keras: overfit one batch, shape errors, NaN losses, learning-rate checks and learning curves, plus exercises and interview questions.

Neural networks fail silently. A model with a bug often still trains and produces numbers; they are just bad. This final lesson is a playbook for finding out what is wrong, followed by practice exercises and the questions interviewers like to ask. Most of the checks work on any model, and the runnable demos use NumPy so you can try them here.

TensorFlow is not available in the browser runner. Keras snippets are read-only. The NumPy demos reproduce the symptoms and the checks.

The debugging order

When training goes wrong

Check the data Shapes, ranges, labels
Overfit one batch Loss must reach near zero
Watch the curves Train and validation
Tune last Learning rate first

Fix bugs before tuning hyperparameters.

1 / 2

1. Check the data first

Most failures are data problems. Before training, print shapes, dtypes, value ranges and a few labels, and look at real examples.

Python
for x, y in train_ds.take(1):
    print(x.shape, x.dtype, float(x.numpy().min()), float(x.numpy().max()))
    print(y.shape, y.dtype, y.numpy()[:10])
Python A quick data sanity check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Typical data bugs: unscaled pixels, NaNs, labels off by one (1..10 instead of 0..9), training and test preprocessed differently, and labels shuffled apart from their inputs.

2. Overfit one small batch

Take 16 examples and train for 100 or so steps. A healthy model drives the loss close to zero (it memorizes them). If it cannot, there is a bug in the model, loss, labels or learning rate, and no amount of tuning will help.

Python
x_small, y_small = next(iter(train_ds.take(1)))
x_small, y_small = x_small[:16], y_small[:16]
model.fit(x_small, y_small, epochs=200, verbose=0)
print(model.evaluate(x_small, y_small))      # expect loss near 0, accuracy 1.0
Python Overfitting one batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls to nearly zero and accuracy reaches 1.0: the pipeline (forward, loss, gradients, update) works.

3. Shape errors

The most common Keras error messages are shape mismatches. Read them literally.

Message fragment Usual cause
expected shape=(None, 28, 28, 1), found shape=(32, 28, 28) Missing channel axis; use x[..., None]
Incompatible shapes: [32,10] vs [32] Labels one-hot but loss is sparse (or vice versa)
Input 0 of layer "dense" is incompatible Flatten missing after convolutions
logits and labels must have the same first dimension Batch sizes disagree between inputs and labels
Output shape (batch, 1) vs target (batch,) Broadcasting to (batch, batch) hides a bug; reshape explicitly

Use model.summary() and print x.shape at every stage. A quick manual check of what broadcasting does:

Python The silent broadcasting bug

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The wrong version compares every prediction with every target and produces a plausible-looking number. Keras losses guard against some of this, but always match shapes explicitly.

4. Loss is NaN or infinite

Causes in rough order of frequency:

  1. A learning rate too high (try 10 times smaller).
  2. NaNs or infinities in the input data.
  3. log(0) or division by zero in a custom loss; add an epsilon.
  4. Missing normalization with large inputs.
  5. float16 overflow in mixed precision without loss scaling.
  6. Exploding gradients in recurrent nets; use clipnorm=1.0.
Python
keras.callbacks.TerminateOnNaN()          # stop immediately when the loss is NaN
tf.debugging.enable_check_numerics()      # find the exact op producing NaN/inf
Python A learning rate that is too high

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With lr=0.9 the weight flips sign each step but shrinks. With lr=1.1 the overshoot exceeds the starting distance, so the magnitude grows every step and explodes.

5. Read the learning curves

Pattern Diagnosis Try
Train and val loss both high and flat Underfitting, or bug Bigger model, train longer, check one-batch test
Train loss falls, val loss rises Overfitting More data or augmentation, dropout, weight decay, smaller model, early stopping
Val better than train early Dropout/augmentation active on train only Usually fine
Loss jumps around Learning rate too high, tiny batches Lower the rate, bigger batch
Loss decreases very slowly Learning rate too low, bad init, saturated activations Raise rate, use ReLU and normalization
Python Detect overfitting from curves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

6. Other common bugs

  • Forgetting training=False for pretrained BatchNorm layers in a frozen base.
  • Evaluating with augmentation or dropout still enabled in a hand-written loop.
  • Data leakage: normalizing before the split, or duplicates across splits.
  • Changing trainable without recompiling.
  • Randomness: set keras.utils.set_random_seed(42) to make runs comparable.
  • Comparing models trained with different input pipelines.

Practice exercises

  1. Take the CNN from Building a CNN and compute its parameter count by hand for a 32 by 32 RGB input with three conv blocks (16, 32, 64 filters, 3 by 3, "same", pool 2) and a 10-class head using global average pooling. Check with model.summary().
  2. Write the NumPy training loop for logistic regression, then verify your gradients against a finite-difference estimate.
  3. Implement 2D max pooling with reshape and max, and test it against a nested-loop version.
  4. Build a bag-of-words classifier for a text set you write yourself, and list its five most positive and negative words.
  5. Take any model and run the overfit one batch test; deliberately break the labels and see what changes.
  6. Using the attention code from Attention and Transformers, add a causal mask and show the weights are lower triangular.
  7. Quantize a weight matrix to int8 and measure the relative output error for a random input.
Python Starter: finite-difference gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Interview questions

  • What is the difference between Sequential, the functional API and model subclassing?
  • Why use ReLU instead of sigmoid in hidden layers?
  • What does from_logits=True do and why is it more stable?
  • Explain vanishing gradients and two ways to reduce them.
  • How does batch normalization behave differently in training and inference?
  • What is the difference between an epoch, a batch and a step?
  • How does a convolutional layer reduce parameters compared with a dense layer?
  • What is transfer learning and when would you fine-tune?
  • How do you avoid leaking test data information into training?
  • What changes when you add GPUs to training?
  • Why can a Transformer be trained faster than an LSTM on the same data?

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model cannot reach near-zero loss on a batch of 16 training examples. What does this most likely indicate?

    Memorizing 16 examples is easy for any adequate model; failing means something in the pipeline is broken.
  2. 2. The training loss falls but the validation loss rises. This is:

    The model memorizes the training set while generalizing worse.
  3. 3. Which is NOT a usual cause of a NaN loss?

    A validation set is unrelated to numerical failure.
  4. 4. Shape (4,) minus shape (4, 1) in NumPy or TensorFlow produces:

    Broadcasting expands both to (4, 4), which silently corrupts a loss computed this way.

Summary

  • Debug in order: data, overfit one batch, shapes, numerical issues, learning curves, then tuning.
  • Many silent bugs come from broadcasting, label formats, missing channel axes and leakage.
  • NaN losses usually mean a learning rate, data or numerical stability problem.
  • Learning curves tell underfitting, overfitting and an unstable learning rate apart.
  • Practice by rebuilding the components in NumPy, then compare with Keras.

Next: Back to the TensorFlow and Keras course