Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 45 ยท Debugging and Practice

Debugging PyTorch Code

Find PyTorch bugs fast: shape and dtype errors, device mismatches, NaN losses, vanishing gradients, anomaly detection, overfitting one batch and seeds.

Machine learning code has a nasty property: wrong code often runs without any error. A model with a bug in its loss still trains, the loss still goes down a bit, and you only discover weeks later that accuracy is stuck at a level a much simpler model beats. Debugging a neural network therefore means two skills: reading the loud failures (shape, dtype and device errors) and testing for the silent ones (a loss that is wrong, gradients that vanish, data that is not what you think).

PyTorch cannot run on this site, so the PyTorch snippets are read-only. The live editors use NumPy, which has the same shape rules, to practice the diagnostics themselves: checking shapes, finding where a NaN appears, measuring gradients layer by layer, and the single most useful test in deep learning, overfitting one batch. These are stand-ins for the PyTorch tools, built on the same ideas.

Loud bugs

Shape errors Dimensions do not line up.
Dtype errors Float meets integer.
Device errors CPU meets GPU.

PyTorch tells you, read the message.

1 / 2

Reading shape errors

Most errors you will ever see are shape errors. The trick is to read the message as a pair of shapes and ask which one is wrong.

Python
import torch
from torch import nn

model = nn.Linear(10, 3)
x = torch.randn(32, 8)          # 8 features, but the layer expects 10
model(x)
# RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x8 and 10x3)

nn.Linear(10, 3) stores a weight that multiplies inputs whose last dimension is 10. Your data has 8 features. Either the data pipeline or the layer definition is wrong. A good habit is to print shapes while building a model, one line per stage:

Python
def forward(self, x):
    print("input", x.shape)
    x = self.conv(x);  print("after conv", x.shape)
    x = x.flatten(1);  print("after flatten", x.shape)
    return self.fc(x)

or, without editing the model, run one batch through it with hooks or a library such as torchinfo. Quieter shape bugs come from broadcasting (see Tensor Math and Broadcasting): subtracting a (32,) vector from a (32, 1) column gives a (32, 32) matrix instead of an error, and the loss then averages nonsense.

Python
pred = model(x)                      # shape (32, 1)
target = y                           # shape (32,)
loss = ((pred - target) ** 2).mean() # silently broadcasts to (32, 32)!
# fix: target = y.unsqueeze(1), or pred.squeeze(1)

PyTorch's MSELoss warns about exactly this mismatch. Assert shapes at the boundaries of your code (assert pred.shape == target.shape) so such bugs fail loudly. You can practice the habit now:

Python Silent broadcasting and a shape guard

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dtype and device mismatches

Three more loud errors, each with a one-line cause:

Python
# 1. Integer labels where floats are expected (or the reverse)
nn.functional.mse_loss(pred, labels_int)
# RuntimeError: result type Float can't be cast to the desired output type Long

# 2. CrossEntropyLoss wants class indices (Long), not floats
nn.functional.cross_entropy(logits, labels.float())
# RuntimeError: expected scalar type Long but found Float

# 3. Tensors on different devices
model.to("cuda")(torch.randn(4, 10))   # input is on the CPU
# RuntimeError: Expected all tensors to be on the same device ...

For (1) and (2) convert with .float() or .long(), and check tensor.dtype when in doubt (see Dtypes, Precision and NumPy Interop). For (3) move everything with .to(device) (see GPUs, Devices and Mixed Precision). On a GPU, errors can show up at a later line than the one that caused them, because GPU work is asynchronous. Setting the environment variable CUDA_LAUNCH_BLOCKING=1 makes every call wait, so the traceback points at the right place; remove it afterwards, because it slows everything down. An index error inside a GPU run (device-side assert triggered) is most often a label that is out of range for the number of classes. Rerun on the CPU and you get a readable message.

NaN and inf losses

When the loss suddenly prints nan, one of a few things happened: a learning rate that is too high, log(0) or a division by zero, a softmax computed without subtracting the maximum, or an overflow in float16. Once a NaN appears in a weight, every later value is NaN, so you need to find the first one.

Python
# Cheap checks
print(torch.isnan(loss).item(), torch.isinf(loss).item())
assert torch.isfinite(xb).all(), "bad values in the input batch"   # check the data too

# Find which operation produced it
with torch.autograd.set_detect_anomaly(True):
    loss = loss_fn(model(xb), yb)
    loss.backward()
# RuntimeError: Function 'LogBackward0' returned nan values in its 0th output.

set_detect_anomaly(True) records where each operation was created and raises an error at the backward step that first produces NaN, with a traceback pointing at the forward line responsible. It is slow, so switch it on only to hunt a bug. Prevention is better: use BCEWithLogitsLoss and CrossEntropyLoss on raw logits instead of computing log(sigmoid(x)) yourself (see Activation Functions and Loss Functions), add a small epsilon when you divide or take a logarithm, and clip gradients (see Learning Rate Schedules).

The editor shows the classic numerical trap: a naive softmax overflows, a stable one does not, and a loss on a probability of exactly zero is infinite.

Python Where NaN and inf come from

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Vanishing and exploding gradients

A network can also fail quietly because the signal shrinks or grows as it travels through many layers. In the backward pass each layer multiplies the gradient by its local derivative and its weights. If the typical factor is below 1 the gradient shrinks exponentially with depth (vanishing, early layers barely learn); above 1 it blows up (exploding, losses jump to inf). Sigmoid and tanh activations have derivatives below 1 (at most 0.25 for sigmoid), which is why ReLU, careful initialization and normalization became standard (see Initialization, Normalization and Dropout).

To see it in your own model, print the gradient norm of each parameter after backward():

Python
loss.backward()
for name, p in model.named_parameters():
    if p.grad is not None:
        print(f"{name:20s} grad norm {p.grad.norm():.2e}  weight norm {p.norm():.2e}")

Healthy training has gradient norms of similar order of magnitude across layers, and an update-to-weight ratio around 1e-3. If the first layers have norms thousands of times smaller than the last ones, you have a vanishing problem. The NumPy stand-in below builds a 12-layer network and measures the backward signal with two activations.

Python Measure gradient size layer by layer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A ratio far below 1 means vanishing, far above 1 means exploding, and near 1 is healthy. The ReLU network with a scale around 1.4 (the "He" initialization, sqrt(2)) keeps the signal steady; the sigmoid network loses it, and a larger scale makes ReLU explode.

The overfit-one-batch test

If you remember one technique from this lesson, make it this one. Take a single small batch, turn off all regularization and augmentation, and train on only that batch for a few hundred steps. A correct model and training loop must drive the loss to nearly zero, because memorizing sixteen examples is easy. If it cannot, the bug is in the code, not in the data or the hyperparameters.

Python
xb, yb = next(iter(loader))
xb, yb = xb.to(device), yb.to(device)

model.train()
for step in range(300):
    optimizer.zero_grad()
    loss = loss_fn(model(xb), yb)
    loss.backward()
    optimizer.step()
    if step % 50 == 0:
        print(step, loss.item())
# expected: loss close to 0; if it stalls, check labels, loss, zero_grad, learning rate

If the loss does not move at all, suspect the learning rate, a missing optimizer.step(), parameters that are not in the optimizer, or requires_grad=False. If it goes down but stalls well above zero, check the loss/label pairing and the model's capacity. This stand-in shows the outcomes with a tiny NumPy network: a correct loop, a data bug (the same inputs carry contradicting labels, so the loss cannot fall below ln 2, about 0.693) and a learning rate that is far too small.

Python Overfit one batch: healthy and broken runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reproducibility and seeds

Debugging is far easier when a run can be repeated. Random numbers enter through weight initialization, data shuffling, dropout and augmentation. Set every source:

Python
import random
import numpy as np
import torch

def seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)              # CPU and all CUDA devices
    torch.cuda.manual_seed_all(seed)

seed_everything(42)

# DataLoader workers: give each worker a derived seed and use a seeded generator
g = torch.Generator()
g.manual_seed(42)
loader = DataLoader(dataset, shuffle=True, num_workers=4, generator=g)

torch.use_deterministic_algorithms(True)   # raises an error for non-deterministic ops

Some GPU operations are non-deterministic by nature, so exact bit-for-bit reproducibility on a GPU can cost speed, and different PyTorch versions or hardware can give different numbers even with the same seed. Aim for "same seed, same machine, same code gives the same loss curve", and compare runs by averaging several seeds rather than trusting one.

Common mistakes

  • Debugging on the full dataset and the full model. Shrink to one batch, a tiny model and the CPU first.
  • Looking only at the loss. Also inspect inputs (print a few samples and labels), outputs and gradient norms.
  • Forgetting model.eval() when validating, so dropout still drops units and BatchNorm still updates statistics.
  • Normalizing the training data one way and the test data another.
  • Leaving anomaly detection or CUDA_LAUNCH_BLOCKING=1 on after the bug is fixed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model outputs shape (32, 1), the labels have shape (32,), and your loss is ((pred - y) ** 2).mean(). What happens?

    A (32, 1) minus (32,) broadcasts to (32, 32). Assert shapes or use y.unsqueeze(1) so the mistake cannot hide.
  2. 2. What does torch.autograd.set_detect_anomaly(True) help you find?

    It adds forward tracebacks to the graph and raises as soon as a backward function returns NaN, pointing to the responsible line. It is slow, so use it only while debugging.
  3. 3. You train on a single batch of 16 samples for 300 steps and the loss stays high. What does that most likely mean?

    Memorizing a tiny batch is easy for any working setup. If it fails, look for bugs before tuning anything else.
  4. 4. The gradient norm of the first layer is 1e-9 and of the last layer is 1e-2. What is this called?

    The signal shrinks as it travels back through the layers. ReLU, good initialization, normalization and residual connections are the usual remedies.

Summary

  • Read shape errors as two shapes; assert shapes at boundaries, because broadcasting can hide bugs.
  • Dtype and device errors are fixed with .float(), .long() and .to(device); use CUDA_LAUNCH_BLOCKING=1 to locate GPU errors.
  • For NaN or inf, check the data, use stable losses on logits, clip gradients and use set_detect_anomaly(True) to find the first bad operation.
  • Print gradient norms per layer to detect vanishing or exploding gradients.
  • Overfit a single batch first: a correct pipeline drives its loss near zero.
  • Seed Python, NumPy and PyTorch, and seed DataLoader workers, to make runs repeatable.

Next, learn how to guard against these bugs automatically in Testing and Deploying Models.