Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 45 ยท Training

The Complete Training Loop

Write a complete PyTorch training loop: epochs and batches, train/validation split, correct metric averaging, eval under no_grad, early stopping and logging.

Every earlier lesson in this chapter built one part of the machine: a model, a loss, an optimizer, a schedule and a data loader. PyTorch deliberately does not give you a fit() function that joins them. You write the loop yourself, and it is only about twenty lines. That freedom is a feature: when something goes wrong you can see every step, and when you need something unusual (two optimizers, gradient accumulation, custom logging) you just edit the loop.

This lesson assembles those parts into a complete training loop with a validation pass, honest metrics and early stopping. The PyTorch code is read-only because PyTorch cannot run in the browser. The live stand-in is a small NumPy neural network with hand-written backpropagation and Adam that trains on synthetic data using exactly the same loop structure, so every number you see is real arithmetic, not a mock-up.

The loop, in four layers

Epochs One full pass over the training data.
Batches Forward, loss, backward, step.
Validation Measure on data the model never trains on.
Stopping Quit when validation stops improving.
1 / 2

The skeleton

Here is the whole pattern in PyTorch. Read it once from top to bottom before we take it apart.

Python
import torch

def run_epoch(model, loader, loss_fn, optimizer=None, device="cpu"):
    training = optimizer is not None
    model.train(training)                       # train() or eval()
    total_loss, correct, seen = 0.0, 0, 0

    with torch.set_grad_enabled(training):      # gradients only when training
        for x, y in loader:
            x, y = x.to(device), y.to(device)
            logits = model(x)
            loss = loss_fn(logits, y)

            if training:
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

            batch = y.size(0)
            total_loss += loss.item() * batch   # weight by batch size
            correct += (logits.argmax(dim=1) == y).sum().item()
            seen += batch

    return total_loss / seen, correct / seen

for epoch in range(1, max_epochs + 1):
    train_loss, train_acc = run_epoch(model, train_loader, loss_fn, optimizer)
    val_loss, val_acc = run_epoch(model, val_loader, loss_fn)
    scheduler.step()
    print(f"epoch {epoch:3d}  train {train_loss:.4f}  val {val_loss:.4f}  val acc {val_acc:.3f}")

Sharing one function between training and validation keeps the two passes consistent: the only differences are the mode switch and the three optimizer lines. Let us go through the pieces.

Splitting the data

Never judge a model on the data it learned from. Split off a validation set (often 10-20%) that the optimizer never sees, using random_split from Datasets and DataLoaders. The training loader shuffles; the validation loader does not need to. Keep a third, untouched test set for one final measurement after all your decisions (architecture, learning rate, when to stop) are made. If you tune against the validation set long enough it stops being an honest estimate.

Train mode, eval mode, no_grad

Two independent switches control the evaluation pass:

Forgetting eval() gives noisy, slightly wrong validation numbers. Forgetting no_grad() still gives correct numbers but wastes memory. The skeleton above handles both with model.train(training) and torch.set_grad_enabled(training). Always switch back to train() at the start of the next epoch, which run_epoch does automatically.

Counting metrics correctly

The loss PyTorch returns is the mean over a batch. The last batch of an epoch is often smaller than the others, so averaging the batch means gives that small batch too much weight. The correct overall mean multiplies each batch loss by its size, sums, and divides by the number of samples, which is what loss.item() * batch and / seen do in the skeleton. .item() converts a one-element tensor to a Python number; accumulating the tensor itself would keep every batch's graph alive in memory.

Python Averaging metrics correctly

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With batches of 4, 4 and 2 samples, the plain mean of batch means (0.5601) differs from the true mean over all samples (0.5482); weighting by batch size recovers it exactly. For accuracy, compare predictions with labels: threshold logits at zero for binary tasks, or take argmax over the class dimension for several classes. You never need to apply softmax or sigmoid just to get a prediction, because they do not change which logit is largest or which side of zero it falls on.

A real loop you can run

The program below is a stand-in: a tiny MLP with manual backpropagation, an Adam optimizer, and the same loop structure as the PyTorch skeleton. It classifies synthetic "two moons" data, with 400 training and 200 validation points. Every piece has a PyTorch counterpart, noted in the comments: the shuffled index permutation is the DataLoader with shuffle=True, evaluate is model.eval() plus no_grad, and the saved-and-restored arrays are what state_dict does for a real model.

Python A training loop on two moons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the log. Training loss falls steadily, and validation loss follows it, then flattens and starts to wobble (0.1699 at epoch 10, 0.1854 at epoch 15, 0.1577 at epoch 20). Validation accuracy sits around 0.95. After epoch 13 the validation loss stops getting better by more than a tiny amount for eight epochs in a row, so the loop stops at epoch 21 and the best weights, from epoch 13 (validation loss 0.1504), are restored. The plot shows the two curves together. The key picture to remember: training loss keeps falling, validation loss eventually stops; the gap between them is overfitting, which Overfitting and Regularization explores.

Early stopping and checkpoints

Early stopping watches a validation metric and ends training when it has not improved for patience consecutive epochs. It saves compute and acts as a regularizer. Two details make it work well:

  1. Remember the weights from the best epoch, not the last one, and restore them at the end. In PyTorch that means copying model.state_dict() (use copy.deepcopy, because state_dict() returns references to the live tensors) or writing it to disk with torch.save, as covered in Saving and Loading Models.
  2. Compare with a small min_delta, so a meaningless improvement in the fifth decimal place does not reset the patience counter.
Python An EarlyStopping helper

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss reaches its best value (0.39) at epoch 4. Epochs 5, 6 and 7 are all worse, so the patience counter reaches 3 and the loop stops with the advice to restore the epoch-4 weights. Wire the same helper into the PyTorch loop like this:

Python
import copy

stopper = EarlyStopping(patience=5, min_delta=1e-4)
best_weights = None

for epoch in range(max_epochs):
    train_loss, _ = run_epoch(model, train_loader, loss_fn, optimizer)
    val_loss, val_acc = run_epoch(model, val_loader, loss_fn)

    if val_loss < stopper.best - stopper.min_delta:
        best_weights = copy.deepcopy(model.state_dict())
    if stopper.step(val_loss, epoch):
        break

model.load_state_dict(best_weights)           # go back to the best epoch

Logging

Print a one-line summary per epoch, as the examples do, and also record the numbers in a list or file so you can plot them later. Useful extras:

  • the learning rate (optimizer.param_groups[0]["lr"]), to confirm the schedule is doing what you expect;
  • the gradient norm returned by clip_grad_norm_, to spot instability early;
  • elapsed time per epoch, to know where the time goes;
  • for bigger projects, a logger such as TensorBoard (torch.utils.tensorboard.SummaryWriter), which draws the curves live while you train.

Before a long run, overfit a tiny batch (say 16 samples) for a few hundred steps. A working loop should drive the loss almost to zero. If it cannot, the bug is in the model, loss or loop, not in the data or hyperparameters. Debugging PyTorch Code develops this idea.

A checklist for your own loop

  • model.train() before the training pass, model.eval() and torch.no_grad() for validation.
  • optimizer.zero_grad(), loss.backward(), optimizer.step() in that order, once per batch.
  • Move both inputs and labels to the same device as the model.
  • Weight loss and accuracy by batch size, and call .item() before accumulating.
  • Step epoch-based schedulers after the epoch, and clip gradients between backward() and step() if needed.
  • Keep the best weights and restore them at the end.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the validation pass in a PyTorch loop use both model.eval() and torch.no_grad()?

    They are independent switches: eval() changes how layers behave, no_grad() stops gradient tracking. Using both gives correct predictions with less memory.
  2. 2. Why multiply each batch loss by the batch size before summing?

    The loss for a batch is already a mean. Weighting by batch size and dividing by the total number of samples gives the exact mean over the whole epoch.
  3. 3. Early stopping with patience=3 and a best validation loss at epoch 4 stops training when...

    The counter increases after every epoch without improvement and resets on a new best. Reaching the patience limit ends the run.
  4. 4. After early stopping, which weights should you keep?

    By the time patience runs out, the model has usually drifted past its best point. Save a copy of the best weights and restore them.
  5. 5. Which line must you NOT forget so gradients from the previous batch do not leak into the next update?

    backward() adds into .grad, so gradients must be cleared once per update. The other lines do not touch the stored gradients.

Summary

  • A PyTorch training loop is written by hand: per epoch, a training pass (zero_grad, forward, loss, backward, step) and a validation pass.
  • Keep a validation set the optimizer never sees, and a final test set you use once.
  • Validate with model.eval() and torch.no_grad(); return to model.train() for the next epoch.
  • Average losses and accuracy weighted by batch size, and use .item() to avoid holding graphs in memory.
  • Early stopping tracks the best validation value with a patience counter, and you restore the best weights afterwards.
  • Log loss, accuracy, learning rate and gradient norm every epoch so problems are visible early.

Next, study why validation loss turns upward in Overfitting and Regularization.