Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 46 ยท Neural Networks for Vision

Training a Small Network on Images

Train a small neural network on handwritten digits, read loss and accuracy curves, spot overfitting, and fight it with data augmentation written in NumPy.

Knowing what a convolution or a pooling layer does is half the story. The other half is training: showing the network labeled images again and again and nudging its weights until its predictions improve. Training is where most real-world problems appear. The loss falls but the test score does not, the model memorizes instead of learning, or a harmless-looking change in the data quietly breaks everything.

In this lesson you will train a small neural network on the 8 by 8 handwritten digits that ship with scikit-learn, watch the loss and accuracy curves change epoch by epoch, see overfitting happen on purpose, and then reduce it with data augmentation (shifts and noise) written in plain NumPy. The same loop works for a deep CNN; only the model and the amount of data grow. If you have not met this dataset, Classifying Handwritten Digits shows it from the classical side.

The training loop in one picture

Every supervised training run repeats the same cycle. A pass over the whole training set is one epoch; a small slice of images processed at once is a batch.

flowchart LR
    A["batch of images<br/>and labels"] --> B["forward pass<br/>predictions"]
    B --> C["loss<br/>how wrong?"]
    C --> D["backward pass<br/>gradients"]
    D --> E["update weights<br/>small step"]
    E --> A

The loop behind every neural network: predict, measure the error, compute gradients, take a small step.

We use MLPClassifier, a multi-layer perceptron: it flattens the 8 by 8 image into 64 numbers and passes them through dense layers. It is not a CNN, so it ignores the 2D layout, but digits are tiny and centered, which makes it fast and instructive. The training loop is the same for CNNs: calling partial_fit once runs one epoch, so we can measure after each one.

Note: Pixel values in this dataset run from 0 to 16, so we divide by 16. Networks train best when inputs are small numbers around 0 to 1. Feeding raw 0 to 255 values to a network is one of the most common reasons for a model that refuses to learn.

Loss and accuracy curves

Two quantities tell the story. The loss (here cross-entropy) is the number the optimizer actually minimizes; it punishes confident wrong answers heavily. Accuracy is the share of correct predictions, which is what you care about but which changes in coarse steps. You always track both, on the training data and on data the network never trains on.

We start with a deliberately small training set of 200 images and test on 899 others. The test data is never used to update weights, so its curve shows how well the model generalizes.

Python Loss and accuracy per epoch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from top to bottom. At epoch 1 the network is barely better than guessing (47.5% on the training set). By epoch 10 it classifies all 200 training images correctly. The training loss keeps shrinking to 0.006, but the test loss stops improving at about 0.25 and the test accuracy flattens near 92%. The model has stopped learning the pattern and started to polish its fit to the 200 images it has seen. That gap between "perfect on training data" and "about 92% on new data" is overfitting.

flowchart TD
    A["train loss down,<br/>test loss down"] -->|"keep training"| B["train loss down,<br/>test loss flat or up"]
    B --> C["overfitting:<br/>memorizing"]
    C --> D["more data, augmentation,<br/>regularization, early stopping"]

The usual shape of a run. The moment the two curves separate is when extra epochs stop paying off.

What overfitting looks like

A network has a lot of freedom. With 64 inputs and 64 hidden units it has roughly 5,000 weights, many more than the 200 training images need to be memorized. The cure that works best is more data. The next experiment trains the same architecture on 40, 100, 300 and 800 images and always tests on the same 899 held-out images.

Python Training set size versus the generalization gap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training accuracy stays near 100% in every row, but test accuracy climbs from 77.2% (40 images) to 96.6% (800 images), so the gap shrinks from 0.228 to 0.027. Training accuracy alone tells you almost nothing; the gap is the signal. Whenever you cannot collect more data, you have three other levers:

  • Regularization: penalize large weights (alpha in scikit-learn, weight decay in deep learning frameworks) or randomly switch off units during training (dropout).
  • Early stopping: keep the weights from the epoch with the best validation score and stop when it stops improving.
  • Data augmentation: create extra, plausible training images from the ones you have.

Warning: Never tune your choices (epochs, learning rate, augmentation) on the final test set. Split off a separate validation set for decisions and touch the test set once at the end, otherwise your reported score is optimistic.

Data augmentation in NumPy

Augmentation applies small random changes that do not change the label: a 3 shifted by one pixel is still a 3. The network sees more variety, and it learns that position and tiny noise do not matter. Typical transforms are shifts, small rotations, crops, brightness changes and noise. Flips depend on the task: mirroring a photo of a cat is fine, but mirroring a digit or a letter can produce a symbol that does not exist or means something else, so you must think about what your labels survive.

The helpers below shift an image by whole pixels (filling the gap with zeros, so nothing wraps around) and add clipped Gaussian noise. We then print one digit in four versions.

Python Shift, noise and mirror a digit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shifted version is the same digit moved diagonally, with a blank first row and column; the noisy version has faint speckles in the background; the mirrored version is a backwards 3, a shape that real handwriting almost never produces, so training on it would teach the network nothing useful. Always ask: would a real image ever look like this?

Does it help? An experiment

Now the test. We train two networks on the same 150 images. One sees the originals; the other sees the originals plus three randomly shifted and noisy copies of each (600 images in total). Both are tested on 899 unseen digits, once as they are and once with each image randomly moved by up to one pixel, imitating a sloppy scan that is not centered.

Python With and without augmentation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is honest rather than magical. On the centered test images, the augmented model is slightly worse (88.9% versus 90.2%): it spent capacity on shifted variants that the clean test set never shows. On the shifted test images the plain model collapses to 38.3% because a dense network treats every pixel position as a different feature, while the augmented model keeps 69.3%. Augmentation buys robustness to the variation you add, not a free accuracy gain, so augment with the changes your deployed system will meet. A CNN is naturally more tolerant to shifts (see The Convolutional Layer), which is one reason it replaced dense networks for images.

The same loop in a deep learning framework

The site cannot run PyTorch in your browser, so this snippet is read-only, but it shows how little changes: the model is a small CNN, the batches come from a data loader, and the augmentation is a transform pipeline applied to every image on the fly.

Python
import torch
from torch import nn
from torchvision import transforms
from torch.utils.data import DataLoader

augment = transforms.Compose([
    transforms.RandomAffine(degrees=0, translate=(0.1, 0.1)),  # shifts
    transforms.ToTensor(),
])

model = nn.Sequential(
    nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Flatten(), nn.Linear(16 * 2 * 2, 10),          # for 8x8 inputs
)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(20):
    model.train()
    for images, labels in train_loader:                 # a DataLoader with augment
        optimizer.zero_grad()
        loss = loss_fn(model(images), labels)
        loss.backward()                                 # gradients
        optimizer.step()                                # update weights
    model.eval()                                        # no dropout, fixed batch norm
    with torch.no_grad():
        val_acc = sum((model(x).argmax(1) == y).sum().item() for x, y in val_loader)

Every line has a counterpart above: loss.backward() and optimizer.step() are the backward pass and update from the diagram, weight_decay is regularization, and the validation loop is the "test" column you watched. Note that augmentation is applied only to training data; validation and test images stay untouched.

A short debugging checklist

  • Loss does not move: check the input scale, the learning rate (try 10 times smaller and larger), and that labels match images.
  • Training accuracy stuck at chance: a bug in the data pipeline is likelier than a bad model. Overfit a batch of 10 images on purpose; if the net cannot memorize 10 images, something is broken.
  • Big gap between train and validation: more data, augmentation, regularization, early stopping, a smaller model.
  • Both curves bad: the model is too small or training too short, or the features do not carry the signal.
  • Set seeds (random_state, default_rng(0)) while experimenting so you can compare runs fairly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is one epoch?

    An epoch is a complete pass through all training images, usually split into several batches with one weight update per batch.
  2. 2. Training accuracy is 100% but validation accuracy is 92%. What does this most likely indicate?

    A large gap between training and validation performance means the model has fitted details of the training images that do not generalize.
  3. 3. Why is mirroring a good augmentation for photos of cats but a risky one for handwritten digits?

    An augmentation must keep the label correct and the image realistic. Flipping preserves the class of a cat photo, but not that of many digits and letters.
  4. 4. In the experiment, why did the augmented network score slightly lower on the clean test set but much higher on the shifted one?

    Augmentation teaches the variation you add. The network became robust to shifts, at the price of some capacity spent on cases the clean test set never contains.
  5. 5. Which data may be used to choose the number of epochs and the augmentation settings?

    Decisions need their own validation data. If you tune on the test set, its score no longer measures performance on unseen images.

Summary

  • Training repeats forward pass, loss, backward pass and weight update over batches; one pass over the data is an epoch.
  • Scale pixel values to a small range (0 to 1) before feeding them to a network.
  • Track loss and accuracy for both training and held-out data. Overfitting shows up as a growing gap between them.
  • More data shrinks the gap most reliably; regularization, early stopping and augmentation help when data is scarce.
  • Augmentation adds label-preserving variation (shifts, noise, crops) to training images only, and only transforms that real images might undergo.
  • Tune on a validation set and keep the test set for one final measurement.

Next up: skip most of the training altogether by reusing someone else's network in Transfer Learning and Modern Vision Models.