Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 45 ยท Projects

Project: Handwritten Digit Classifier

Build a handwritten digit classifier end to end: the PyTorch CNN pipeline, then a live scikit-learn version with a confusion matrix and an error gallery.

Recognizing handwritten digits is the classic first real machine-learning project, and it contains every step of a professional workflow in miniature: look at the data, preprocess it, split it fairly, train, measure on unseen data, and then study the mistakes. In this project you will do all of that. The part that matters for PyTorch is the pipeline and the way you think about the results, and both are the same whatever the library.

PyTorch and torchvision cannot run on this site, and downloading MNIST needs a network, so the PyTorch CNN is read-only. The live editors use scikit-learn's built-in digits dataset (1,797 small 8 by 8 grayscale images) and a small neural network from scikit-learn as a stand-in. Preprocessing, splitting, the confusion matrix and the error gallery are written in NumPy and are exactly what you would do with PyTorch tensors. Read Image Classification Pipeline and Convolutions and CNNs first if you have not.

The project, step by step

Look Shapes, ranges, class balance.
Preprocess Scale, normalize, split.
Train A small network, Adam.
Analyze Confusion matrix and mistakes.

The last step teaches the most.

1 / 2

Step 1: look at the data and preprocess it

Before any model, answer four questions. What shape is one example? What range do the values have? Are the classes balanced? Which part of the data is held out for testing? The first editor answers them for digits, builds the equivalent of torchvision's ToTensor plus Normalize by hand, and shows a few images together with a randomly shifted copy of each (a simple augmentation, like RandomAffine).

Two details deserve attention. The normalization statistics (mean and standard deviation) are computed from the training images only, then applied to everything; computing them on the whole dataset would leak a little information about the test set. And the split is random but seeded, so the experiment is reproducible.

Python Inspect, normalize and augment the digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The images are 8 x 8 arrays of floats from 0 to 16 (ink darkness), and the ten classes are almost perfectly balanced (174 to 183 images each), so plain accuracy is a fair metric. The training pixels have a mean of about 0.305 and a standard deviation of about 0.376 on the 0-to-1 scale; after normalization they are exactly 0 and 1. The gallery shows that a one-pixel shift is a small change to a person but a big change to a model that has memorized pixel positions, which is the reason the third editor exists.

Step 2: the PyTorch version of the pipeline

With real PyTorch you would use MNIST (28 by 28 pixels, 60,000 training images) and a small CNN. This is the whole project in read-only code. Compare each part with the stand-ins below.

Python
import torch
from torch import nn
from torch.utils.data import DataLoader, random_split
from torchvision import datasets, transforms

device = "cuda" if torch.cuda.is_available() else "cpu"

tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
full = datasets.MNIST("data", train=True, download=True, transform=tf)
test = datasets.MNIST("data", train=False, transform=tf)
train, val = random_split(full, [54_000, 6_000], generator=torch.Generator().manual_seed(0))

train_loader = DataLoader(train, batch_size=128, shuffle=True)
val_loader = DataLoader(val, batch_size=512)
test_loader = DataLoader(test, batch_size=512)

class DigitNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),   # 16 x 14 x 14
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # 32 x 7 x 7
        )
        self.head = nn.Sequential(nn.Flatten(), nn.Dropout(0.25), nn.Linear(32 * 7 * 7, 10))

    def forward(self, x):
        return self.head(self.features(x))             # raw logits, no softmax

model = DigitNet().to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

def evaluate(loader):
    model.eval()
    correct = 0
    with torch.no_grad():
        for x, y in loader:
            correct += (model(x.to(device)).argmax(1).cpu() == y).sum().item()
    return correct / len(loader.dataset)

for epoch in range(5):
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        loss_fn(model(x), y).backward()
        optimizer.step()
    print(f"epoch {epoch}: val accuracy {evaluate(val_loader):.4f}")

print("test accuracy:", evaluate(test_loader))        # touch the test set once, at the end

A network like this typically reaches around 99% test accuracy on MNIST after a few epochs on a laptop CPU. Note the discipline in the code: the validation set guides decisions during training, and the test set is used once at the end. The model returns logits and CrossEntropyLoss applies the softmax internally, as described in Activation and Loss Functions. To collect predictions for the analysis steps below you would run the model over test_loader under torch.no_grad() and keep argmax(1) and the softmax probabilities.

Step 3: train and evaluate the stand-in

The next editor trains scikit-learn's MLPClassifier (one hidden layer of 64 ReLU units, the Adam optimizer, cross-entropy loss: the same ingredients as the PyTorch model, minus the convolutions) on the 1,400 training images and tests it on the other 397. It then draws the confusion matrix: a 10 by 10 table where the row is the true digit, the column is the predicted digit, and the diagonal holds the correct answers.

Python Train, test and draw the confusion matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training stops after 58 iterations (scikit-learn stops when the loss no longer improves), and the model scores 100% on the training images but about 98.2% on the test images: 7 mistakes out of 397. That gap is the usual sign of a model that has memorized its training set a little. The confusion matrix shows that the errors are scattered rather than concentrated: the most frequent mix-up is a true 9 predicted as a 5, only twice. In a bigger project, a bright off-diagonal cell tells you which pairs of classes need more data or better features.

Step 4: study the mistakes

A single accuracy number hides what is going wrong. Two habits help. First, look at the confidence of the model: a classifier that is right with 99% confidence and wrong with 74% confidence on average is well calibrated enough to be useful, because low confidence can trigger a human review. Second, look at the actual wrong images. The error gallery below sorts the mistakes by confidence, so the first images are the ones where the model was most sure and most wrong; they are often ambiguous or badly written digits, and sometimes a mislabeled example, which is a data problem and not a model problem.

The same editor then tests robustness. A model that scores 98% on neatly centered digits can collapse when the digits move by one pixel. The editor evaluates the plain model and a second model that was trained on the originals plus two randomly shifted copies of every image, both on a test set where each image was shifted by up to one pixel.

Python Error gallery and the effect of augmentation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers tell a clear story. The same 7 mistakes appear (average confidence 0.74, against 0.991 on correct answers), but on shifted test images the plain model falls to about 46% accuracy, while the model trained with shifted copies reaches about 92%. Same architecture, same optimizer, same number of epochs: the only change is what the training data looked like. In PyTorch you get this for free by putting a random transform such as transforms.RandomAffine(degrees=0, translate=(0.1, 0.1)) in the training pipeline, so every epoch shows a slightly different version of every image. Apply no augmentation to validation and test data, because you want a stable measurement.

A CNN would also be less sensitive to shifts than the MLP used here, because a convolution applies the same small filter at every position (see Convolutions and CNNs), but augmentation helps CNNs as well.

Extend the project

  • Add noise, small rotations or a stretch to the augmentation and see which one helps the shifted test set most.
  • Replace the confusion-matrix counts by row percentages, so each row sums to 100% and rare classes are easier to compare.
  • Report accuracy per class, and find the weakest digit.
  • Plot confidence on correct versus wrong answers as two histograms and pick a threshold below which the model should say "not sure".
  • In PyTorch, add Dropout and weight decay, plot train and validation accuracy per epoch, and use early stopping on the validation curve.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are the normalization mean and standard deviation computed from the training images only?

    Any statistic computed on the test data makes the evaluation slightly optimistic. Compute the statistics on the training set and apply them unchanged to validation and test data.
  2. 2. What does a bright off-diagonal cell in a confusion matrix mean?

    Rows are true classes and columns are predictions, so off-diagonal cells count specific mix-ups, for example true 9 predicted as 5.
  3. 3. The plain model scores about 98% on the test set but about 46% when each test image is shifted by a pixel. What explains the drop?

    The training data had no variation in position, so the model did not learn to ignore it. Augmentation with shifted copies fixes much of the problem.
  4. 4. Which data should receive random augmentation in a PyTorch pipeline?

    Augmentation enlarges the variety the model sees while learning. Validation and test data must stay deterministic so measurements are comparable.

Summary

  • A complete classification project is: inspect the data, preprocess with training-set statistics, split reproducibly, train, evaluate on held-out data, then analyze the mistakes.
  • The PyTorch pipeline uses torchvision transforms, DataLoaders, a CNN that outputs logits, CrossEntropyLoss, Adam, a validation set for decisions and a test set used once.
  • The confusion matrix shows which classes are mixed up; the error gallery shows why, and sorting mistakes by confidence finds ambiguous or mislabeled examples.
  • A model can be accurate on clean test data and fragile under small shifts; augmenting the training data (not the test data) addresses that.
  • The live stand-in used scikit-learn's MLPClassifier, with the same ingredients as the PyTorch model but no convolutions.

Continue with Project: Sentiment Classifier.