Skip to content
Tutorials Center
My learning

Topics

Lesson 62 of 73 ยท Projects

Project: Cat vs Dog and CIFAR-10 Classifier

Build an image classifier project: data augmentation and transfer learning with a NumPy CNN on toy images, plus the PyTorch recipe for CIFAR-10 and cats vs dogs.

Two image data sets are the "first real project" of almost every deep learning course. Cats vs. dogs is a binary problem on full-size photographs: tens of thousands of pictures, each one either a cat or a dog. CIFAR-10 is a ten-class problem on tiny 32 by 32 color images: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks, with 50,000 training images and 10,000 test images. They teach the two techniques that decide most real image projects: data augmentation, which stretches a small data set, and transfer learning, which borrows what a network learned on a much larger one.

The in-browser runner has no GPU and cannot download photo data sets, so this project works in two layers. The ideas are run for real on a small convolutional network written in NumPy, using generated 12 by 12 pictures of six simple shapes in place of photographs, so that every number below comes from code you can execute. The real-photo recipe (PyTorch and torchvision) is shown as read-only code that you can copy to a machine with a GPU. Treat the percentages here as demonstrations of a pattern, not as benchmarks for CIFAR-10.

Two tools for small image data sets

Augmentation Random shifts and flips create new training views for free.
Transfer learning Start from filters trained on a big data set.

Both reduce how many labeled images you need.

1 / 2

The data: six shapes standing in for photos

Our stand-in data set has six classes: disc, square, diamond, cross, ring and triangle. Each image is 12 by 12 pixels with Gaussian noise. The training shapes are always at the center, and the test shapes appear anywhere within three pixels of it. That is the same situation as a real photo collection: your training images cannot cover every position, zoom, angle and lighting condition that will turn up later. The CNN in the code is the one from Training a CNN From Scratch: one 3 by 3 convolution layer with 8 filters, ReLU, 2 by 2 max pooling and a dense softmax layer, trained with Adam.

Step 1: a baseline and the problem of too little data

We give the network only 20 training images per class (120 in total) and train for 40 epochs. Without any augmentation it memorizes: training accuracy reaches 100% while the test accuracy collapses, because the test shapes sit in places the network has never seen them. Data augmentation repairs this by applying random label-preserving changes each time a batch is drawn. Here that means shifting the picture by up to three pixels and flipping it horizontally. The network never sees the identical image twice, so it cannot memorize pixel positions, and it is forced to learn features that work wherever the shape is. The idea was introduced in Dropout and Data Augmentation.

Python Augmentation with a NumPy CNN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is stark. Without augmentation the network scores 100.0% on its training images and 23.0% on the test set, barely above the 16.7% of random guessing among six classes. With shift-and-flip augmentation the training accuracy falls to 60.8% (the training task has become harder, since the shapes now move) but the test accuracy more than doubles, to 51.0%. That is the signature of augmentation: lower training accuracy and higher test accuracy. Still, 51% is not good; with only 120 images and a single small convolution layer, the network is still starved of data. The next step attacks that directly.

The augmentations you choose must match what varies in the real world. Horizontal flips are safe for cats and dogs but wrong for digits or text. Small random crops and shifts help nearly every task; color jitter helps when lighting varies; vertical flips only make sense for satellite or microscope images. Apply augmentation only to the training data, never to the validation or test set.

Step 2: transfer learning

A convolutional network learns its early layers first, and those layers learn generic things: edges, corners, blobs, textures. They are useful for almost any image task. Transfer learning exploits that. Train (or download) a network on a big source data set, keep its convolutional filters, and train only a new output layer for your own small problem. You can then optionally fine-tune: continue training the borrowed layers too, with a smaller learning rate. See Transfer Learning and Fine-Tuning for the full picture.

We imitate this in miniature. The source task has four shapes (disc, square, diamond, cross) and 1,500 labeled images. The target task has two new shapes, ring and triangle, with only 5, 10, 20 or 50 labeled images per class. We compare three networks: trained from scratch; with the source filters frozen and a new head trained; and with the source filters used as the starting point and then fine-tuned.

Python Frozen and fine-tuned features versus training from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The source network reaches 75.0% on its four shapes, which is not perfect, but its filters have learned useful local detectors. On the new two-class task the scratch network needs lots of labels: 71.5% with 20 images per class and 94.2% with 50. Frozen pretrained filters give 85.2% at 20 images per class, about 14 points better, and 72.5% against 67.2% at 10. With only 5 images per class every method is weak (56.5% to 60.3%) and the differences are within noise, and with 50 images per class the advantage disappears (93.0% against 94.2%). The fine-tuned variant did not beat the frozen one here: with such a tiny network and only 30 epochs there is nothing much to gain, and a few points either way is noise from the 600-image test set. In real projects the pattern is similar but the gap is larger, because a ResNet pretrained on ImageNet has seen a million photographs.

Each cell of that table is one training run on 600 test images, so differences of a few points are not reliable. For a real decision, repeat with several random seeds and compare the averages, as the reinforcement-learning lesson also advises.

Step 3: the real-photo recipe

On real data the same two ideas look like this. This code is read-only (it needs PyTorch, torchvision and a GPU for reasonable speed).

Python
# Read-only: CIFAR-10 with augmentation and a small CNN trained from scratch
import torch, torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

mean, std = (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)      # per-channel statistics of the training set
train_tf = transforms.Compose([transforms.RandomCrop(32, padding=4), transforms.RandomHorizontalFlip(),
                               transforms.ToTensor(), transforms.Normalize(mean, std)])
test_tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean, std)])   # no augmentation here
train = DataLoader(datasets.CIFAR10("data", train=True, download=True, transform=train_tf), batch_size=128, shuffle=True)
test = DataLoader(datasets.CIFAR10("data", train=False, transform=test_tf), batch_size=256)

def block(i, o):
    return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(), nn.MaxPool2d(2))
model = nn.Sequential(block(3, 32), block(32, 64), block(64, 128), nn.Flatten(), nn.Dropout(0.3), nn.Linear(128 * 4 * 4, 10))

A network like this, with batch normalization, dropout and augmentation, reaches roughly 85 to 90% on CIFAR-10 after a few dozen epochs. Without augmentation it overfits sooner and lands noticeably lower. For cats vs. dogs, the usual route is transfer learning from a pretrained backbone:

Python
# Read-only: transfer learning for cats vs dogs (folders: data/train/cat, data/train/dog, ...)
from torchvision import models, datasets, transforms

weights = models.ResNet18_Weights.DEFAULT
backbone = models.resnet18(weights=weights)
for p in backbone.parameters():
    p.requires_grad = False                                  # freeze the pretrained filters
backbone.fc = nn.Linear(backbone.fc.in_features, 2)          # a new 2-class head: the only trainable part

tf = transforms.Compose([transforms.RandomResizedCrop(224), transforms.RandomHorizontalFlip(),
                         transforms.ToTensor(), transforms.Normalize(weights.transforms().mean, weights.transforms().std)])
data = datasets.ImageFolder("data/train", transform=tf)
optimizer = torch.optim.Adam(backbone.fc.parameters(), lr=1e-3)     # only the head is optimized
# After a few epochs, unfreeze the last block and fine-tune with lr=1e-4 for the extra points.

With a pretrained ResNet, cats vs. dogs typically reaches about 97 to 99% accuracy within a couple of epochs, a result that would take far longer and far more data from scratch. For CIFAR-10 the images must be resized (for example to 224 pixels) to suit an ImageNet backbone, and the result usually beats the scratch CNN clearly.

Checklist for your own version

  1. Look at the data first: class balance, a grid of samples, suspicious or mislabeled images.
  2. Split properly: train, validation and test, with no augmented copy of a test image in the training set.
  3. Normalize with the training set's mean and standard deviation (or the pretrained model's statistics when using a pretrained backbone).
  4. Baseline with a small CNN, and write down its accuracy.
  5. Add augmentation that matches how real images vary, and compare the train and test accuracies.
  6. Use a pretrained backbone; freeze it first, then unfreeze the top blocks with a smaller learning rate.
  7. Read the errors: confusion matrix, then look at the images the model gets wrong (cat vs dog mistakes, CIFAR-10 cat vs dog and truck vs car confusions are classic).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the augmentation experiment, why did the training accuracy go down while the test accuracy went up?

    Memorizing positions gave 100% training accuracy but failed on new positions. Augmented training makes the task harder to memorize and pushes the network to learn shift-tolerant features.
  2. 2. Which augmentation is a bad idea for a digit classifier?

    A flipped 2 or 5 is not a valid digit, so the label no longer matches the image. Augmentations must preserve the label.
  3. 3. What does freezing the pretrained layers mean?

    A frozen layer still does its forward computation, but its parameters receive no updates, which keeps the pretrained features intact.
  4. 4. When did transfer learning help most in the NumPy experiment?

    At 20 images per class the frozen features gave about 14 points more than training from scratch. With 50 images per class the gap closed, since there was enough data to learn the filters directly.
  5. 5. Why should augmentation be applied to the training set only?

    Evaluating on altered images would measure something else, and an augmented copy of a training image in the test set would leak information.

Summary

  • Image projects on small data sets are decided by two techniques: data augmentation and transfer learning.
  • Augmentation applies random label-preserving changes to training batches. It lowers training accuracy and raises test accuracy when the model would otherwise memorize.
  • Transfer learning reuses filters trained on a large source data set; freeze them and train a new head, then optionally fine-tune with a smaller learning rate.
  • Gains from pretrained features are largest when labeled data is scarce and shrink as you collect more.
  • The NumPy experiments ran on generated shapes; on CIFAR-10 or cats vs. dogs, the same recipe is a few lines of PyTorch with torchvision transforms and a pretrained ResNet.
  • Always normalize with training statistics, split cleanly, repeat noisy comparisons across seeds and read the errors.

Next project: Chest X-ray Pneumonia and Lung Cancer Detection.