Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 73 ยท Projects

Project: Handwritten Digit Classifier

Build a complete digit classifier: split and scale the data, train an MLP with Adam in NumPy, read the confusion matrix, and study the errors it makes.

Time to put the pieces together. In this project you build a complete handwritten-digit classifier: a multi-layer perceptron that reads 8 by 8 pixel images and names the digit. You will do everything a real project needs, in order: look at the data, split it honestly, preprocess it, set a baseline, train a network with the Adam optimizer, evaluate it on data it has never seen, and then study the mistakes it makes.

The dataset is the small digits set that ships with scikit-learn: 1,797 images, 8 by 8 pixels each, with pixel values from 0 to 16. Everything else is NumPy code from earlier lessons, so each step should look familiar. The goal is not a record score; it is to practice the workflow and the habit of looking at errors.

The project plan

Look Inspect images and class counts.
Split Train, validation and test sets.
Preprocess Scale pixels, standardize with train stats.
Baseline Know what "simple" scores first.

Never touch the test set until the very end.

1 / 2

Step 1: look at the data and split it

Always start by looking. Check the shapes, the class balance, and a few images. Then split the data into three parts. The training set fits the weights. The validation set guides decisions (when to stop, which settings to use). The test set is touched once, at the end, to report an honest number. A random permutation with a fixed seed makes the split reproducible (see Experiments and Reproducibility).

Next, a baseline. A model is only impressive relative to something simple, so first try always predicting the most common class, and then a plain logistic regression from scikit-learn. If your network cannot beat logistic regression, something is wrong or the network is not needed.

Python Look, split and baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classes are roughly balanced, so accuracy is a fair metric here (for imbalanced data see Evaluating Models). The majority baseline scores 0.078, which is the floor, and logistic regression already reaches 0.953 on the validation set. That is a strong bar on this easy dataset, and the network has to clear it.

Step 2: preprocess

Scaling pixels into 0 to 1 is the first step. A second, standard step is to standardize each input feature: subtract the training mean and divide by the training standard deviation, so every input has mean 0 and spread 1. Gradient descent likes inputs on similar scales (see Gradient Descent).

Two rules prevent subtle mistakes:

  1. Compute the statistics on the training set only, then apply those same numbers to validation and test data. Using statistics from all the data leaks information from the test set into training.
  2. Some pixels in this dataset are always 0 (the image corners), so their standard deviation is 0 and dividing by it would blow up any test image that has a non-zero value there. A small floor on the standard deviation keeps it safe.

Step 3: the network and Adam

The model is a multi-layer perceptron: 64 inputs, one hidden layer of 64 ReLU units, 10 output logits and a softmax cross-entropy loss. Weights use He initialization (Weight Initialization). Training uses mini-batches of 32 and the Adam optimizer with its usual defaults (beta1 = 0.9, beta2 = 0.999) and learning rate 0.001 (Momentum, RMSProp and Adam).

The loop tracks training loss, validation loss and validation accuracy after every epoch, and remembers the weights from the epoch with the lowest validation loss, a simple form of early stopping.

Python Train the digit classifier with Adam

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the curves before the final numbers. Training loss falls from 2.04 to 0.015 and the training accuracy reaches 1.000: the network can memorize 1,000 images. The validation loss improves more slowly (0.158 at epoch 20, 0.122 at epoch 40), so a gap opens between the two curves, the usual sign of overfitting beginning. Here the validation loss was still creeping down at epoch 40, so the best epoch is the last one; with longer training you would expect it to turn upward, and keeping the weights from the best validation epoch is cheap protection. Validation accuracy is 0.960, only a little above logistic regression's 0.953, which says this dataset is easy and the real gains from deeper models come on harder images. Only now is the test set used, once, for the headline number: 0.970. (If you then go back and tune the model to improve the test score, the test set stops being honest, so stick to the validation set for every decision.)

Step 4: the confusion matrix

Accuracy compresses everything into one number. A confusion matrix shows how the mistakes are spread: row i, column j counts images of true digit i that the model labeled j. A perfect model has numbers only on the diagonal. Off-diagonal cells tell you which pairs of digits get mixed up, and per-class recall (diagonal divided by row total) shows which digits are hardest.

Step 5: error analysis

The most valuable step is the one most often skipped: look at the images the model got wrong. Ask whether the mistakes are reasonable (a sloppy 1 that looks like a 7), caused by bad labels, or revealing a systematic gap (for example, a digit that is rarely written in a certain way in the training data). Fixes follow from the diagnosis: more data or augmentation for weak classes, a better architecture such as a convolutional network, or cleaning labels. This TryIt repeats the training code so it runs on its own, then analyzes the errors.

Python Confusion matrix and the misclassified digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The matrix is dominated by its diagonal: 15 mistakes among 497 test images. Recall is perfect for 2, 5 and 7 and lowest for 8 (0.93) and 9 (0.92), and the most common confusions (9 as 8, 9 as 5, 4 as 7, 1 as 8) each happen only twice. With so few errors per cell, treat these as hints and not as firm conclusions. The few off-diagonal counts show which digits the network confuses, and the gallery shows the images behind them, ordered by how confident the model was while being wrong. Overconfident mistakes are the most informative: they often point at ambiguous or oddly written digits, and sometimes at a mislabeled example. On an 8 by 8 image, even a person has trouble with some of these.

Ideas to extend the project

  • Compare hidden sizes (16, 32, 64, 128) and plot validation accuracy; see Overfitting, Underfitting and Regularization.
  • Add L2 weight decay or dropout (Dropout and Data Augmentation) and see whether the validation loss improves.
  • Augment the training set with images shifted by one pixel.
  • Train with several seeds and report the mean and spread of the test accuracy.
  • Add a second hidden layer, or switch to the CNN of the later lessons, and compare fairly.
  • Check a gradient on a tiny version of the network to prove the backward pass is right.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should the input standardization statistics (mean and standard deviation) be computed on the training set only?

    Statistics from all the data would let information about the held-out sets influence training, making the evaluation optimistic.
  2. 2. What does the cell in row 4, column 9 of a confusion matrix count?

    Rows are the true classes and columns are the predictions, so the cell counts true 4s labeled as 9.
  3. 3. Why do you compare the network with a logistic regression baseline?

    A baseline gives context. If a simple model matches the network, the network is not adding value.
  4. 4. When is the test set used in this workflow?

    All decisions use the validation set. The test set is kept untouched so that its score is an honest estimate.
  5. 5. Which mistakes are the most useful to inspect during error analysis?

    A model that is confidently wrong has learned something systematic, or the label itself is suspect. Both are worth knowing.

Summary

  • A project workflow: inspect the data, split into train, validation and test, preprocess with training statistics, set baselines, train, evaluate once on the test set, analyze errors.
  • Standardizing inputs helps optimization; compute its statistics on the training set only and guard against zero variance.
  • An MLP trained with mini-batch Adam beats a majority-class guess by a wide margin and competes with logistic regression on this easy dataset.
  • Keep the weights from the best validation epoch; watch the gap between training and validation loss.
  • The confusion matrix and per-class recall show where the model fails; looking at the failing images shows why.

Next project: Predicting House Prices With a Network.