Skip to content
Tutorials Center
My learning

Topics

Lesson 95 of 111 ยท Projects

Project: Handwritten Digit Recognizer

Build a handwritten digit classifier with scikit-learn: compare SVM, random forest and neural network, read the confusion matrix, and speed up training with PCA.

Recognizing handwritten digits is the "hello world" of image classification, and it is a good project for a different reason than spam filtering: there is no text to engineer and no class imbalance to fight. The data is already a table of numbers, so you can spend your attention on choosing between models and on understanding exactly where the winner fails.

scikit-learn ships a small digits dataset with 1,797 scans of the digits 0 to 9, each only 8 by 8 pixels. It is much smaller than the famous 28 by 28 collections, which is why it runs in your browser in seconds. In this project you will look at the data, compare three model families against a baseline, study the confusion matrix and the individual mistakes, and finally use PCA to make training faster without losing much accuracy.

The plan

Look Plot some digits first.
Compare SVM, forest and neural net.
Diagnose Confusion matrix and wrong guesses.
Speed up PCA keeps the signal, drops pixels.

A model is finished when you know why it fails, not when the score looks good.

1 / 2

Step 1: Look at the data

Each image is stored as 64 numbers between 0 and 16, one per pixel, in digits.data. The same pixels also exist as an 8 by 8 grid in digits.images, which is what you need for plotting. Always look at a few examples before modelling: it tells you whether the labels make sense, and it gives you a feel for how hard the task is for a human eye. We then split with stratify, so every digit appears in the same share in train and test (Train, Validation and Test Sets).

Python Step 1: load, plot and split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classes are almost perfectly balanced (about 180 images per digit), so plain accuracy is a fair metric here. That is not true for most real problems, but it makes this a clean place to compare models. The majority-class baseline would score about 10%, the share of any single digit.

Step 2: Compare three models

We try a support vector machine with an RBF kernel (Kernels and Nonlinear SVMs), a random forest (Random Forests) and a small multi-layer perceptron (Gaussian Processes, Neural Nets and Choosing a Model). The SVM and the MLP are sensitive to feature scale, so they get a StandardScaler inside a pipeline. The forest does not need scaling. The dummy classifier is our baseline (Baselines and Dummy Models).

Instead of trusting one split, we use 3-fold stratified cross-validation on the training data to compare the models, and only afterwards look at the test set for the winner (Cross-Validation). The printed times will differ on your computer; the ranking is what matters.

Python Step 2: cross-validated comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three real models score about 96% or more in cross-validation, and the baseline sits at about 10%. The SVM leads with roughly 98%, the forest follows at 97%, and the MLP comes last at 96% with the largest spread between folds (it also prints a convergence warning, because we capped its training at 100 iterations to stay inside the browser's time limit). With differences this small, the standard deviation matters: when the gap between two models is about the size of the fold-to-fold spread, you should not claim a winner from one run. We pick the SVM for the rest of the project because it scores highest, is fast to train and has few settings.

An MLP on 8 by 8 images is small enough to train in a browser tab. On real photographs you would reach for a convolutional network instead, and for that you move to Deep Learning.

Step 3: Read the confusion matrix

Now fit the chosen model on the whole training set and score it on the test set once. The accuracy is a single number, but the confusion matrix shows which digits get confused with which: rows are the true digit, columns are the predicted digit, and everything off the diagonal is a mistake (Classification Metrics).

The code lists the most common confusions and then plots the misclassified test images with their true and predicted labels. Looking at the actual mistakes is the fastest way to decide whether a model needs more data, better features, or just a human to accept that some scrawls are ambiguous.

Python Step 3: confusion matrix and the mistakes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The test accuracy is about 98%, which means only a handful of the 450 test images are wrong. Notice that the mistakes are spread thinly: the most common confusion happens only twice. With such a small test set, a pattern like "6 is mistaken for 4" is a hint, not a proven weakness. On a 450-image test set, one more correct or wrong image moves accuracy by about 0.2 percentage points, so do not tune the model to fix individual mistakes you see here.

When you look at the plotted errors, ask what a person would say. Some of the scrawls are genuinely hard to read at 8 by 8 pixels, and no model can be expected to recover information that the image does not contain.

Step 4: Speed up with PCA

Each image has 64 pixels, but neighbouring pixels are strongly correlated, so far fewer numbers are needed to describe a digit. Principal Component Analysis finds a smaller set of directions that keeps most of the variance. Putting PCA(n_components=k) between the scaler and the SVM gives us a model that trains on only k features.

We try several values of k and also ask how many components are needed to keep 80%, 90% and 95% of the variance. The fit times again depend on your machine, but the pattern should hold: fewer components train faster, and accuracy stays high until k gets too small.

Python Step 4: how many components do we need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 20 components keep 80% of the variance, 30 keep 90% and 39 keep 95%. The accuracy with 20 to 30 components is only slightly below the full 64-pixel model, while 5 components lose much more. That is the usual shape of the curve: the first few components carry most of the signal, so accuracy climbs fast and then flattens.

On this tiny dataset the time saving is small and noisy, and it can even disappear, because the SVM is already fast and PCA has its own cost. The trick pays off when the number of features is large (thousands of pixels, for example) and the model's cost grows with the feature count. Treat PCA here as a way to learn the trade-off rather than as a recommendation for 8 by 8 images.

What to try next

  • Tune the SVM's C and gamma with a grid search (Hyperparameter Tuning), using cross-validation on the training set only.
  • Augment the training data by shifting each image one pixel. This teaches the model that a digit stays the same digit when it moves slightly.
  • Calibrate the SVM or the forest and look at the images the model is least sure about, which are a good candidate list for human review (Probability Calibration and Uncertainty).
  • Use t-SNE to draw all 1,797 digits in two dimensions (t-SNE and Manifold Learning).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do the SVM and the MLP get a StandardScaler in their pipelines while the random forest does not?

    Kernel distances and gradient-based training both depend on the size of each feature, so features are put on a common scale. A tree asks whether one feature is above a threshold, which does not change when the feature is rescaled.
  2. 2. In a confusion matrix, what do the numbers off the diagonal mean?

    Rows are true classes and columns are predicted classes, so the diagonal holds the correct predictions and every other cell counts one kind of mistake.
  3. 3. Two models score 98.2% and 97.9% in cross-validation, and the fold-to-fold standard deviation is about 1 percentage point. What is the honest conclusion?

    A gap smaller than the spread between folds is not evidence of a real difference. Choose on cost, speed, interpretability or maintenance instead.
  4. 4. What does PCA with n_components=30 do to the 64 pixel features?

    Each principal component is a weighted combination of all the original pixels. Keeping the first 30 keeps the directions of largest variance and discards the rest.

Summary

  • The 8 by 8 digits dataset is balanced, so accuracy is a fair metric here; always compare against a baseline anyway.
  • Compare model families with cross-validation on the training data and look at the spread as well as the mean.
  • Give scale-sensitive models (SVM, MLP) a scaler inside a pipeline; trees do not need one.
  • Use the confusion matrix and the plotted mistakes to see where a model fails, and avoid overreacting to a handful of errors on a small test set.
  • PCA can shrink the feature count with little loss in accuracy, and it helps most when there are many features.

Next, move from labelled images to unlabelled customers in Project: Customer Segmentation.