Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 46 ยท Projects

Project: Handwritten Digit Recognizer

Build a digit recognizer end to end: preprocess images, engineer features, pick a model with cross-validation, study the errors and write a short report.

This project ties the classical half of the course together. You will take 1,797 small images of handwritten digits and build a recognizer that reads them with more than 98% accuracy, using nothing but NumPy and scikit-learn. The point is not the number. The point is the process a careful practitioner follows: look at the data, lock away a test set, turn images into features, compare models fairly with cross-validation, tune the winner, and finally study the mistakes instead of just celebrating the score.

You have met every piece before. Classifying Handwritten Digits trained a first model, Hand-Crafted Features and HOG explained gradient histograms, and Evaluating Vision Models warned about leakage. Here they run as one pipeline.

flowchart LR
    A["Load and inspect<br/>1,797 images"] --> B["Split once:<br/>train and test"]
    B --> C["Preprocess<br/>and features"]
    C --> D["Cross-validate<br/>on train only"]
    D --> E["Tune the<br/>best model"]
    E --> F["One final score<br/>on test"]
    F --> G["Error gallery<br/>and report"]

The test set is touched exactly once, at the end. Every decision before that uses only the training rows.

Step 1: Look at the data

Always start by looking. The scikit-learn digits set holds 8 by 8 grayscale images with pixel values from 0 (blank) to 16 (full ink). The helper below prints an image as text, so you can see what the model sees.

Python Inspect the digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three facts matter. The classes are balanced (174 to 183 images each), so plain accuracy is a fair headline number. The pixel range is 0 to 16, so dividing by 16 puts every value between 0 and 1, which helps many models. And each image has only 64 pixels, so the strokes are coarse: a 7 and a 1, or an 8 and a 9, can look alike even to a person.

Note: Real handwriting sets such as MNIST use 28 by 28 images and need slightly more preprocessing (centering, size normalization). The workflow in this project carries over unchanged.

Step 2: Split once, then engineer features

Before computing anything clever, hold out a test set. A stratified split keeps the class proportions equal in both parts. After this point the 450 test images are off limits until the very end; looking at them to choose a feature or a model would leak information and make the final score optimistic.

Next, the features. Raw pixels are a strong baseline for digits, but you can add the gradient-orientation histograms from the HOG lesson: for each 4 by 4 quadrant, add up edge strength per direction (8 directions), then normalize. That gives 32 numbers that describe stroke direction regardless of exact pixel position.

Python Features and a fair comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table honestly. Gradient histograms alone are weaker (0.885) because the 2 by 2 grid of cells is coarse for such tiny images. Raw pixels reach 0.973. Combining both gives 0.978, a small gain, but the spread between folds shrinks from 0.011 to 0.003, which means the result is also more stable. Each number comes from four folds trained on 1,347 training images only; the test images never entered.

Step 3: Compare models with cross-validation

A single train/validation split can flatter or punish a model by luck. Cross-validation splits the training rows into folds, trains on all but one, scores on the held-out fold, rotates, and reports the average and the spread. Using the same cv object for every candidate means they all see identical folds, so the comparison is fair.

Python Four candidate models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four land between 0.978 and 0.987, which is a tight race. The RBF-kernel SVM is on top with 0.987, k-nearest neighbours follows at 0.986, and the folds vary by a few tenths of a percent. With differences this small you should not over-interpret the ranking; a gap smaller than the fold-to-fold standard deviation is a tie. The RBF SVM is a sensible pick because it is both the best and fast to train on this data, so the next step tunes it.

Step 4: Tune, then spend the test set once

An RBF SVM has two main knobs. C controls how hard the model tries to fit every training point, and gamma controls how far one training image influences its neighbours. GridSearchCV tries every combination with the same cross-validation, picks the best average score, and then refits that model on all training rows. Only then do we score the 450 test images, exactly once.

Python Tune, test and read the errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The grid shows how sensitive the model is to its settings: with C=1, gamma=0.005 the cross-validated accuracy is only 0.932, while C=10, gamma=0.02 reaches 0.987. Defaults rarely sit at the peak. The tuned model then makes 4 mistakes on 450 test images, a test accuracy of 0.9911, slightly above the cross-validation estimate. That is normal variation: with 450 images, a single error moves accuracy by 0.2 points, so treat 0.99 as "about 98.5 to 99.5", not as a precise figure.

A score tells you how often the model fails; the error gallery tells you why. The four errors above are (true to predicted) 3 to 7, 4 to 8, 8 to 1 and 9 to 5. Look at the printed images: at 8 by 8 resolution some strokes are broken or blobby, so a few shapes really are ambiguous. Errors of this kind suggest the next improvement is more information (higher resolution, more training data), not a fancier classifier. If instead the mistakes had clustered on one digit, you would look for a missing feature or a labeling problem.

A short report keeps the project honest and reproducible. Write down:

  1. Data: 1,797 images, 8 by 8, balanced classes, split 75% / 25% stratified with random_state=0.
  2. Features: 64 scaled pixels plus 32 gradient-histogram values.
  3. Model selection: four models with 4-fold cross-validation on the training rows; RBF SVM won with 0.987.
  4. Final model: SVC(C=10, gamma=0.02), test accuracy 0.9911 (4 errors of 450), worst recall 0.977.
  5. Limits: tiny, clean, centered digits; no test of different handwriting, rotation or noise.

The last line matters most. A model that is 99% accurate on this data says nothing about a photo of a house number. Mentioning what you did not test is what separates a report from a brag.

Ideas to extend it

  • Add the training-set augmentation from Training a Small Network on Images (shifts and noise) and see whether cross-validation improves.
  • Replace the 2 by 2 gradient cells with 4 by 4 cells of size 2 and compare.
  • Use PCA to keep 20 components and check the speed and accuracy trade-off.
  • Corrupt the test images with noise from Resolution, Sampling and Noise and measure how fast accuracy falls.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do you split off the test set before choosing features or models?

    Any choice guided by test images leaks information into the model selection and makes the final score optimistic. The test set must be used once, at the end.
  2. 2. Four models score 0.978, 0.984, 0.986 and 0.987 in cross-validation, with fold standard deviations of about 0.005. What is the fair conclusion?

    Differences smaller than the fold-to-fold spread are within noise. Other criteria then decide: training time, simplicity, robustness.
  3. 3. What does the gamma parameter of an RBF SVM control?

    A small gamma lets each training point influence a wide area (smooth boundary); a large gamma makes the boundary follow individual points closely.
  4. 4. What is the main purpose of an error gallery?

    Looking at the misclassified images shows whether errors come from ambiguity, labeling mistakes or a missing feature, which tells you what to fix next.
  5. 5. The test set has 450 images. By how many percentage points does a single extra error change accuracy?

    One error is 1/450, about 0.22 percentage points, so small accuracy differences on a small test set are not very meaningful.

Summary

  • A good project follows a fixed order: inspect, split, features, cross-validate, tune, test once, analyze errors, report.
  • A stratified hold-out split keeps class proportions and protects the final score from leakage.
  • Combining raw pixels with gradient-histogram features gave a small but stable gain over either alone.
  • Cross-validation with a shared cv object compares models fairly; gaps smaller than the fold spread are ties.
  • GridSearchCV tunes C and gamma; the tuned RBF SVM reached 0.9911 on the test set with 4 errors out of 450.
  • The error gallery and an honest list of limits matter as much as the accuracy number.

Next up: test yourself on the whole course with Practice Exercises with Solutions.