Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 46 ยท Evaluating and Shipping Vision Systems

Evaluating Vision Models

Learn to judge an image model honestly: why accuracy misleads, per-class precision and recall, confusion matrices, mAP, data leakage, dataset shift and slices.

A vision model that scores 97% in a notebook can still fail badly in the real world. The score is only as honest as the way you measured it. Which images were held out, which metric you chose, and which kinds of images the test set contains all decide whether the number means anything.

This lesson collects the habits that separate a trustworthy evaluation from a flattering one. You will see why plain accuracy misleads on imbalanced data, build a confusion matrix and per-class metrics from scratch, compute average precision (the building block of mAP), catch data leakage, simulate a dataset shift, and break errors down by slice. Everything uses small NumPy and scikit-learn demos. If you want the box and mask metrics themselves, they come in Object Detection Basics and Semantic Segmentation and Its Metrics.

Why accuracy alone misleads

Accuracy is the share of predictions that are correct. It is easy to compute and easy to fool. Imagine a factory camera that checks parts for cracks, and only 5 parts in 100 are cracked. A "model" that always says "no crack" is right 95 times out of 100 and catches zero cracks.

The fix is to look at errors per class. For a class c, with true positives (TP, correctly found), false positives (FP, wrongly flagged) and false negatives (FN, missed):

TEXT
precision = TP / (TP + FP)     "of everything I flagged, how much was right?"
recall    = TP / (TP + FN)     "of everything that was there, how much did I find?"
F1        = 2 * precision * recall / (precision + recall)

A confusion matrix collects all of this in one table. Row i, column j counts the images whose true class is i and predicted class is j. The diagonal holds the correct answers. Row sums give the true counts per class and column sums give the predicted counts, so recall comes from rows and precision from columns.

Python A confusion matrix and per-class metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy is 0.86, yet the rare class 2 has a recall of only 0.20: four of its five images are missed. The always-0 model scores 0.80 without learning anything. Macro F1 averages the per-class F1 scores with equal weight, so a weak rare class pulls the number down (0.625 here) instead of hiding behind the big class. Pick the metric that matches the cost of mistakes: high recall when missing something is expensive (a tumor), high precision when false alarms are expensive (an alert that wakes someone up).

flowchart LR
    A["Predictions<br/>and true labels"] --> B["Confusion matrix"]
    B --> C["Per-class<br/>precision and recall"]
    C --> D["F1 or a<br/>cost-weighted score"]
    B --> E["Which classes<br/>get confused?"]

The matrix is the raw material. Every other classification metric is a ratio of its cells.

Average precision and mAP

A detector (see Object Detection Basics) returns boxes with confidence scores, so precision and recall depend on where you cut the score. Instead of picking one cutoff, average precision (AP) sweeps all of them. Sort the detections by score, walk down the list, and after each one record the precision and recall so far. AP is the area under the resulting precision-recall curve. mAP (mean average precision) averages AP over all classes, and some benchmarks also average over several IoU thresholds for matching a box to the truth.

The demo below takes ten detections of one class, already judged as correct or wrong by an IoU rule, and computes AP. There are six real objects in the images, so recall can never pass 5/6 if one object is never found.

Python Average precision from a ranked list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The envelope step matters: a false positive makes the raw precision dip, but you could have avoided it by stopping earlier, so the standard definition uses the best precision available at each recall level. Here AP comes out near 0.66. A single class AP says little on its own; compare the per-class values, and always state the IoU threshold used for matching.

Data leakage: the silent score inflator

Data leakage means information from the test set sneaks into training, so the score no longer measures generalization. In image work the classic leak is near-duplicates. Frames from the same video, several crops of one photo, or augmented copies of one image land on both sides of a random split. The model then "recognizes" the training twin instead of learning the concept.

The demo builds 240 digit images, makes 5 slightly noisy copies of each, and trains a 1-nearest-neighbour classifier. Splitting by image lets twins land on both sides; splitting by group (all copies of one original stay together) does not.

Python A random split versus a group split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random split reports a perfect 1.0 because 176 originals appear on both sides; the honest group split gives 0.89. Leakage has more disguises than duplicates: normalizing with statistics from the whole dataset before splitting, tuning hyperparameters on the test set, or splitting video frames randomly instead of by video. The rule of thumb is to split first, by whatever unit the real world would never share (patient, scene, camera, day), and touch the test set once.

Warning: If you look at the test score and then change the model to improve it, the test set has quietly become a training signal. Keep a separate validation set for tuning and read the test score only at the end.

Dataset shift

A model is trained on one distribution of images and deployed on another. This mismatch is dataset shift. The camera is replaced, the season changes, the room is dimmer, or users upload photos with different compression. The test set from your original data cannot reveal this, because it comes from the same distribution as the training data.

You can probe shift by deliberately changing the test images. The demo trains a logistic regression on clean digits and tests it on versions made dimmer, brighter or noisier.

Python Simulating dataset shift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy falls from 0.970 to 0.874 when the images are merely halved in brightness, even though a person would read them just as easily. Cures include training on varied data, augmentation (see Training a Small Network on Images), normalizing inputs, and monitoring live data, which the next lesson covers in Pitfalls, Ethics and Deployment.

Error analysis by slice

One overall number hides where a model is weak. A slice is a subset of the test set defined by something meaningful: a class, a lighting condition, a camera model, an object size, a skin tone. Compute the metric on each slice and look for outliers. Then open the actual failing images; this is the most productive hour in model development.

Python Accuracy by class and by slice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Digit 8 stands out with an accuracy of 0.885 against 0.96 or more for the rest, and light-ink images do worse than heavy-ink ones. Neither fact is visible in the overall 0.970. Note the sample sizes too: with about 54 images per digit, one extra mistake moves a slice by almost two points, so do not over-read small differences.

Putting it together

flowchart TD
    A["Split first,<br/>by the unit that must not be shared"] --> B["Tune on validation data only"]
    B --> C["Report the test score once"]
    C --> D["Per-class metrics<br/>and confusion matrix"]
    D --> E["Slice analysis<br/>and shifted test sets"]
    E --> F["Look at the failing images"]

A reliable evaluation is a routine, not a single number.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A defect detector sees 3 defective parts in 100 and always predicts "fine". What is its accuracy and its recall for the defect class?

    It is right on the 97 fine parts, so accuracy is 97%, but it never finds a defect, so recall for that class is 0.
  2. 2. In a confusion matrix with true classes as rows, how do you get the recall of class c?

    The row sum is how many images truly belong to class c, so diagonal over row sum is the share of them that were found. Diagonal over column sum is precision.
  3. 3. Why does average precision use the best precision available at each recall level?

    You could have stopped at a higher-precision cutoff, so the standard AP replaces each precision with the maximum at that or any higher recall.
  4. 4. Video frames are randomly split into train and test sets. What is the main risk?

    Frames from the same clip look almost identical. Split by video so that no clip appears on both sides.
  5. 5. A model scores 97% on its test set but drops to 87% on photos from a dimmer room. What is this called?

    The deployed images follow a different distribution than the training and test images. A test set from the original data cannot reveal it.

Summary

  • Accuracy hides weak classes on imbalanced data. Report per-class precision, recall and F1, and read the confusion matrix.
  • Recall comes from rows (true counts), precision from columns (predicted counts). Macro F1 weights every class equally.
  • Average precision is the area under the precision-recall curve of a ranked list. mAP averages AP over classes (and sometimes IoU thresholds).
  • Data leakage, especially near-duplicates across a random split, inflates scores. Split by patient, scene, video or camera, tune on validation data and read the test score once.
  • Dataset shift (lighting, camera, compression) is not visible in a same-distribution test set. Test on deliberately shifted images and monitor live data.
  • Slice the errors by class and by conditions, check slice sizes, and look at the failing images.

Next up: the human side of shipping, in Pitfalls, Ethics and Deployment.