Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 46 ยท Evaluating and Shipping Vision Systems

Pitfalls, Ethics and Deployment

What goes wrong when a vision model leaves the notebook: biased data, privacy, adversarial examples, lighting changes, latency, quantization and monitoring.

A good score on a clean test set is the start of the work, not the end. Once a model looks at real cameras and real people, new kinds of problems appear. The data may under-represent some of the people it will see. Images may contain private information. Someone may try to fool the model on purpose. Lighting and cameras will change. The model has to run fast enough on cheap hardware, and its behavior will drift over the months after launch.

This lesson walks through those problems with small, runnable demonstrations and a few read-only snippets for tools that cannot run in the browser. You should already know how to measure a model honestly; if not, start with Evaluating Vision Models.

Bias starts in the dataset

A model learns whatever is in its training data. If one group of images is rare, the model sees few examples of it and usually performs worse on it. This is how face systems have ended up less accurate for some skin tones and ages, how a skin-lesion classifier can fail on dark skin, and how a pedestrian detector can miss people in unusual clothing. The same thing happens with cameras: a model trained mostly on one phone often struggles on another.

You can reproduce the mechanism with digits. Here a rare "camera" produces dim, noisy images, and they make up only about 3% of the training set. The test set contains both kinds of images. The second model is trained on a set that includes a dim copy of every image.

Python A rare group gets worse results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The imbalanced model is fine on the normal camera (0.969) but drops to 0.866 on the dim one; adding dim examples to the training data raises the dim score to 0.907 at almost no cost to the other. The overall test accuracy of the first model would have looked fine, which is why the slice analysis from the last lesson matters so much. Other fixes are collecting more data from the weak group, re-weighting examples, and testing on a benchmark built to cover groups evenly.

Note: Bias is not only a data-balance problem. Labels can carry human prejudice, a task can be ill-defined (guessing emotion or character from a face has weak scientific support), and a system can be accurate and still harmful because of how it is used. Ask who is affected by a mistake, and who gets to appeal it.

Privacy and face data

An image is rarely only what you meant to capture. Photos contain faces, license plates, house numbers, documents on a desk and the metadata stored in the file (GPS position, camera model, time). Faces are biometric data, and in many places the law treats them as especially sensitive: the EU GDPR, Illinois BIPA and various other statutes restrict collecting and processing them, often requiring consent and a clear purpose.

Good practice starts with using less data:

  • Collect the minimum needed for the task, and say why.
  • Process on the device when you can, so raw images never leave the camera or phone.
  • Anonymize early: blur or pixelate faces and plates, and strip the metadata, before storing or sharing.
  • Limit retention and access, and be able to delete a person's images on request.
  • Check dataset licenses and consent for training data; a photo being public does not make it free to use.
  • Prefer detection over identification. Counting people in a room needs no names.

Pixelation, one of the simplest anonymizers, is block averaging, which you can write in three lines of NumPy. Note that heavy pixelation destroys detail but is not a guarantee; models have been shown to recover some identities from mild blur, so use strong settings, or remove the region completely.

Python Pixelating a region

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 2-pixel blocks you can still tell where the eyes and mouth were, 4-pixel blocks leave a vague smudge, and 8-pixel blocks leave only four flat patches, so the features are gone. Choose the block size from the size of the thing you want to hide, not from how the result looks.

Adversarial examples

An adversarial example is an input changed by a tiny, deliberate amount so that a model gives a wrong answer with confidence. To a person the change is invisible or looks like faint noise. The method behind the classic attack, the fast gradient sign method (FGSM), is short: compute how the loss changes with every pixel, then move each pixel a small step eps in the direction that increases the loss:

TEXT
x_adv = clip( x + eps * sign( d loss / d x ), 0, 1 )

For a multi-class linear (softmax) model the gradient is the prediction error (p - onehot(label)) multiplied by the weight matrix, so you can compute the attack without a deep learning library. Real attacks on deep networks use the same idea with automatic differentiation.

Python FGSM on a linear digit classifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A step of 0.05 per pixel, small enough to look like mild noise, already drops accuracy from 0.970 to 0.874, and at 0.2 almost every prediction is wrong. Deep networks are at least as fragile. Physical versions exist too: stickers on a stop sign, patterned glasses that confuse a face recognizer. Defenses include adversarial training (adding attacked images to training), input checks, ensembles and keeping a human in the loop for high-stakes calls, but none is a complete cure. For safety-critical systems assume that an attacker with access to the model can fool it.

Lighting and camera variation

Most failures in the field are not attacks but ordinary variation: sun, shade, night, glare, a dirty lens, a different sensor, a new JPEG quality setting. Two cheap defenses are augmentation during training (random brightness, contrast, blur and noise, see Training a Small Network on Images) and normalization at inference. Per-image standardization subtracts the image mean and divides by its standard deviation, which cancels any change of the form a * x + b.

Python Per-image normalization against lighting changes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The standardized model gives the same 0.974 in every row, because those changes are exactly linear in the pixel values. The raw model falls to 0.798 for dim images. Normalization only handles the changes it was designed for; it does nothing about blur, glare or a rotated camera, so keep testing on shifted data.

Latency, size and quantization

A model that is accurate but too slow or too large is not deployable. Phones, drones and cameras have small CPUs, little memory and a power budget. Track three numbers: latency (milliseconds per image, measured on the target device, including preprocessing, and as a high percentile such as p95 rather than the average), model size (parameters times bytes per parameter) and throughput (images per second when batching).

Ways to shrink a model:

  • Choose a smaller architecture (the mobile-friendly designs from Pooling and CNN Architectures).
  • Lower the input resolution; cost grows with the number of pixels.
  • Quantization: store weights (and sometimes activations) as 8-bit integers instead of 32-bit floats, a 4x size cut that often runs faster on hardware with integer support.
  • Pruning and distillation: remove weights, or train a small model to imitate a large one.

The demo quantizes the weights of the digit classifier. It stores one int8 per weight plus a single float scale, and rebuilds approximate floats when predicting.

Python Quantizing weights to 8 bits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weights shrink from 2,560 to 640 bytes and not one prediction changes. Real networks lose a little accuracy, so always re-run your evaluation after quantizing; per-channel scales, calibration data and quantization-aware training reduce the loss. The commands below belong to PyTorch and ONNX, which this site cannot run, so they are shown read-only:

Python
# Read-only: typical PyTorch / ONNX deployment steps.
import torch

model.eval()
quantized = torch.ao.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8)       # int8 weights for Linear layers

example = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, example, "model.onnx")        # portable format for ONNX Runtime

Monitoring after launch

Deployment is not finished at launch, because the world keeps changing: new camera firmware, a new store layout, the winter sun. Plan to watch three things. First, input statistics (brightness, size, the share of blank frames), which need no labels. Second, prediction statistics such as the share of each class and the average confidence. Third, real accuracy on a small, regularly sampled and human-labeled slice of live data.

A simple drift check compares a recent batch to a reference period. The demo reports each day's average brightness and a z-score against the reference. The average of 200 images has a standard error of reference_std / sqrt(200), so a z-score beyond about 4 is a strong sign that something changed.

Python A brightness drift alarm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The alarm stays quiet for the first five days, then fires on day 6 when the average is 106.9 against a reference of about 120. A common rule of thumb reads a population stability index below 0.1 as stable and above 0.25 as a major shift; the darker batch scores about 1.0. When an alert fires, do not retrain blindly. Look at sample images, find the cause, and then fix the camera, update the preprocessing or collect new data.

flowchart LR
    A["Train and<br/>evaluate"] --> B["Shrink and<br/>benchmark on device"]
    B --> C["Deploy with<br/>logging and a rollback"]
    C --> D["Monitor inputs,<br/>outputs, sampled accuracy"]
    D -->|"drift or errors"| E["Investigate and<br/>collect data"]
    E --> A

Deployment is a loop. Monitoring feeds the next round of data collection and training.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A face model is trained mostly on one group of people and tested only on its overall accuracy. What is the most likely hidden problem?

    A small group contributes little to the average. Reporting metrics per group (slices) reveals the gap.
  2. 2. Which step best protects privacy before images are stored?

    Anonymizing the sensitive regions and removing embedded metadata reduces what a leak or misuse could reveal. Renaming or converting files changes nothing about the content.
  3. 3. What does FGSM do?

    It uses the sign of the loss gradient with respect to the pixels, scaled by a small eps, so a nearly invisible change can flip the prediction.
  4. 4. Why is per-image standardization (subtract the mean, divide by the standard deviation) useful?

    If every pixel is scaled and shifted by the same amounts, the standardized values are identical. It does nothing for blur, rotation or glare.
  5. 5. After quantizing a model to int8, what should you do before shipping it?

    Quantization saves size and often time, but it can cost accuracy, and the real speedup depends on the hardware, so both must be measured.

Summary

  • Models inherit the gaps in their data. Evaluate by group, balance or augment weak groups, and ask who is hurt by a mistake.
  • Images carry private information. Collect the minimum, process on the device when possible, anonymize early and delete on request.
  • Adversarial examples (FGSM and its relatives) show that tiny, deliberate changes can flip predictions. Do not rely on a vision model alone for high-stakes decisions.
  • Ordinary lighting and camera variation causes most field failures. Use augmentation and normalization, and test on shifted data.
  • Measure latency (p95), size and throughput on the target device. Quantization cuts weight storage 4x, but always re-evaluate afterward.
  • Monitor input statistics, prediction statistics and sampled accuracy. Alert on drift, investigate before retraining.

Next up: put the whole toolbox to work in Project: Document Scanner.