Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 54 ยท Performance and Production

Testing Computer Vision Code

Learn to test image-processing code with synthetic images, golden files, tolerance comparisons and fixed random seeds, using pytest-style asserts on NumPy arrays.

Image code is hard to test the "normal" way. A function that returns a 480 by 640 array of numbers has no single right answer you can type into an assert, and eyeballing a window after every change does not scale. Still, vision code breaks like any other code: someone swaps an operation, a rounding mode changes in a new library version, a refactor flips rows and columns. Tests are how you notice.

The trick is to test small, known inputs and to compare with a tolerance instead of demanding exact equality. This lesson shows four tools: synthetic images whose correct answer you can compute by hand, golden images for regression checks, tolerance-based comparison, and determinism (fixed seeds, fixed dtypes). The examples use plain NumPy arrays, which is what OpenCV images really are. The cv2 calls in read-only blocks cannot run on this site, so the live code tests small functions we write ourselves. The testing ideas are identical for real cv2 functions.

Four tools for vision tests

Synthetic images Tiny inputs with an answer you know.
Golden images Saved good output to compare against.
Tolerances Close enough beats exactly equal.
Determinism Fixed seeds, fixed dtypes.
1 / 2

Synthetic images: inputs with a known answer

A synthetic image is one you build in code: a black canvas with a white square, a horizontal gradient, a checkerboard. Because you made it, you know exactly what a correct filter should do to it. A threshold applied to a gradient must split it at a predictable column. A blur applied to a flat gray image must change nothing. A vertical flip applied twice must give back the original.

These are properties, and they are the best vision tests because they do not depend on any stored file.

Python Test a threshold function on a gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice there is no image file anywhere. The test is fast, it runs on any machine, and when it fails the message points at a specific property, not at "the pictures look different".

Good synthetic shapes to keep in a helper module: an all-black image, an all-white image, a single bright pixel (shows how filters spread it), a filled rectangle at a known position (for contour and bounding-box tests), and a gradient (for thresholds and histograms).

Tip: A single bright pixel on a black background is called an impulse. Filtering it shows you the filter kernel itself, which makes it a great test for blurs and convolutions.

Pytest-style asserts on arrays

Plain assert a == b fails on arrays, because comparing two arrays gives an array of booleans, and Python cannot turn that into one True or False. Use NumPy's helpers instead.

Helper Use it for
np.array_equal(a, b) Exact equality, including shape. Returns a bool.
np.allclose(a, b, atol=...) Float arrays that should match within a tolerance.
np.testing.assert_array_equal(a, b) Same as array_equal, but a failure prints where the arrays differ.
np.testing.assert_allclose(a, b, atol=...) Tolerance check with a readable failure report.

In a real test suite these live in functions named test_something() and pytest discovers them automatically. In the sandbox below we run the same functions by hand so you can see a failure message.

Python Asserts with readable failures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The third test fails on purpose. Read the report: NumPy tells you how many elements mismatched and by how much, which is far more useful than a bare AssertionError.

Tolerances: close enough is the goal

Exact pixel equality is a trap. Different OpenCV builds may use different SIMD paths, a resize may round 127.5 up on one version and down on another, and JPEG compression changes pixels by design. A test that demands bit-for-bit equality will fail for reasons nobody cares about.

Instead, measure the difference and set a limit. Three numbers cover most needs:

  • Max absolute difference: the worst single pixel. Good for strict checks ("no pixel off by more than 1").
  • Mean absolute error (MAE): the average difference. Good for "overall close".
  • PSNR (peak signal-to-noise ratio): 10 * log10(255^2 / MSE) in decibels, where MSE is the mean squared error. Higher is better; identical images give infinity. Above about 40 dB the images are visually indistinguishable.

A critical detail: subtracting uint8 arrays wraps around (3 - 5 becomes 254), so convert to a wider type before comparing. This is one of the traps in Data Types, Ranges and Saturation.

Python Comparing images with a tolerance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The nudged image passes (one level of change is invisible and harmless), while the mirrored image fails by a wide margin. Pick tolerances from experience: run your function on the new OpenCV version, look at the numbers, and set the limit just above normal variation.

Golden images: regression tests for the big picture

A golden image is a saved correct output. The test runs the pipeline on a fixed input and compares the result with the stored file. If someone changes the pipeline and the picture shifts, the test fails.

Golden images catch what property tests miss (the overall look), but they have costs: binary files in your repository, brittle comparisons, and a temptation to regenerate them whenever a test fails without understanding why. Follow these rules:

  1. Use lossless formats (PNG, or NumPy's .npy). Never JPEG, which changes pixels on save.
  2. Keep goldens small. A 64 by 64 synthetic input is plenty.
  3. Compare with a tolerance, not equality.
  4. Make regenerating a golden an explicit, reviewed action, for example a --update-golden flag.
  5. On failure, save the actual output and a difference image next to the test so a human can look.

A typical pytest test with real OpenCV looks like this. It cannot run here, so treat it as a read-only template:

Python
import cv2
import numpy as np
from pathlib import Path

GOLDEN = Path(__file__).parent / "golden" / "document_scan.png"

def test_scanner_output_matches_golden():
    page = cv2.imread(str(Path(__file__).parent / "inputs" / "page.png"))
    result = scan_document(page)                       # your function

    expected = cv2.imread(str(GOLDEN), cv2.IMREAD_GRAYSCALE)
    assert expected is not None, "golden file missing"
    assert result.shape == expected.shape

    diff = cv2.absdiff(result, expected)
    assert diff.mean() < 1.0
    assert diff.max() <= 40

Here cv2.absdiff does the safe subtraction for you: it takes the absolute difference with saturation, so there is no wrap-around.

The live stand-in below stores a "golden" in memory and uses a hash of its bytes for a strict check and a tolerance check for the tolerant one.

Python A golden image check, in memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A byte hash is a fine fast check for pure NumPy code you control, but the tolerance check is the one to use whenever the pipeline touches OpenCV, codecs or floating point.

Determinism: make tests repeatable

A flaky test is worse than no test. Vision code has three common sources of flakiness:

  • Random data. Noise for a denoising test must come from a seeded generator: rng = np.random.default_rng(42). The same seed gives the same noise on every machine. Never call np.random.rand() unseeded in a test.
  • Algorithms with internal randomness. RANSAC in cv2.findHomography samples random point sets, so results can vary a little between runs. Test properties ("at least 80% inliers", "the matrix maps point A near point B") with tolerances, not exact matrix values. Many OpenCV functions can be seeded with cv2.setRNGSeed(0).
  • Hidden type changes. A float32 pipeline fed a float64 array may give slightly different answers. Assert dtypes at the boundaries.
Python Seeded noise and a denoising test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the shape of the final assertion. We did not say "the output equals this exact array". We said "the filter reduced the error by at least half", which stays true across library versions and platforms.

A small testing checklist

For each image function you write, aim for these tests, roughly in this order:

  1. Shape and dtype on a tiny input.
  2. Identity cases: a flat image, or the identity kernel, returns the input.
  3. Impulse or edge case: one bright pixel, or a one-pixel-wide image, to catch off-by-one errors in borders.
  4. Known geometry: a rectangle at known coordinates yields the expected bounding box, remembering that OpenCV points are (x, y) while array indexes are [row, column] (see Common Mistakes and Debugging).
  5. One golden image for the whole pipeline, compared with a tolerance.

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should you convert uint8 images to a wider type such as int16 before subtracting them in a test?

    NumPy uint8 arithmetic is modular. A small negative difference becomes a huge positive number, so you must widen first, or use cv2.absdiff, which saturates safely.
  2. 2. What is the main advantage of a synthetic test image over a photograph?

    Because you built the image in code, you can compute or reason out what a correct filter should produce, such as a split at an exact column.
  3. 3. Which format is suitable for golden images?

    Lossy codecs change pixel values when they save, so the golden would differ from what your code produced. PNG and NumPy files store the exact values.
  4. 4. A test uses cv2.findHomography, which relies on RANSAC. What is the best way to assert on its result?

    RANSAC samples random subsets, so exact matrix values can vary slightly. Properties and tolerances stay valid while the exact numbers move.

Summary

  • Build small synthetic images with a known correct answer and test properties: shape, dtype, values, idempotence, symmetry.
  • Use np.array_equal, np.allclose and np.testing.assert_* instead of == on arrays.
  • Compare images with a tolerance: max absolute difference, mean error or PSNR, always after widening from uint8 (or with cv2.absdiff).
  • Keep golden images small and lossless, compare with a tolerance, and regenerate them only deliberately.
  • Make tests deterministic: seeded generators, asserted dtypes, and property checks for randomized algorithms.

Next, put everything together in a real build: Project: Document Scanner.