Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 41 ยท CI/CD for ML

Testing ML Code

Write unit tests for feature code, shape and sanity tests for models, and behavioral tests (invariance, direction, minimum functionality) that run in CI.

You cannot unit-test accuracy the way you test add(a, b). A model is statistical: it is right most of the time and wrong some of the time, and both are normal. But a surprising amount of an ML project is ordinary code (cleaning, feature building, metric calculation, request parsing), and ordinary code has exact right answers. Test that code hard. Then add a second layer of tests that check the behavior of the model without demanding perfection.

This lesson covers three layers: unit tests for deterministic functions, sanity tests for training and prediction, and behavioral tests that express what the model must and must not do. All use Python's standard unittest module so they run anywhere; pytest offers the same ideas with shorter syntax, shown at the end.

Layer 1: unit tests for deterministic code

Anything that turns raw data into features should be a small pure function. Pure functions are easy to test: same input, same output.

Python Unit tests for feature code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Test the boundaries (19,999 versus 20,000), the empty and bad inputs (None, ""), and the happy path. These three cover most bugs in feature code.

If a function is hard to test, it usually does too much. Split "read file, clean, compute features, train" into four functions and test the middle two.

Layer 2: sanity tests for training and prediction

You cannot assert "accuracy is 0.93", but you can assert things that must always be true:

  • the output has the right shape and type;
  • probabilities are between 0 and 1 and sum to 1;
  • the model can overfit a tiny dataset (if it cannot memorize eight rows, something is broken);
  • training is deterministic with a fixed seed;
  • the model beats a trivial baseline.
Python Sanity tests for a model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each test is cheap (a fraction of a second) because it uses a small synthetic dataset. That is what makes it fit for every pull request.

Layer 3: behavioral tests

Behavioral testing asks "does the model act sensibly?" instead of "is it accurate?". Three patterns cover a lot of ground:

Pattern Idea Example
Invariance An irrelevant change must not change the prediction Changing a customer's name does not change the loan decision
Directional expectation A change in one feature must move the output in a known direction More years of experience must not lower predicted salary
Minimum functionality A hand-picked set of easy cases must be right An obviously fraudulent transaction is flagged

These tests encode domain knowledge. They often catch bugs that overall accuracy hides, such as a feature column that was shuffled or a sign that was flipped.

Python Behavioral tests: direction and invariance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tolerance (1_000) matters. Tree models are not perfectly monotonic, so a test demanding exact monotonicity would be flaky. Choose a tolerance that is small compared with the real effect but larger than harmless noise.

The same tests with pytest

In a real repository you will probably use pytest: plain functions, plain assert, and fixtures. This is read-only here because pytest is not part of the browser runtime.

Python
# tests/test_features.py
import pytest
from features import clean_age

@pytest.mark.parametrize("raw, expected", [("34", 34), (34.9, 34), ("abc", None), (400, None)])
def test_clean_age(raw, expected):
    assert clean_age(raw) == expected

Run it in CI with pytest -q tests/unit. A failing test returns a non-zero exit code, which is what stops the pipeline.

Where to put what

  • Every pull request: unit tests, shape and sanity tests, behavioral tests on a small model. Target a few minutes in total.
  • Nightly or pre-release: full training and the model quality gate.
  • Never skipped: tests for code that runs at serving time, because the training code and the serving code must compute features identically. Disagreement between them is called training-serving skew, and a shared tested function is the cure.

Common mistakes

  • Flaky tests from unseeded randomness. Seed everything or assert properties that hold for any seed.
  • Asserting an exact metric on a toy dataset. Assert ranges or "better than baseline" instead.
  • Testing only the model, not the glue. Request parsing and feature code break more often than the algorithm.
  • Huge fixtures. If a test needs a 2 GB dataset it will not run on every commit; build a tiny synthetic one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is assert accuracy == 0.93 a poor unit test?

    Exact metric values are fragile. Prefer ranges, baselines and behavioral properties.
  2. 2. An invariance test checks that

    For example, changing a customer name or ID should not change a credit decision.
  3. 3. Why test whether a model can overfit eight rows?

    Memorizing a tiny dataset is a cheap sanity check of the whole training path.
  4. 4. What is training-serving skew?

    Sharing one tested feature function between training and serving prevents it.

Summary

  • Test deterministic code (cleaning, features, metrics) with ordinary unit tests: happy path, boundaries and bad input.
  • Sanity-test models: shapes, probability ranges, determinism, overfitting a tiny set, beating a baseline.
  • Behavioral tests (invariance, direction, minimum functionality) encode domain knowledge without demanding a fixed accuracy.
  • Keep pull-request tests small and seeded; run heavy checks on a schedule.
  • Share feature code between training and serving to avoid skew.

Next: Data Validation in CI