Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 39 ยท Best Practices

Testing Notebook Code

Test notebooks at three levels: pytest for extracted functions, asserts inside cells, and whole-notebook execution with nbmake or nbconvert. Includes CI and data checks.

"It ran on my machine" is not a test. Notebooks produce numbers and charts that people use to make decisions, so they deserve checks just like application code. The good news is that testing notebooks does not need a special universe: you test the functions the way you test any Python code, add a few assertions where the data matters, and run the whole notebook automatically to prove it still finishes.

None of the notebook tools mentioned here run on this site, so their commands are read-only. The runnable cells use plain asserts and the unittest module, which work in Python anywhere.

Three levels of testing

flowchart TD
  A["Unit tests: functions in modules"] --> B["Cell checks: assertions on data and results"]
  B --> C["Notebook tests: run the whole notebook"]

Move up the levels as the risk grows. Most of your confidence comes from the first two.

Level 1: unit tests for extracted functions

Once logic lives in a module (see moving code into modules), test it with pytest:

Python
# tests/test_cleaning.py
import pytest
from sales.cleaning import monthly_totals

def test_ignores_refunds():
    rows = [{"date": "2024-01-05", "amount": 10}, {"date": "2024-01-06", "amount": -3}]
    assert monthly_totals(rows) == {"2024-01": 10}

def test_empty_input():
    assert monthly_totals([]) == {}

@pytest.mark.parametrize("amount,expected", [(5, 5.0), (0, 0.0)])
def test_amounts(amount, expected):
    assert monthly_totals([{"date": "2024-03-01", "amount": amount}]).get("2024-03", 0.0) == expected
Terminal
pytest -q

Cover normal input, an empty input, one edge case, and each bug you have ever fixed. Tests that fail teach you more than tests that pass.

Python Unit tests with unittest

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three tests pass. unittest runs in a notebook cell like this, which is handy for quick checks, but real projects put tests in files and run them with pytest in the terminal or CI.

Level 2: assertions inside the notebook

Many bugs in analyses are not code bugs but data surprises: duplicate ids, negative prices, a column that arrives as text. Put checks right after loading and after major transformations:

Python
assert orders["order_id"].is_unique, "duplicate order ids"
assert (orders["amount"] >= 0).all(), "negative amounts"
assert orders["date"].between("2020-01-01", "2030-01-01").all()
assert len(orders) > 1000, f"suspiciously small: {len(orders)} rows"

A failing assertion stops the notebook with a message, which is much better than a quietly wrong chart. For richer checks use libraries such as pandera or Great Expectations to declare a schema once and validate it everywhere.

Python Data checks with clear messages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Returning a list of problems (rather than raising on the first) tells you everything wrong in one run. Inside a pipeline you would then assert not problems, problems.

Test results, not exact floats

Floating-point results and statistical outputs rarely match exactly. Compare with tolerances:

Python
import math
assert math.isclose(model_score, 0.82, abs_tol=0.02)
# or with pytest:
# assert model_score == pytest.approx(0.82, abs=0.02)

Fix random seeds so that the tolerance can be tight (see seeds and determinism). For charts, test the data that goes into them instead of the pixels.

Level 3: run the whole notebook

A notebook can break without any function changing: a column is renamed, a cell is deleted, an import is missing. Running it end to end catches that.

Terminal
# nbmake: a pytest plugin that executes notebooks
pip install pytest nbmake
pytest --nbmake notebooks/                  # fail if any cell raises
pytest --nbmake --nbmake-timeout=600 notebooks/report.ipynb

# or plain nbconvert
jupyter nbconvert --to notebook --execute --output /tmp/out.ipynb notebooks/report.ipynb

pytest --nbmake treats each notebook as a test. For parameterized notebooks, run them with a small sample of the data (see Papermill) so the check finishes in seconds. Another tool, nbval (pytest --nbval), compares the new outputs to the stored ones cell by cell, which suits tutorials with stable output but needs care with random or time-dependent cells (mark those with # NBVAL_IGNORE_OUTPUT).

Continuous integration

Put the checks in a workflow that runs on every pull request:

YAML
name: tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt pytest nbmake
      - run: pytest -q
      - run: pytest --nbmake notebooks/

Because CI builds a clean environment from your requirements file, it also proves that the project rebuilds from scratch.

Testing checklist

  • Pure functions in a module with pytest tests; every bug fix adds a test.
  • Data assertions after loading and after risky transformations.
  • Tolerances for floats; seeds for randomness.
  • The whole notebook executes in CI on a small data sample.
  • Tests are fast: slow tests are skipped, and then nobody runs them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the best place to test notebook logic with pytest?

    Extracted functions are importable, so a normal test runner can exercise them.
  2. 2. Why add assertions on data inside a notebook?

    Surprising data produces wrong results silently unless something checks it.
  3. 3. How should floating-point results be compared?

    Tiny rounding differences are normal; tolerances accept them while still catching real changes.
  4. 4. What does pytest --nbmake notebooks/ check?

    nbmake executes every notebook and fails the test if any cell raises.

Summary

  • Test at three levels: unit tests for functions, assertions on data, and whole-notebook execution.
  • Use tolerances for floats and fixed seeds for randomness.
  • nbmake or nbconvert --execute turns notebooks into tests; CI runs them on every change.
  • Keep tests fast and add one for every bug.

Next: Performance and Memory