Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 38 ยท Testing and CI

Coverage, tox and nox

Measure which lines your tests run with coverage.py, then test across Python versions with tox or nox. Includes a real coverage run and a tox matrix expander.

Two questions come up once you have tests. How much of my code do the tests actually run? and Does my package work on every Python version I claim to support? The first is answered by coverage.py, the second by a test matrix driven by tox or nox.

Coverage tells you where tests are missing, not whether the tests are good. Treat it as a flashlight for dark corners, not as a score to maximize. This lesson shows both tools: coverage runs for real below, while tox and nox need to start other Python interpreters, so they are shown as read-only configuration with a runnable stand-in for the matrix logic.

Measuring coverage

coverage.py records which lines run while your tests execute. The usual command-line workflow is:

Terminal
pip install coverage pytest
coverage run -m pytest
coverage report -m          # table with missing line numbers
coverage html               # browsable report in htmlcov/

You can also drive it from Python. This example measures a function that has three branches, but the "test" only exercises one:

Python Measure coverage with coverage.py

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table reads: Stmts is the number of statements, Miss how many never ran, Branch and BrPart count decision outcomes (a partly covered if is a partial branch), and Missing lists the line ranges that never executed. Here lines 4 to 7, the circle and error cases, were never tested.

Configuration and thresholds

Configure coverage in pyproject.toml:

TOML
[tool.coverage.run]
branch = true
source = ["myproject"]
omit = ["*/tests/*", "*/_version.py"]

[tool.coverage.report]
show_missing = true
skip_covered = true
fail_under = 90
exclude_also = [
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
]
  • branch = true measures decisions as well as lines; always turn it on.
  • source limits measurement to your package, so third-party code is not counted.
  • fail_under makes coverage report exit non-zero below the threshold, which lets CI enforce a minimum.
  • exclude_also removes lines that are not worth testing from the numbers.

With pytest, the pytest-cov plugin wraps the same engine: pytest --cov=myproject --cov-report=term-missing.

Chasing 100% invites tests that execute lines without checking anything. A test that calls a function and asserts nothing earns coverage and proves nothing. Use the report to find untested behavior, then write assertions about it.

Why a test matrix

Your code runs on your Python version, with your installed dependency versions, on your operating system. Users have other combinations. A matrix runs the test suite across several of them: Python 3.10 through 3.14, the oldest and newest allowed dependency versions, Linux and Windows. Each cell runs in its own isolated environment.

flowchart LR
    S["source tree"] --> B["build sdist and wheel"]
    B --> E1["env: py311"]
    B --> E2["env: py312"]
    B --> E3["env: py313"]
    E1 --> T1["install wheel + run tests"]
    E2 --> T2["install wheel + run tests"]
    E3 --> T3["install wheel + run tests"]

Each environment installs the built package, not the source folder, and runs the same tests.

tox

tox reads a configuration, creates a virtual environment for each target, installs your package and runs commands. The classic form is tox.ini; modern versions also accept a [tool.tox] table in pyproject.toml.

INI
[tox]
envlist = py{310,311,312,313}, lint, type
isolated_build = true

[testenv]
extras = test
commands = coverage run -m pytest {posargs}

[testenv:lint]
skip_install = true
deps = ruff
commands = ruff check .

[testenv:type]
deps = mypy
commands = mypy src

py{310,311,312,313} is a brace expansion that produces four environments at once. Run them all with tox, one with tox -e py312, in parallel with tox -p.

nox

nox has the same purpose, but the configuration is a Python file, noxfile.py. That makes loops, conditions and shared helpers natural:

Python
import nox

nox.options.default_venv_backend = "uv"

@nox.session(python=["3.10", "3.11", "3.12", "3.13"])
def tests(session):
    session.install(".[test]")
    session.run("pytest", *session.posargs)

@nox.session
def lint(session):
    session.install("ruff")
    session.run("ruff", "check", ".")

Run nox -l to list sessions and nox -s tests-3.12 to run one. Choose tox if you like declarative files, nox if you prefer plain Python. Both work with uv for fast environment creation.

Expanding a matrix

The environment list in tox.ini is a tiny language. This stand-in expands brace patterns the way tox does, then groups the results, which is also how CI matrices multiply:

Python Expand a tox-style environment list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Matrices grow multiplicatively, so be selective: test the oldest and newest Python you support in full, and run the middle versions only with the default dependencies. Add a job that installs the lowest allowed dependency versions (uv pip install --resolution lowest-direct), because it proves your declared floors are true.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the Missing column in a coverage report show?

    It lists the lines (or ranges) that never ran, so you know where to add tests.
  2. 2. Why enable branch coverage?

    Partial branches, such as an if that was never false, are invisible in line-only coverage.
  3. 3. What does py{310,311}-{lin,win} expand to in tox?

    Brace groups multiply: two Python versions times two platforms gives four combinations.
  4. 4. Which statement describes nox compared with tox?

    Both create isolated environments and run commands. Nox sessions are ordinary Python functions in noxfile.py.

Summary

  • coverage.py shows which lines and branches tests execute; enable branch = true and read the Missing column.
  • Use fail_under as a floor, not a goal; coverage cannot judge assertion quality.
  • tox and nox run your tests across Python versions and dependency sets in isolated environments built from the installed package.
  • tox is declarative (tox.ini); nox is a Python file; both can use uv.
  • Keep the matrix selective, and test the lowest declared dependency versions too.

Next: GitHub Actions for Python Projects