Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Practice and Review

Testing Numerical Code

Test numerical Python reliably. Use np.testing.assert_allclose with sensible tolerances, known-solution tests, fixed seeds and property checks.

How do you know that a numerical routine is right? You cannot compare to an exact answer, because floating-point results carry small errors. You cannot rely on a quick look at the output, because a wrong answer often looks plausible. And you cannot treat randomness as an excuse, because a simulation that gives a different result on every run is impossible to debug.

Good numerical tests solve these three problems. They compare numbers with explicit tolerances, they check results against problems with known solutions, they fix random seeds, and they verify properties that must hold for every input. This lesson shows each technique with SciPy functions. The examples use plain Python functions and a tiny runner so that they work in the browser; on your own machine you would put the same functions in a file for pytest.

flowchart LR
    A["Numerical function"] --> B["Known-solution tests<br/>(exact answers)"]
    A --> C["Property tests<br/>(identities for random inputs)"]
    A --> D["Regression tests<br/>(stored reference values)"]
    B --> E["assert_allclose<br/>with rtol and atol"]
    C --> E
    D --> E
    E --> F["Pass or a readable failure"]

Three kinds of checks, one comparison tool. Seeds make the random inputs repeatable.

assert_allclose: the workhorse

numpy.testing.assert_allclose(actual, desired, rtol=1e-7, atol=0) raises an AssertionError unless every element satisfies |actual - desired| <= atol + rtol * |desired|. Note that the defaults differ from np.isclose: the default rtol is 1e-7 and the default atol is 0. Its failure message is detailed: it shows which elements are wrong and by how much, so you can fix the problem without printing anything by hand.

Prefer it over assert np.allclose(...). The plain assert only tells you "False", while assert_allclose tells you the numbers. Another detail: NaN values compare equal by default (equal_nan=True), which is convenient when NaN is the expected result.

Python What assert_allclose tells you when a test fails

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing tolerances

A tolerance is a statement about how accurate you expect the code to be, so choose it from the algorithm, not from trial and error:

  • Direct linear algebra on well-conditioned matrices: about 1e-10 relative, often much better.
  • Numerical integration with quad: use its own error estimate, or a value like 1e-8.
  • ODE solvers: the error is larger than the requested rtol because errors accumulate. If you asked for rtol=1e-9, test at about 1e-7.
  • Optimizers: the solution is only as good as the solver tolerance (xtol, tol), and flat minima make the argument inaccurate even when the value is excellent. Compare function values with a tighter tolerance than arguments.
  • Statistical estimates from random samples: tolerance comes from the standard error, such as four standard errors, not from machine epsilon.

A tolerance that is too tight makes tests that fail on another CPU or library version. One that is too loose lets bugs through. If in doubt, measure the actual error once, then set the tolerance to about ten times that.

Known-solution tests

The most convincing test compares your code against a problem whose answer is known on paper. Mathematics supplies many of them:

  • integral of exp(-x**2) over the whole real line equals sqrt(pi).
  • The ODE y' = -2y, y(0) = 1 has the solution exp(-2t).
  • The root of x**2 - 2 is sqrt(2).
  • A curve fit to data generated from known parameters plus small noise should recover those parameters.

This last idea is called a round-trip or parameter recovery test, and it is the standard way to validate fitting code: generate synthetic data from known parameters, fit, and check that you get them back within a tolerance based on the noise.

Python A small test suite built on known solutions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last test fails on purpose, to show that a failing test prints a useful reason. Remove it when you adapt the suite. Notice the pattern: every function name starts with test_, each one does a single check, and the runner collects them by name. That is exactly how pytest finds tests.

Include edge cases in a suite: an empty array, a single element, a matrix that is exactly singular, a value exactly at an interval boundary. Numerical bugs hide in the corners far more than in typical inputs.

Fixed seeds

Any test that uses random numbers must use a seeded generator: rng = np.random.default_rng(123). The same seed gives the same stream on every run, so a failure can be reproduced. The old global np.random.seed(...) works, but it shares hidden state across all code; a local Generator is safer, because tests cannot affect each other.

Seeds do not make a test meaningful by themselves. A test that passes only for seed 123 and fails for seed 124 is telling you something: either the code is fragile or the tolerance is too tight. Run important simulations with a handful of seeds while developing, then pin one in the test.

When the result is a random estimate (a Monte Carlo mean, a bootstrap interval), do not compare to the exact value with a tiny tolerance. Compare within a few standard errors: for the mean of 10,000 draws with standard deviation 2, the standard error is 2 / sqrt(10000) = 0.02, so a tolerance of 0.08 is four standard errors.

Property checks

Often you do not know the exact answer but you know a property the answer must satisfy for every input: an identity, a symmetry, an invariant. Properties are wonderful because you can check them on many random inputs.

Function Property to check
linalg.solve(A, b) A @ x reproduces b
linalg.qr(A) Q.T @ Q is the identity and Q @ R equals A
fft.fft the inverse transform recovers the input; Parseval's identity holds
dist.ppf and dist.cdf cdf(ppf(q)) == q
any CDF it never decreases
an optimizer the result is no worse than the starting point
Python Property checks on random inputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first six checks pass and the intentionally wrong one fails, which is what a healthy test suite looks like when you break a rule on purpose. Test the tests: if a check can never fail, it proves nothing.

Regression tests and pytest

Once your function is correct, you can freeze its behaviour with a regression test: store a few reference outputs (computed by a trusted method or an earlier version) and compare future runs to them. Regression tests do not prove correctness, but they alert you when a refactor, a library upgrade or a new platform changes the numbers.

In a real project the tests live in a file such as tests/test_numerics.py, and pytest runs them. The parametrize decorator runs one test over several inputs. This block is read-only here, since pytest is not part of the browser runner:

Python
import numpy as np
import pytest
from numpy.testing import assert_allclose
from scipy import special

@pytest.mark.parametrize("n, expected", [(0, 1), (5, 120), (10, 3628800)])
def test_factorial(n, expected):
    assert_allclose(special.factorial(n), expected, rtol=1e-12)

def test_singular_matrix_raises():
    from scipy import linalg
    with pytest.raises(linalg.LinAlgError):
        linalg.inv(np.zeros((2, 2)))

Use pytest.raises to confirm that bad input fails loudly, and keep each test small so that a failure points at one cause.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What are the default tolerances of numpy.testing.assert_allclose?

    assert_allclose defaults to rtol=1e-7 and atol=0. np.isclose has different defaults (1e-5 and 1e-8), a common source of confusion. With atol=0, comparing to an expected zero requires you to pass atol.
  2. 2. Why is assert_allclose(a, b) better than assert np.allclose(a, b) in a test?

    np.allclose only returns True or False, so the failure message is just 'assert False'. assert_allclose prints the mismatch details.
  3. 3. A curve-fitting function is tested by generating data from known parameters, fitting and comparing. What is this technique called?

    The synthetic data have a known truth, so the fit must recover the parameters within a tolerance set by the noise.
  4. 4. Why should tests of random simulations use default_rng(seed)?

    A seeded generator replays the same stream. It does not remove statistical error from an estimate, so compare estimates within a few standard errors.
  5. 5. Which of these is a property check for scipy.linalg.qr?

    Properties are identities that must hold for any input, and QR has two: it reconstructs A and Q has orthonormal columns.

Summary

  • Use np.testing.assert_allclose with explicit rtol and atol; remember its defaults (rtol=1e-7, atol=0) and give an atol when comparing to zero.
  • Pick tolerances from the algorithm's real accuracy: looser for ODE solvers and optimizers, much tighter for direct linear algebra, and standard errors for random estimates.
  • Known-solution tests (exact integrals, closed-form ODEs, parameter recovery) are the most convincing.
  • Seed every generator with np.random.default_rng(seed), and compare Monte Carlo results within a few standard errors.
  • Property checks (reconstruction, orthogonality, inverse round trips, monotone CDFs) test many random inputs at once.
  • Add regression tests to freeze behaviour, and pytest.raises to check failure modes. Break a rule on purpose to confirm that a test can fail.

Next: practise everything in Exercises with Solutions.