Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท Practice and Review

Performance, Accuracy and Common Pitfalls

Make numerical Python fast and trustworthy. Vectorize loops, pick dtypes, understand floating-point error and tolerances, and know when SciPy is not enough.

Numerical code can fail in two different ways. It can be too slow, or it can be quietly wrong. Slow code is annoying but honest: you notice it. Wrong code prints a confident number and moves on. This lesson collects the habits that guard against both: vectorize loops, choose data types on purpose, understand why floating-point arithmetic loses digits, compare numbers with tolerances, and pick the right matrix storage.

Everything here applies to the whole library, so you will see echoes of earlier lessons such as Matrices, Solving Systems and Inverses (condition numbers) and Sparse Matrices (storage formats). The last section is honest about the limits of SciPy and points at tools you may reach for next.

flowchart TD
    A["Slow or suspicious result"] --> B{"Which one?"}
    B -->|"slow"| C["Python loop over array elements?<br/>Vectorize"]
    C --> D["Right algorithm and storage?<br/>(sparse, banded, factor once)"]
    D --> E["Still too slow?<br/>numba, JAX, parallel workers"]
    B -->|"wrong or unstable"| F["Check dtype and tolerances"]
    F --> G["Check conditioning<br/>and cancellation"]
    G --> H["Test against a known answer"]

A debugging map. Fix the algorithm and the data layout before reaching for heavier tools.

Vectorize, do not loop

NumPy and SciPy are fast because their inner loops run in compiled C or Fortran. A Python for loop over array elements throws that away: every element becomes a Python object, and each iteration pays interpreter overhead. The cure is to express the computation on whole arrays: comparisons produce boolean masks, masks select elements, and reductions such as np.dot or .sum() do the arithmetic.

Python Loop versus vectorized, and the size of dtypes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only relative comparisons are printed because exact timings differ from machine to machine. The vectorized version is typically one or two orders of magnitude faster.

Some practical rules:

  • Avoid for i in range(len(a)) over array data. Look for a ufunc (np.exp, np.where, np.minimum), a reduction with an axis argument or a SciPy routine.
  • Use broadcasting instead of building repeated copies (see NumPy Refresher for SciPy).
  • Avoid growing arrays with np.append inside a loop. Preallocate with np.empty or collect values in a list and convert once.
  • Many SciPy functions accept arrays and an axis argument, so one call can process thousands of rows. The statistics functions and scipy.stats.bootstrap rely on this.
  • Pass analytic gradients and Jacobians to optimizers and ODE solvers when you can; finite differences cost extra function calls.

Choosing dtypes

NumPy arrays have a fixed element type, and that choice has consequences.

  • float64 (the default) has about 15 to 16 significant digits. It is the right choice for almost all scientific work.
  • float32 halves memory and is common for images and neural networks, but gives only about 7 digits and many SciPy routines convert it back to float64 internally anyway.
  • Integer types wrap around silently on overflow (np.int8(127) + np.int8(1) is -128) and integer division behaves differently from float division. Convert early with astype(float) if you are about to do mathematics.
  • Mixing types upcasts: float32 array plus a Python float stays float32, but float32 plus a float64 array becomes float64.

Low precision also hurts sums. Adding a very large number of small values one at a time loses digits, because each addition rounds. NumPy's sum uses pairwise summation to reduce the damage, but cumsum and your own loops do not.

Python float32 accumulation error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The straightforward float32 running total ends near 1,087,937, almost 9 percent too high. The pairwise sum() is close, and a float64 accumulator is nearly exact (the tiny difference comes from the float32 inputs, which are not exactly 0.1). When precision matters, accumulate in a wider type with the dtype= argument.

Floating-point error and cancellation

Computers store most real numbers approximately. 0.1 + 0.2 == 0.3 is False, and that is correct behaviour, not a bug. Single operations are accurate to about one part in 10 quadrillion (1e-16) for float64, which is harmless. The damage comes from specific patterns:

  • Catastrophic cancellation: subtracting two nearly equal numbers wipes out the leading digits that agreed, leaving mostly rounding noise. 1 - cos(x) for tiny x is the classic case.
  • Absorption: adding a tiny number to a huge one changes nothing (1e16 + 1.0 == 1e16).
  • Overflow and underflow: exp(1000) is infinity, exp(-1000) is zero.

The cure is usually an algebraically equivalent formula that avoids the dangerous step, and SciPy and NumPy supply many of them: np.log1p(x) for log(1 + x), np.expm1(x) for exp(x) - 1, scipy.special.logsumexp for log(sum(exp(z))), and math.fsum for exactly rounded sums.

Python Cancellation and its cures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the last example the textbook quadratic formula gets the first digit wrong (-7.45e-09 instead of -1e-08) because -b + d subtracts two nearly equal numbers. Computing the large root first and deriving the small one from root1 * root2 = c / a avoids the subtraction.

Conditioning is different from rounding

Even a perfect algorithm cannot rescue an ill-conditioned problem: one where small changes in the input cause large changes in the answer. The condition number measures that. For a linear system, the relative error of the solution can be about the condition number times 1e-16. The Hilbert matrix, a standard troublemaker, shows this: its condition number explodes with size.

Python Condition number versus accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error grows in step with the condition number, and at n = 12 (condition number near 1e16) not a single digit of the answer can be trusted. In normal use solve prints a LinAlgWarning for such matrices. Treat that warning as an error. Check np.linalg.cond(A) when results look odd, and consider regularization or a different formulation.

Compare with tolerances, never with ==

Because results carry small errors, test numbers with a tolerance. np.isclose(a, b, rtol=1e-5, atol=1e-8) accepts values when |a - b| <= atol + rtol * |b|.

  • rtol is relative: it scales with the size of the numbers.
  • atol is absolute: it matters when the expected value is zero. Comparing 1e-10 with 0.0 using only rtol always fails, because the relative error to zero is infinite. The default atol=1e-8 accepts it.
  • Choose tolerances from the problem. Values from an optimizer are good to about the optimizer's tol (often 1e-6 to 1e-8), not to machine precision.

Testing Numerical Code goes deeper into tolerances in automated tests.

Sparse versus dense

Sparse storage wins when most entries are zero, but it is not free. A CSR matrix stores for every non-zero a value (8 bytes) and a column index (4 bytes), plus one row pointer per row. A dense matrix stores 8 bytes for every entry. Setting those equal puts the break-even at a density around two thirds, and in practice sparse arithmetic is slower than dense BLAS long before that, because of indirect memory access.

Python Memory of sparse versus dense

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Remember also that the inverse of a sparse matrix is generally dense, so never write inv(A) @ b for a sparse A. Use spsolve, splu or an iterative solver, as in Sparse Linear Algebra and Graph Algorithms. Favour banded or structured solvers (solve_banded, solve_toeplitz) when your matrix has a pattern, and factor once with lu_factor or splu when you solve with many right-hand sides.

A checklist of common pitfalls

  • Inverting matrices. Use solve, never inv(A) @ b. It is faster and more accurate.
  • Integer arrays. Convert to float before doing mathematics; watch for integer division and overflow.
  • Ignoring success. Optimizers and solvers return a result object. Check res.success and res.message before using res.x.
  • Views versus copies. Slicing returns a view; assigning to it changes the original. Use .copy() when you need independence.
  • Unseeded randomness. Use np.random.default_rng(seed) so that results can be reproduced and tested.
  • Wrong axis. Reductions default to all axes (or axis 0, depending on the function). Pass axis= explicitly.
  • Poor starting guesses. Local optimizers return the nearest minimum. Try several starts or a global method for rough landscapes.
  • Over-tight tolerances. Demanding rtol=1e-14 from an iterative method may never converge. Ask for what you need.

When SciPy is not enough

SciPy is excellent for medium-sized problems on a single CPU core. Beyond that, there are tools worth knowing. They are shown here as read-only snippets, because they need packages that are not available in the browser runner.

Python
# numba: compile a hot Python loop to machine code (read-only)
from numba import njit

@njit
def count_hits(xs, threshold):
    total = 0
    for v in xs:
        if v > threshold:
            total += 1
    return total
Python
# JAX: automatic differentiation and GPU support (read-only)
import jax
import jax.numpy as jnp

def loss(w):
    return jnp.sum((w - 3.0) ** 2)

gradient = jax.grad(loss)    # exact gradient, no finite differences
Python
# Parallel workers: split independent jobs across CPU cores (read-only)
from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as pool:
    results = list(pool.map(run_one_simulation, range(100)))
  • Reach for numba when you have an unavoidable loop with simple numeric code.
  • Reach for JAX or PyTorch when you need gradients of complicated functions, or GPUs.
  • Reach for multiprocessing or a cluster tool when you have many independent runs (parameter sweeps, Monte Carlo batches). Many SciPy functions also accept a workers= argument, for example differential_evolution and scipy.fft.
  • Reach for specialized libraries (PETSc, FEniCS, scikit-learn, statsmodels) when a whole domain is involved.

In the browser runner, threads and subprocesses are not available, which is one more reason to learn the vectorized style first.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which change usually gives the biggest speed-up for numeric code on large arrays?

    Array operations run in compiled code. A Python loop pays interpreter overhead for every element.
  2. 2. What is the problem with computing (1 - cos(x)) / x**2 for x = 1e-8 directly?

    cos(1e-8) rounds to exactly 1.0, so 1 - cos(x) is 0 instead of about 5e-17. Rewriting with 2*sin(x/2)**2 avoids the subtraction.
  3. 3. np.isclose(1e-10, 0.0, rtol=1e-5, atol=0) returns False. Why?

    With atol=0 the allowed difference is rtol times |0|, which is 0. Add a small atol when the expected value can be zero.
  4. 4. When is a dense array preferable to a CSR matrix?

    CSR stores an index with every value, so a mostly full matrix takes more memory and runs slower than a dense array with optimized BLAS.
  5. 5. A solver returns a result with success=False. What should you do?

    success=False means the stopping criteria were not met. The message tells you why, for example too many iterations or a precision problem.

Summary

  • Vectorize: express work on whole arrays or use axis arguments instead of looping over elements in Python.
  • Pick dtypes deliberately. float64 is the default for good reason; integers can overflow, and low-precision accumulation drifts, so accumulate in a wider type.
  • Floating-point errors come from cancellation, absorption and overflow. Use stable forms such as log1p, expm1, logsumexp and math.fsum.
  • Condition number measures sensitivity of the problem itself; no algorithm can beat it.
  • Compare floats with isclose using both rtol and atol, chosen from the accuracy you actually have.
  • Sparse storage pays off only for mostly-zero matrices, and products or inverses can fill in.
  • When SciPy is not enough, consider numba, JAX, or parallel workers (shown read-only here).

Next: learn to prove your numerical code is right in Testing Numerical Code.