Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 68 ยท Linear Algebra and Numerical Methods

Floating-Point Error and Conditioning

Understand round-off, truncation error, catastrophic cancellation and condition numbers, learn stable formulas, and see how automatic differentiation works.

A computer does not store real numbers. It stores a finite set of binary fractions, and every arithmetic operation rounds its result to the nearest one. Most of the time, this is harmless. But when it matters, it matters a lot: a bridge simulation, a trajectory, or a model fit can silently lose all its significant digits. Knowing where numerical error comes from lets you choose formulas and methods that keep it small, and lets you recognize a wrong answer that looks plausible.

This lesson separates three ideas that beginners often blur together: round-off error (the computer's arithmetic), truncation error (the mathematical approximation, like cutting off a Taylor series), and conditioning (how sensitive the problem itself is). It builds on The Derivative as a Rate of Change, where we saw error versus step size, and on Matrices and Linear Systems.

How doubles work

Python floats are IEEE 754 double-precision numbers: 53 significant bits, about 15 to 16 decimal digits. The spacing between 1 and the next number is the machine epsilon, about 2.22e-16. The relative error from rounding one operation is at most half of that.

Python What doubles can and cannot do

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

0.1 has no exact binary form, just as 1/3 has no exact decimal form, so 0.1 + 0.2 is 0.30000000000000004. Two lessons follow. First, never compare floats with ==; use math.isclose or np.isclose with a tolerance. Second, adding a tiny number to a huge one can lose the tiny one entirely: 1e16 + 1 equals 1e16 because the spacing between doubles near 1e16 is 2. Single precision (float32) runs out at 16,777,216, which is why careless use of it in accumulators goes wrong.

Truncation versus round-off

When you replace a derivative by (f(x+h) - f(x)) / h, you commit a truncation error that shrinks as h shrinks (proportional to h for the forward difference, and to h**2 for the central difference). But the subtraction f(x+h) - f(x) loses digits as h gets small, because the two values become nearly identical. That round-off error grows like eps / h. The total error has a U shape, with a best h in the middle.

flowchart LR
    A["Large h"] --> B["Truncation error dominates<br/>error falls as h falls"]
    C["Optimal h"] --> D["Smallest total error"]
    E["Tiny h"] --> F["Round-off dominates<br/>error rises as h falls"]
    B --> C --> E

Making h smaller helps only until round-off takes over.

Python The U-shaped error curve of finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The forward-difference error falls by a factor of 10 per row until h is around 1e-8, then rises again, finally reaching a complete loss of accuracy near 1e-16. The central difference is far better (best error around 1e-10) and reaches its optimum near h = 1e-5. A rule of thumb: the best step is about eps**(1/2) for forward differences and eps**(1/3) for central ones, times the scale of x.

Catastrophic cancellation

Subtracting two nearly equal numbers wipes out their leading digits and leaves mostly rounding noise. The subtraction itself is exact; the damage is that earlier rounding errors, which were tiny relative to the numbers, are now large relative to the difference. This is catastrophic cancellation, and the cure is nearly always algebra: rewrite the formula so the subtraction disappears.

Three classic cases:

  • (1 - cos x) / x**2 for small x. Using the half-angle identity, 1 - cos x = 2 sin(x/2)**2, the stable form has no subtraction.
  • The quadratic formula when b**2 is much larger than 4ac. One root, (-b + sqrt(b*b - 4ac)) / 2a, subtracts nearly equal numbers. Compute the other root first, q = -(b + sign(b) sqrt(b*b - 4ac)) / 2, and then the small one as c / q (the product of roots is c/a).
  • The textbook variance mean(x**2) - mean(x)**2 for data with a large mean. The two-pass formula mean((x - mean(x))**2) is stable.
Python Naive formulas against stable ones

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By x = 1e-8 the naive expression returns 0, not 0.5: a 100% error from a formula that looks innocent. The naive quadratic root is wrong in its second digit and leaves a residual of about 0.25, while the stable one gives -1e-8 with a residual near 1e-16. The naive variance is not just inaccurate; it can come out as a negative number, which is impossible for a variance.

Stable formulas rarely look better on paper. When a formula has a subtraction, ask: can the two terms be nearly equal? If so, look for an identity, a series for small arguments (np.expm1, np.log1p and np.hypot exist for this reason), or a different algorithm.

Conditioning: sensitivity of the problem

Some problems are hard no matter how carefully you compute. The condition number measures how much a relative change in the input can change the output. For a function, kappa(x) = |x f'(x) / f(x)|. For solving A x = b, kappa(A) = ||A|| ||A^-1||, available as np.linalg.cond. A rough rule: you lose about log10(kappa) decimal digits of accuracy.

Do not confuse the two ideas. A well-conditioned problem solved with an unstable algorithm gives a bad answer (the cancellation examples above). An ill-conditioned problem gives bad answers with any algorithm, because the input error alone is amplified, and even the rounding of the input data is enough to ruin it.

Python Condition numbers of Hilbert matrices and functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Hilbert matrix is the standard ill-conditioned example. Its condition number grows from about 1.6e4 at size 4 to about 1.8e16 at size 12, which is the reciprocal of epsilon: no digits are left. The reported error in the solution tracks this growth, from about 1e-12 at size 4 to about 0.4 at size 12 (the solution is off by 40%). The function examples show the same idea locally: x - 1 near 1 has a huge condition number (a tiny relative change in x is a big relative change in x - 1), while the square root has condition number 0.5 and is harmless.

Automatic differentiation: a better derivative

The finite-difference dilemma above suggests a question: can we get derivatives without the truncation error and without the cancellation? Yes. Automatic differentiation (AD) applies the chain rule mechanically, as the program runs. In forward mode, every number carries a pair (value, derivative); each operation updates both by its own calculus rule (the sum, product and quotient rules from Differentiation Rules). The result is the exact derivative of the program, up to ordinary round-off. This is not symbolic algebra, and it is not finite differences. It is the engine inside machine-learning libraries.

Python A tiny forward-mode automatic differentiator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The automatic derivative matches the central difference to about 9 digits here, and unlike the finite difference it does not depend on a step size. Real libraries (JAX, PyTorch, autograd) use the same idea, in "reverse mode" for functions with many inputs. You will use it for gradients in Project: Gradient Descent from Calculus.

A checklist

  • Compare floats with a tolerance, never ==.
  • Look for subtractions of nearly equal quantities and rewrite them.
  • Pick step sizes near eps**(1/2) (forward) or eps**(1/3) (central), scaled to your variable.
  • Estimate np.linalg.cond before trusting a linear solve; do not form inv(A) explicitly.
  • Use float64 unless you have a reason; use math.fsum or compensated sums for long sums.
  • Test with a case whose answer you know.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does (1e16 + 1) - 1e16 give 0 in double precision?

    Doubles near 1e16 are 2 apart, so adding 1 rounds back to the same value.
  2. 2. In a forward-difference derivative, what happens as h becomes extremely small?

    Truncation error shrinks with h, but the subtraction of nearly equal values amplifies rounding error roughly like eps/h, producing a U-shaped total error.
  3. 3. A condition number of about 1e10 for matrix A means roughly what for the solution of Ax = b in double precision?

    A rule of thumb is that you lose about log10(cond) digits, so only about 6 correct digits remain.
  4. 4. How does automatic differentiation differ from a finite difference?

    AD applies the chain rule to each elementary operation as the program runs, giving the derivative of the program up to ordinary round-off.

Summary

  • Doubles carry about 16 significant digits (epsilon is about 2.2e-16); compare with tolerances, not ==.
  • Total error in numerical differentiation is truncation plus round-off, so there is an optimal step size.
  • Catastrophic cancellation comes from subtracting nearly equal numbers; fix it with algebraic rewriting and functions like expm1, log1p.
  • The condition number measures a problem's sensitivity; an ill-conditioned problem is hard for every algorithm.
  • Automatic differentiation gives exact derivatives of a program by carrying (value, derivative) pairs.

Next: Probability and Random Variables.