Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 47 Β· Numerical Foundations

Floating Point Numbers and Errors

Understand IEEE 754 doubles, machine epsilon, overflow, catastrophic cancellation and safe summation, with runnable examples on the quadratic formula and long sums.

Every number in a simulation lives in 64 bits. That is plenty for ordinary work: roughly 16 significant decimal digits, a range from about 1e-308 to 1e308. But "plenty" is not "exact", and a few specific habits turn the small gap into disasters. This lesson explains how doubles work and the three traps you will meet most often: unequal equals, cancellation and long sums.

A 64-bit double: sign, exponent and fraction fields, and a number line whose gaps double each octave
A double is one sign bit, 11 exponent bits and 52 fraction bits. Gaps between representable numbers double with every power of two.

What you will learn

  • How a double is built, and what machine epsilon means.
  • Why the spacing of floats grows with the size of the number.
  • What catastrophic cancellation is and how to avoid it.
  • Safe ways to compare and sum numbers.

Key terms

IEEE 754: the standard that defines floating point formats used by nearly all hardware.

Machine epsilon (Ξ΅): the gap between 1.0 and the next larger double, 2^-52 β‰ˆ 2.22e-16.

Relative error: |computed βˆ’ true| / |true|; for a single rounding it is at most Ξ΅/2.

Cancellation: loss of significant digits when subtracting two nearly equal numbers.

Overflow / underflow: results too large or too small to represent.

How a double is stored

A double has three fields: 1 sign bit s, 11 exponent bits e and 52 fraction bits f. For normal numbers

value = (βˆ’1)^s Γ— (1 + f) Γ— 2^(e βˆ’ 1023)

where f is the fraction read as a binary number between 0 and 1. The leading 1 is implied, so 53 binary digits of precision fit in 52 stored bits. That is about 53 Γ— log10(2) β‰ˆ 15.95 decimal digits.

Two consequences follow:

  1. Precision is relative. The gap between neighbors is Ξ΅ Γ— 2^(exponent). Near 1 it is 2.2e-16; near 1e6 it is about 1.2e-10; near 1e16 it is 2. Adding 1 to 1e16 does nothing.
  2. The range is finite. The largest double is about 1.8e308 (beyond it you get inf), and the smallest normal one about 2.2e-308.
Python Inspecting doubles

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Round-off in single operations

For one addition, subtraction, multiplication or division, the computed result is the exact result rounded to the nearest double. Each such step carries a relative error of at most Ξ΅/2 β‰ˆ 1.1e-16. A chain of a million operations carries more, but usually the errors partly cancel, so the total drifts like a random walk rather than adding up straight.

Subtraction is the dangerous operation. If a and b are accurate to 16 digits and agree in their first 10 digits, then a βˆ’ b is accurate to only about 6 digits. This loss is catastrophic cancellation.

Cancellation in the quadratic formula

For xΒ² + bx + c = 0 with b = 10⁸ and c = 1, the roots are close to βˆ’10⁸ and βˆ’10⁻⁸. The textbook formula x = (βˆ’b Β± √(bΒ² βˆ’ 4c)) / 2 subtracts nearly equal numbers when computing the small root.

The cure is to compute the large root without cancellation, then use Vieta's relation x₁ xβ‚‚ = c for the other one.

Python A quadratic with cancellation

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive answer has lost almost every correct digit (it is off by about 25 percent), while the stable form is accurate to full precision. The lesson generalizes: look for subtractions of nearly equal numbers and rewrite the formula. Other classic examples are 1 βˆ’ cos x for small x (use 2 sinΒ²(x/2)) and √(x+1) βˆ’ √x for large x (multiply by the conjugate).

Sums of many numbers

Adding N numbers one at a time into a running total loses a little at each step, especially when the total becomes much larger than the next term. Three remedies:

  • math.fsum computes a correctly rounded sum.
  • np.sum uses pairwise summation, which has much smaller error than a plain loop.
  • Kahan (compensated) summation keeps a running correction term.
Python Adding 0.1 one million times

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loop is off by about 1e-6 in a total of 1e5, a relative error near 1e-11: still small, but tens of thousands of times the machine epsilon. In a long simulation such drift can matter.

Comparing floats

Because results carry rounding, test closeness instead of equality. math.isclose(a, b, rel_tol=1e-9, abs_tol=0.0) uses a relative tolerance; add abs_tol when comparing to zero. NumPy offers np.isclose and np.allclose for arrays.

Common misconceptions

"0.1 + 0.2 != 0.3 is a Python bug." It is the nature of binary floating point and appears in every language that uses IEEE doubles.

"Using float32 saves nothing." Single precision has only about 7 digits, so it magnifies every problem here. Physics code normally uses float64 unless memory or speed demand otherwise and the algorithm is known to tolerate it.

"Round-off is random, so errors average out." Often yes, but systematic bias (always rounding the same direction, adding tiny numbers to huge ones) can accumulate linearly.

Check your understanding

Quick check Β· 6 questions

of 6 answered You scored / 6 Β· perfect!

  1. 1. How many bits does a double use for the fraction (mantissa)?

    A double has 1 sign bit, 11 exponent bits and 52 fraction bits.
  2. 2. What is the gap between 1.0 and the next larger double?

    That gap is machine epsilon, 2^-52.
  3. 3. Why is 1e16 + 1 equal to 1e16 in double precision?

    Precision is relative: near 1e16 neighbors are 2 apart.
  4. 4. Which operation is the main source of catastrophic cancellation?

    Subtracting nearly equal numbers cancels the leading digits and exposes the rounding noise.
  5. 5. How can the small root of x^2 + 1e8 x + 1 = 0 be computed accurately?

    Vieta's formula gives the second root without a cancelling subtraction.
  6. 6. Which test is the right way to ask whether two computed floats agree?

    A tolerance accounts for rounding.

Summary

  • A double stores a sign, an 11-bit exponent and a 52-bit fraction: about 16 digits, relative precision Ξ΅ β‰ˆ 2.2e-16.
  • Gaps between floats grow with the size of the number.
  • Subtracting nearly equal numbers loses digits; rewrite the formula to avoid it.
  • Prefer fsum, np.sum or compensated sums for long sums.
  • Compare with tolerances, never with ==.

Next: Derivatives and Integrals on a Grid