Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 67 ยท Data Representation

Floating-Point Numbers and IEEE 754

See how computers store real numbers as sign, exponent and mantissa, why 0.1 + 0.2 is not 0.3, and how to decode float bits yourself with Python's struct.

Integers are exact but cannot hold 0.5 or 3.14159 or the mass of an electron. For real numbers, almost every computer uses the IEEE 754 floating-point standard. It stores a number in scientific notation, in binary, using a fixed number of bits. That makes a huge range possible (from about 10^-308 to 10^308 for a double) at the price of limited precision.

Understanding the format explains many "bugs" that are not bugs: why 0.1 + 0.2 is not 0.3, why adding a tiny number to a huge one does nothing, and why comparing floats with == is risky. In this lesson you will decode the bits of real floats with Python's struct module and see exactly where the errors come from.

Scientific notation in binary

In decimal, 6.02 x 10^23 has a significand (6.02) and an exponent (23). IEEE 754 does the same in base 2:

TEXT
value = (-1)^sign  x  1.fraction  x  2^(exponent - bias)

A number is stored as three fields packed side by side:

Format Total bits Sign Exponent Fraction Bias
Half (binary16) 16 1 5 10 15
Single (binary32) 32 1 8 23 127
Double (binary64) 64 1 11 52 1023
flowchart LR
  S["sign<br/>1 bit"] --- E["exponent<br/>8 bits (single)"] --- F["fraction<br/>23 bits (single)"]

A 32-bit float: one sign bit, an 8-bit biased exponent and a 23-bit fraction.

Three ideas make the format work:

  • Normalization. Every normal number is written with exactly one non-zero digit before the point. In binary that digit is always 1, so it is not stored. This "hidden bit" gives you 24 bits of precision for single and 53 for double while only storing 23 and 52.
  • Biased exponent. The exponent field is an unsigned number. Subtracting the bias (127 for single) gives the real exponent, which can be negative. This lets floats be compared as if they were integers (for positive values).
  • Sign-magnitude. The sign is a separate bit. So there is both +0.0 and -0.0.

Decoding a float by hand

Take -6.25. In binary, 6.25 is 110.01 (4 + 2 + 0.25). Normalize it: 1.1001 x 2^2. So:

  • sign = 1 (negative)
  • exponent = 2 + 127 = 129 = 10000001
  • fraction = 1001 followed by zeros (the leading 1 is hidden)

The 32 bits are 1 10000001 10010000000000000000000, which is 0xC0C80000.

You do not need to do this by hand. Python's struct module packs a float into the bytes the CPU would use. '>f' means big-endian single, '>d' big-endian double and '>e' half.

Python Look at the bits of a float

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at 0.5: its exponent field is 01111110 (126), which is -1 after removing the bias, and its fraction is all zeros, because 0.5 is exactly 1.0 x 2^-1. Numbers that are sums of powers of two (0.5, 0.25, 6.25) are stored exactly. Numbers such as 0.1 are not.

Why 0.1 cannot be stored exactly

A decimal fraction like 1/10 is a repeating pattern in binary, just as 1/3 repeats in decimal (0.3333...):

TEXT
0.1 (decimal) = 0.0001100110011001100110011... (binary, repeating)

Only a finite number of bits fit, so the pattern is cut off and rounded. The stored value is a tiny bit different from 0.1. In single precision the stored value is about 0.100000001490116...; in double precision it is 0.1000000000000000055511151231257827.... Each calculation then adds its own rounding step, and those small errors accumulate.

Python The famous 0.1 + 0.2

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lesson is not "floats are broken". It is that floats are binary approximations. Use a tolerance when comparing (math.isclose), and use decimal types (Python's decimal, or integer cents) for money.

Rounding

When a result falls between two representable numbers, the hardware must pick one. The default IEEE rule is round to nearest, ties to even: if the value is exactly halfway, choose the neighbour whose last fraction bit is 0. This avoids a systematic bias upward or downward. Other modes (toward zero, toward plus or minus infinity) exist for special needs such as interval arithmetic.

The gap between neighbouring floats is not constant. It grows with the magnitude: near 1.0 the gap in double precision is 2^-52 (about 2.2 x 10^-16), called machine epsilon; near 10^16 the gap is already 2.

Python Gaps between floats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is absorption: a small number added to a large one is lost when it is smaller than half the gap. It is why summing a long list from smallest to largest is more accurate than the reverse, and why math.fsum exists.

Special values

The exponent field reserves two extreme patterns for non-numbers and tiny numbers:

Exponent field Fraction Meaning
all 0s 0 Zero (+0.0 or -0.0)
all 0s non-zero Subnormal (no hidden 1, gradual underflow)
1 to max-1 any Normal number
all 1s 0 Infinity (+inf, -inf)
all 1s non-zero NaN ("not a number")
  • Infinity results from overflow or from dividing a non-zero number by zero (in languages that follow IEEE rules; Python raises an exception for 1/0 but returns inf for 1e308 * 10).
  • NaN results from undefined operations such as inf - inf or 0.0 * inf. NaN is not equal to anything, including itself, so x != x is the classic NaN test (Python has math.isnan).
  • Subnormals fill the gap between zero and the smallest normal number, so values shrink gradually instead of snapping to zero.
  • +0.0 == -0.0 is true, but dividing by them gives opposite infinities.
Python Zero, infinity, NaN and subnormals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Half, single and double

Different precisions suit different jobs.

Format Approx. decimal digits Max value Typical use
Half (16 bit) 3 to 4 65,504 Graphics, machine learning inference
Single (32 bit) 7 about 3.4 x 10^38 Games, GPUs, audio
Double (64 bit) 15 to 16 about 1.8 x 10^308 Science, Python's float, JavaScript numbers

Smaller formats save memory and bandwidth, which matters for GPUs and neural networks, but they overflow and lose precision much sooner. Python's float is always a double.

Python Same number, three precisions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Half precision stores only 11 significant bits, so integers above 2048 start skipping numbers; single precision does the same above 16,777,216 (2^24).

Practical rules

  1. Never test floats for equality after arithmetic. Compare with a tolerance.
  2. Do not use floats for money. Use integers (cents) or a decimal type.
  3. Order matters. (a + b) + c can differ from a + (b + c). Floating-point addition is commutative but not associative.
  4. Beware subtracting nearly equal numbers. The leading digits cancel, and the remaining digits are mostly rounding noise (catastrophic cancellation).
  5. Know your type. Choose single or half only when the precision loss is acceptable.

The same rules hold in every language: C, Java, JavaScript and Python all use IEEE 754 doubles for their usual decimal type. See Python's numbers lesson for the language side.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How many bits does an IEEE 754 single-precision number use for the exponent and for the fraction?

    Single precision is 1 sign bit, 8 exponent bits and 23 fraction bits. Double uses 11 and 52, half uses 5 and 10.
  2. 2. Why is 0.1 not stored exactly in binary floating point?

    1/10 is a repeating fraction in base 2, just like 1/3 in base 10, so only an approximation fits in a finite number of bits.
  3. 3. What is true of NaN?

    NaN compares unequal to everything, so x != x is true only for NaN. It also comes from operations like inf - inf.
  4. 4. What does the hidden bit in a normal IEEE 754 number mean?

    Normalized significands always start with 1, so that bit is left out, giving one extra bit of precision for free.
  5. 5. Which approach is best for comparing the results of floating-point calculations?

    Rounding errors make exact equality unreliable, so compare within an absolute or relative tolerance.

Summary

  • IEEE 754 stores a number as sign, biased exponent and fraction, with an implied leading 1 for normal numbers.
  • Half, single and double use 16, 32 and 64 bits, trading range and precision for size.
  • Only sums of powers of two are exact; most decimal fractions such as 0.1 are rounded approximations.
  • Results round to nearest, ties to even, and the gap between floats grows with magnitude.
  • Special patterns encode zero (signed), subnormals, infinities and NaN.
  • Compare with tolerances, avoid floats for money, and remember that addition is not associative.

Next, turn to text with Characters, ASCII and Unicode.