Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 45 ยท Operators and Math

Numeric Types and Precision

Understand double, single and integer types, saturation arithmetic, eps, floating-point surprises, Inf, NaN and complex numbers in MATLAB and Octave.

When you type x = 5 in MATLAB or Octave, you get a 1-by-1 array of the class double: a 64-bit floating-point number. Nearly all numeric work uses this default, and it is a good one. But sooner or later you will meet a smaller type such as uint8 (the type of an image pixel), an integer that refuses to go above 255, or a comparison like 0.1 + 0.2 == 0.3 that comes out false. This lesson explains why those things happen and how to handle them.

You will learn the numeric classes, how integer arithmetic saturates instead of wrapping around, what eps and floating-point rounding mean in practice, how Inf and NaN behave and how complex numbers work.

MATLAB and Octave are not installed on this site, so matlab blocks are read-only. The TryIt blocks are NumPy stand-ins. NumPy integers wrap around on overflow while MATLAB's saturate, so the stand-ins define a small helper that clamps values the way MATLAB does.

The numeric classes

double 64 bits. The default for everything.
single 32 bits. Half the memory, about 7 digits.
intN Signed whole numbers, int8 to int64.
uintN Unsigned whole numbers, uint8 to uint64.

Check with class(x), intmax('int8') and realmax.

1 / 2

Floating-point classes

double stores about 15 to 16 significant decimal digits, with values up to realmax (about 1.8e308) and down to realmin (about 2.2e-308) before reaching zero. single uses half the memory, about 7 significant digits and a range up to roughly 3.4e38. Convert with single(x) and double(x).

MATLAB
class(5)             % 'double'
class(single(5))     % 'single'
class(5 > 3)         % 'logical'
class('hi')          % 'char'

x = single(pi)       % 3.1416 on screen, but the stored value is 3.14159274...
double(x) - pi       % about 8.7423e-08: the error of the 32-bit rounding

Arithmetic between a single and a double gives a single, so one stray single can quietly lower the precision of a whole calculation. Use single when memory or speed on very large arrays matters, or when you must match 32-bit data; otherwise stay with double.

Integer classes and saturation

The integer classes come in signed (int8, int16, int32, int64) and unsigned (uint8, uint16, uint32, uint64) flavors. The limits come from the bit width: an int8 holds -128 to 127 and a uint8 holds 0 to 255. Query them with intmin('int8') and intmax('uint8').

What happens beyond the limit is where MATLAB differs from C, Java and NumPy. Integer arithmetic saturates: results are computed exactly and then clamped to the nearest representable value.

MATLAB
uint8(200) + uint8(100)   % 255   not 44
int8(-100) - int8(100)    % -128
uint8(5) - uint8(10)      % 0     unsigned values cannot go below 0
uint8(300)                % 255   conversion also saturates
int8(-200)                % -128

Several other rules matter. Conversions from a fraction round to the nearest integer (halves away from zero): uint8(2.5) is 3. Integer division rounds too: int32(7) / int32(2) is 4 and int32(-7) / int32(2) is -4. If you want truncation toward zero, use idivide(int32(7), int32(2)), whose default is 'fix' and gives 3. An integer combined with a double gives the integer class (int8(5) + 2.7 is the int8 value 8), while combining two different integer classes is an error. NaN converts to 0.

Python Integer limits and saturation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Saturation has a real consequence for image processing: adding brightness to a uint8 image clips at white (255) instead of wrapping to dark, which is usually what you want, but averaging two uint8 images as (a + b) / 2 loses information because a + b clips first. Convert with double(a) before such arithmetic.

Floating-point precision

A double has only a finite number of values, so most decimal fractions are stored as the nearest binary approximation. That is why 0.1 + 0.2 is not exactly 0.3, in MATLAB just as in every language that uses IEEE 754 doubles.

MATLAB
0.1 + 0.2 == 0.3          % 0 (false)
0.1 + 0.2 - 0.3           % 5.5511e-17
abs((0.1 + 0.2) - 0.3) < 1e-12   % 1 (true)

The key number is eps, the gap between 1 and the next larger double: eps is about 2.2204e-16. The gap grows with the size of the number: eps(x) gives the spacing at x, so eps(1000) is about 1.1369e-13. This means a number added to a much larger one can vanish completely: 1 + eps/2 == 1 is true. The same effect limits whole numbers: above flintmax (2^53, about 9.007e15) not every integer can be stored, and 2^53 + 1 == 2^53 is true. In single the gap is much larger, eps('single') is about 1.1921e-07, and single(16777216) + 1 still equals 16777216.

Practical rules follow from this:

  • Never test floats for equality. Compare with a tolerance: abs(a - b) < tol, scaled to the size of the numbers, for example abs(a - b) <= 1e-9 * max(abs(a), abs(b)).
  • Prefer integer counters in loops where you want exact stepping, and use linspace rather than accumulating x = x + 0.1.
  • Expect tiny errors in results that "should" be exactly zero, such as sin(pi) (about 1.2246e-16) or A * inv(A) - eye(n).
Python eps, rounding errors and single precision

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inf, NaN and complex numbers

Floating-point arithmetic never throws a "divide by zero" error. Instead it produces special values:

MATLAB
1/0          % Inf
-1/0         % -Inf
0/0          % NaN
Inf - Inf    % NaN
Inf + 1      % Inf
realmax * 10 % Inf     overflow

NaN ("not a number") is contagious: almost any arithmetic with it gives NaN, and it is the only value not equal to itself, so use isnan(x), isinf(x) and isfinite(x) for tests, never x == NaN. See Built-in Math and Statistics for ways to skip NaN in sums and means.

Complex numbers are built in. Write 3 + 4i (or 3 + 4j) and every function understands them. Use real(z), imag(z), abs(z) (the magnitude), angle(z) (the phase in radians) and conj(z). Square roots and logarithms of negative numbers automatically become complex:

MATLAB
z = 3 + 4i;
abs(z)            % 5
angle(z)          % 0.9273
z * conj(z)       % 25  (equal to abs(z)^2)
sqrt(-4)          % 0 + 2.0000i
exp(1i * pi)      % -1.0000 + 0.0000i  (Euler's identity, with rounding error)

The name i is also an ordinary variable name. If you use i as a loop counter, then 3 + 4i still works (a number followed directly by i is always the imaginary unit) but 3 + 4*i does not. Write 1i for the imaginary unit when i may have been reused.

Python Inf, NaN and complex numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does uint8(200) + uint8(100) return in MATLAB?

    Integer arithmetic saturates: the exact result 300 is clamped to the largest uint8 value, 255. NumPy and C would wrap around to 44.
  2. 2. What is the class of int8(5) + 2.7?

    Combining an integer with a double gives the integer class. The exact result 7.7 is rounded to the int8 value 8.
  3. 3. Which is the right way to test whether a and b are equal floating-point results?

    Rounding makes exact equality unreliable (0.1 + 0.2 == 0.3 is false). Compare with a suitable tolerance.
  4. 4. What does 0/0 give?

    MATLAB follows IEEE 754: 1/0 is Inf and 0/0 is NaN. No error is raised.
  5. 5. What does eps represent?

    eps is the spacing of doubles at 1. The smallest positive normal number is realmin, and flintmax is the limit for exact integers.

Summary

  • The default class is double; single halves memory at about 7 digits, and int8 to uint64 store whole numbers in fixed ranges.
  • Integer arithmetic and conversions saturate at the limits, round fractions to the nearest integer and give the integer class when mixed with double.
  • Floating-point values are approximations: eps is the spacing at 1, eps(x) at x, and equality tests need a tolerance.
  • Division by zero gives Inf or NaN rather than an error; test with isinf, isnan and isfinite.
  • Complex numbers are built in (3 + 4i) with real, imag, abs, angle and conj, and sqrt(-4) yields 0 + 2i.
  • Convert uint8 data to double before arithmetic that could clip.

Next: work with text in Strings and Character Arrays.