Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 49 ยท Working with Real Data

Missing Data and NaN

Handle missing values in NumPy with np.nan, nan-aware functions, masked arrays, boolean filters, fill strategies and linear interpolation with np.interp.

Real data has holes. A sensor drops out, a survey question is skipped, a file has an empty cell. NumPy has no special "missing" type for integers, but floating-point arrays have a built-in marker: NaN, short for "not a number". Handling it correctly is the difference between a result and a column full of nan.

This lesson shows how NaN behaves (it is stranger than you expect), which functions skip it, how to filter or replace it, how masked arrays track missing values without using NaN at all, and how to fill gaps by interpolation. For the building blocks, revisit Boolean Masks and Aggregations and Axes.

NaN is contagious and never equal to itself

NaN follows IEEE 754 floating-point rules. Any arithmetic involving NaN produces NaN, and NaN compares unequal to everything, including itself. That is why you must test for it with np.isnan, never with == np.nan.

Python How NaN behaves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three rules follow from this output:

  • A single NaN turns sum, mean and max into NaN. The result is "unknown", not an error.
  • Comparisons with NaN are False, including a > 2 and a <= 2. A filter such as a[a > 2] silently drops NaN rows.
  • Integer and boolean arrays cannot hold NaN. If your data may have gaps, load it as float64 (see Data Types).

NaN-aware functions

For nearly every reduction there is a twin that ignores NaN: np.nansum, np.nanmean, np.nanmedian, np.nanstd, np.nanvar, np.nanmin, np.nanmax, np.nanargmin, np.nanargmax, np.nancumsum, np.nanpercentile. They accept axis and keepdims like the originals.

An all-NaN slice has no valid values, so nanmean returns NaN and emits a RuntimeWarning ("Mean of empty slice"). Check np.isnan(arr).all(axis=...) first if that can happen.

flowchart TD
    A["Array with NaN"] --> B{"What do you need?"}
    B -->|"Just a statistic"| C["np.nanmean, np.nansum, ..."]
    B -->|"Drop bad rows"| D["arr[~np.isnan(arr).any(axis=1)]"]
    B -->|"Keep shape, fill in"| E["np.where(np.isnan(arr), fill, arr)"]
    B -->|"Keep shape, remember gaps"| F["np.ma.masked_invalid(arr)"]
    B -->|"Time series"| G["np.interp over valid points"]

Choose the strategy by what you need afterward. Dropping loses data, filling invents data, masking keeps both.

Finding, dropping and replacing NaN

Because NaN lives in a normal array, all the tools of boolean indexing apply. On 2D data, any(axis=1) collapses each row to "does it contain a NaN?".

Python Filtering and filling NaN in a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A few cautions about filling:

  • Zero is rarely a neutral fill. It pulls means down and can mean "no reading" or "a real zero". Use a value that makes sense for the data.
  • Fill with the mean/median of valid values keeps the average unchanged but shrinks the variance.
  • np.nan_to_num also converts infinities to huge finite numbers by default. Pass posinf and neginf explicitly.
  • Use np.isfinite when you want to reject both NaN and infinity at once.

Masked arrays

Sometimes the missing marker is not NaN at all. Many data sources use -999 or -1 for "no value", and integer arrays cannot hold NaN. Masked arrays (np.ma) store the data plus a boolean mask that says which entries to ignore. Reductions, arithmetic and mean skip masked entries automatically.

Python Masked arrays with np.ma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Masked values print as --. The original numbers stay in .data, so the mask is reversible. The trade-off: masked arrays are slower than plain arrays and many functions outside np.ma ignore the mask, so call .filled() or .compressed() before passing them to other libraries.

Ordinary NumPy functions applied to m.data see the -999 values. Use the methods and functions in np.ma (m.mean(), np.ma.sum(m)), not np.mean(m.data).

Interpolating gaps with np.interp

For ordered data such as a time series, a better fill than a constant is linear interpolation: draw a straight line between the nearest valid neighbors. np.interp(x_new, x_known, y_known) does exactly that. Give it the positions you want, and the positions and values you trust. The known x values must be increasing.

Python Filling gaps in a time series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the default behavior outside the known range: np.interp clamps to the first or last value instead of extrapolating. Pass left=np.nan, right=np.nan if you would rather keep those positions unknown. Interpolation assumes the values change smoothly; do not use it for categories or for gaps so long that the straight line is fiction.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does arr[arr == np.nan] never select anything?

    By IEEE 754 rules NaN == NaN is False. Use np.isnan(arr) to build the mask.
  2. 2. What does np.mean(np.array([1.0, np.nan, 3.0])) return?

    NaN propagates through arithmetic, so the mean is nan. np.nanmean would return 2.0.
  3. 3. Which array can NOT store np.nan?

    Integer dtypes have no NaN representation. Assigning np.nan to an int array raises ValueError; convert to float first.
  4. 4. What is a benefit of a masked array compared with filling NaN with a number?

    A masked array keeps data and mask separately, so the operation is reversible and statistics skip masked entries. It is not faster, and other libraries may ignore the mask.

Summary

  • Floating-point arrays mark missing values with np.nan; integer arrays cannot hold it.
  • NaN propagates through arithmetic and is never equal to anything. Detect it with np.isnan and np.isfinite.
  • np.nansum, np.nanmean, np.nanmax and friends skip NaN; all-NaN slices still give NaN with a warning.
  • Drop rows with arr[~np.isnan(arr).any(axis=1)], or fill with np.where or np.nan_to_num, choosing a fill that makes sense for the data.
  • np.ma masked arrays keep data and mask apart and handle sentinel values like -999.
  • np.interp fills gaps in ordered data with straight lines and clamps outside the known range.

Next, read tables and text into arrays in Structured Arrays and Text Data.