Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 64 ยท Data Types

Missing Data

Find, count, drop and fill missing values in pandas. Compare NaN, NA and None, use dropna, fillna, ffill, bfill and interpolate, and avoid common traps.

Real datasets have holes: a sensor that stopped reporting, a survey question someone skipped, a price nobody entered. pandas represents these holes as missing values and gives you a consistent set of tools to find them, remove them or fill them in.

How you handle missing data changes your results. Dropping rows shrinks the sample; filling with zero can drag an average down; filling with the mean hides variation. There is no single right answer, so this lesson focuses on seeing the gaps clearly and choosing deliberately. It builds on Data Types and Conversion, because the dtype decides which missing marker a column uses.

Three markers: NaN, None and NA

You will see three spellings of "nothing here":

  • NaN (np.nan) is a special floating-point value, "not a number". It is the classic missing marker for float columns.
  • None is Python's own null. pandas converts it to the right marker for the column when it can.
  • pd.NA is pandas' newer, dtype-independent marker used by the nullable dtypes (Int64, boolean, string).

Whatever the marker, isna() treats them all the same way.

Python One test for every kind of missing value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Never test for missing values with == np.nan; it is always False. Use isna() / notna() (or pd.isna(x) for a single value).

Finding and counting gaps

df.isna() returns a DataFrame of booleans. Because True counts as 1, summing it counts the gaps per column, and taking the mean gives the share missing.

Python Counting missing values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

df.isna().sum() is probably the first line you will run on any new table. count() is the mirror image: it counts the non-missing values per column.

flowchart LR
    A["Table with gaps"] --> B["isna().sum()<br/>how many?"]
    B --> C{"Decide"}
    C --> D["dropna<br/>remove"]
    C --> E["fillna / ffill<br/>replace"]
    C --> F["interpolate<br/>estimate"]
    C --> G["keep as NA<br/>report it"]

Count first, then choose a strategy; keeping the gaps is a valid choice too.

Dropping with dropna

dropna() removes rows (or columns) that contain missing values. Its options control how strict it is:

Option Meaning
how="any" (default) drop a row if any value is missing
how="all" drop a row only if every value is missing
subset=[...] look only at these columns
thresh=n keep rows with at least n non-missing values
axis=1 drop columns instead of rows
Python dropna options compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The default how="any" is aggressive. With many columns, nearly every row has some gap, and you could lose most of your data. Prefer subset= to require only the columns that matter for the question you are asking.

Filling with fillna

fillna(value) replaces gaps with a constant, a per-column dict, or a computed value such as the column mean.

Python Filling with constants and statistics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last line: filling with 0 pulls the average from 80.0 down to 48.0, because the zeros are now counted as real scores. Decide whether a gap means "zero" (no sales that day) or "unknown" (the sensor was off). Those are different facts.

Carrying values: ffill, bfill and interpolate

For ordered data such as time series, the neighbouring values are often the best guess.

  • ffill() (forward fill) copies the last known value forward.
  • bfill() (backward fill) copies the next known value backward.
  • interpolate() draws a line between known points. With method="time" and a date index, it respects uneven spacing.
Python Forward fill, backward fill and interpolation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two details to remember. bfill leaves the final gap because nothing comes after it. interpolate fills a trailing gap with the last known value by default, but it never invents values before the first known point. The limit= argument stops a fill from reaching too far: a sensor that has been dead for a week should not be "filled" with last Tuesday's reading.

Warning: Filling on unsorted data gives nonsense. Sort by the time column first, and in grouped data fill within each group, or one customer's last value leaks into the next customer's rows.

How missing values behave in calculations

Arithmetic with NaN gives NaN, but pandas aggregation methods skip missing values by default:

Python Missing values in arithmetic and statistics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that mean() divides by 2, not 3: it only counts the values that exist. And s <= 2 is False for the gap, so the row vanishes from your filter. If you want to keep missing rows in a filter, add | s.isna() explicitly.

Nullable dtypes handle gaps better

With classic dtypes, an integer column with a gap silently becomes float64. The nullable dtypes keep the type and use pd.NA, which also behaves more carefully in logic: pd.NA > 1 is unknown, not False.

Python Nullable integers and booleans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you read data with pd.read_csv(..., dtype_backend="numpy_nullable") or call convert_dtypes(), you get these dtypes without naming them column by column.

Other markers hiding as data

Not every gap is a NaN. Files often use "-", "N/A", "unknown" or -999. Tell the reader about them with na_values= when loading, and they become real missing values from the start:

Python Teaching read_csv what means missing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without na_values, the stray "-" made the whole column text. After declaring it as missing, the column is numeric again. Reading CSV Data covers the other loading options.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is df['x'] == np.nan not a good way to find missing values?

    NaN compares unequal to everything, including itself, so the result is all False. Use isna() or notna().
  2. 2. What does df.dropna(thresh=2) do?

    thresh is the minimum number of non-missing values a row needs to be kept.
  3. 3. A Series is [80, NaN, 95]. What does s.mean() return?

    Aggregations skip missing values by default, so the mean is (80 + 95) / 2 = 87.5. Pass skipna=False to get NaN.
  4. 4. Which method estimates a missing value from the points on either side of it?

    interpolate draws a line between known neighbours. ffill and bfill copy a single neighbour, and dropna removes the row.

Summary

  • Missing values appear as NaN, None or pd.NA; test for them with isna() / notna(), never with ==.
  • df.isna().sum() counts gaps per column; .mean() gives the share missing.
  • dropna supports how, subset, thresh and axis; prefer subset so you only drop for columns that matter.
  • fillna takes constants, dicts or statistics; ffill, bfill and interpolate suit ordered data (sort first, fill within groups).
  • Sums and means skip gaps by default; comparisons with gaps are False, so filters can silently lose rows.
  • Nullable dtypes keep integers as integers and use pd.NA.
  • Tell read_csv which placeholders mean missing with na_values.

Next: Working with Text.