Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 40 ยท Data Preparation

Handling Missing Values

Detect and fill missing data with SimpleImputer, KNNImputer and IterativeImputer, add missing-value indicators, and know which models accept NaN directly.

Real datasets have holes: a survey question skipped, a sensor that dropped out, a field nobody filled in. scikit-learn marks them as NaN (not a number), and most models refuse to train on them. Imputation means filling the holes with sensible estimates. This lesson shows three imputers and the most important rule about using them.

Look before you fill

Count what is missing first, per column.

Python Find the missing values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ask why the data is missing. "Not applicable" and "forgot to record" mean different things, and sometimes the fact of being missing is itself informative.

SimpleImputer: one value per column

SimpleImputer replaces NaN with the column mean, median, most frequent value or a constant.

Python SimpleImputer strategies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • mean is fine for symmetrical data; median is better with outliers (note how 100 pulls the mean).
  • most_frequent works for categories.
  • constant lets you choose a placeholder such as "missing" or 0.

Learn on train, apply to test

An imputer is a transformer, so the usual rule holds: fit it on the training data and only transform the test data. The fill value must come from training rows.

Python Fit on train, transform test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keep the signal: missing indicators

If missingness carries information (for example, high earners skipping the income question), add a flag column. add_indicator=True appends a 0/1 column per feature that had missing values in training.

Python add_indicator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

KNNImputer: borrow from similar rows

KNNImputer fills a gap with the average of that feature in the k most similar rows, measured on the other columns. It uses relationships between columns, which a column-wise mean ignores.

Python KNNImputer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The KNN estimate comes from the neighbours at 160 and 170 cm, a more believable value than the overall average. K-NN needs scaled features to measure similarity well, see Scaling and Normalization.

IterativeImputer: model each column

IterativeImputer (experimental, so it needs an extra import) predicts each incomplete column from all the others with a regression model, looping until stable. It is slower but often the most accurate for numeric data.

Python IterativeImputer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Models that accept NaN

A few estimators handle missing values themselves, for example HistGradientBoostingClassifier and HistGradientBoostingRegressor, and recent tree models. Everything else needs imputation first. Even when a model tolerates NaN, an explicit imputer can be clearer and lets you compare strategies.

Python A model that accepts NaN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dropping rows or columns

Removing is sometimes right: drop a column that is 80% empty, or drop the few rows missing the target. Otherwise you lose data and can bias the sample, so impute by default.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which SimpleImputer strategy suits a text column?

    Mean and median need numbers. most_frequent fills with the commonest category.
  2. 2. What does add_indicator=True do?

    Indicator columns preserve the information that a value was missing.
  3. 3. Why is KNNImputer sometimes better than a column mean?

    It estimates a gap from the most similar samples instead of a global average.
  4. 4. On which data should an imputer be fit?

    Fill values come from training data; the same imputer then transforms the test set.
  5. 5. Which model class can take NaN inputs directly?

    Histogram gradient boosting treats missing values natively.

Summary

  • Count NaN per column, and ask why it is missing.
  • SimpleImputer fills with mean, median, most frequent or a constant; KNNImputer and IterativeImputer use relations between columns.
  • Fit imputers on training data only.
  • add_indicator=True keeps the "was missing" signal.
  • Histogram gradient boosting accepts NaN itself.

Next: Encoding Categorical Features