Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 111 ยท Feature Engineering and Pipelines

Handling Missing Data

Deal with missing values using SimpleImputer, KNNImputer and IterativeImputer, add missing indicators, understand MCAR, MAR and MNAR, and use models that accept NaN.

Real data has holes. A sensor failed, a customer skipped a form field, a lab test was never ordered. Most scikit-learn models refuse to run when the input contains NaN, so you must decide what to do with the gaps. Deleting rows is tempting, but with 20 columns each missing 5% of the time, nearly two thirds of the rows lose at least one value. Usually you fill the gaps in, which is called imputation.

How you fill them matters less than why they are missing. This lesson introduces the imputers, the three classic reasons for missingness, and the trick of telling the model which values were missing. It follows Encoding Categorical Features and prepares you for Pipelines.

Why is the value missing?

MCAR Missing completely at random. A coin flip decided.
MAR Missing depends on other columns you can see.
MNAR Missing depends on the hidden value itself.

The harder the type, the more careful you must be.

1 / 2

Three kinds of missingness

  • MCAR (missing completely at random): the chance of a value being missing has nothing to do with any data. A dropped network packet is an example. Deleting rows is unbiased here, only wasteful.
  • MAR (missing at random): the chance depends on other observed columns. Younger respondents skip the income question more often, and age is recorded. The gaps can be corrected using the other columns.
  • MNAR (missing not at random): the chance depends on the missing value itself. High earners refuse to state their income. No technique can fully repair this from the data alone; you need domain knowledge, and often the fact of missingness is itself a signal.

The simulation below hides income values in each of the three ways. The true mean income is about 60. See what a naive average of the visible values reports.

Python How missingness bends an average

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

MCAR leaves the mean almost untouched. MAR and MNAR both pull it below the truth, but only MAR can be corrected, because the reason for the gaps is a column we have. Under MNAR the visible values are a biased sample and nothing in the table says by how much.

SimpleImputer

SimpleImputer replaces each missing value with a statistic of its own column: strategy="mean", "median" (robust to outliers), "most_frequent" (works for categories too) or "constant" with fill_value. It learns the statistic during fit, so, as with scalers, fit it on the training data only and reuse it on new data.

add_indicator=True appends one extra 0/1 column for each feature that had missing values in training, marking where gaps were. This lets the model use the fact of missingness, which matters when it carries information (MNAR, or a skipped form field that signals something).

Python SimpleImputer and missing indicators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result has six columns: the three imputed features and an indicator for each. Note that the median of the binary owns_car column is a questionable fill (a value of 1 for "owns a car" is not an average); for binary or categorical columns most_frequent, or a separate "unknown" category, is often more honest.

KNNImputer and IterativeImputer

Filling with a column statistic ignores the other columns. If income is strongly related to age, a young person with missing income should get a lower guess than an older one. Two imputers use that structure:

  • KNNImputer finds the n_neighbors rows most similar on the observed columns and averages their values for the missing one. Scale your features first, because it uses distances (see Scaling and Normalization).
  • IterativeImputer models each column with missing values as a regression on the other columns and cycles through them until the guesses settle. It is still marked experimental, so you import enable_iterative_imputer first.
Python Which imputer recovers hidden values best?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both smarter imputers roughly halve the error of mean filling here, because the columns are correlated. If the columns were unrelated, they would offer no advantage over the mean. Their cost is also higher: KNNImputer slows down on large data, and IterativeImputer fits many models.

Warning: Imputing, like scaling, must happen inside the cross-validation loop. Filling gaps using statistics from the whole dataset leaks information about validation rows, as explained in Nested Cross-Validation and Data Leakage. Put the imputer in a pipeline.

When missingness is a feature

Sometimes the gap tells you something. If sicker patients get more tests, then a missing test result hints at a healthier patient. In the next experiment, one feature is hidden much more often for the positive class. Plain mean imputation throws that clue away, while an indicator column or a model with native NaN handling keeps it.

HistGradientBoostingClassifier and HistGradientBoostingRegressor accept NaN directly: while training each split, they learn which side the missing values should go to. Some other tree-based estimators in recent scikit-learn versions do too; check the documentation of the model you use, because most (linear models, SVMs, k-NN) do not.

Python Mean imputation versus indicator versus native NaN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The indicator column lifts accuracy by about ten points because the pattern of gaps is predictive, and the gradient-boosting model finds the same signal without any preprocessing.

A practical recipe

  1. Count first. df.isna().mean() shows the share missing per column. Drop columns that are almost entirely empty and carry no signal.
  2. Ask why. Talk to whoever collected the data. MNAR cannot be seen in the table.
  3. Start simple: median for numbers, most frequent or an "unknown" category for categories, plus add_indicator=True.
  4. Try a smarter imputer only if columns are clearly related, and compare by cross-validation, not by gut feeling.
  5. Keep the imputer in the pipeline and fit it on training data only.
  6. Do not impute the target. Rows with a missing label are usually dropped or handled with semi-supervised methods.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Older customers skip the income field more often, and age is recorded for everyone. What type of missingness is this?

    The chance of a gap depends on another observed column (age), which is the definition of MAR. Because age is known, it can be used to correct the gaps.
  2. 2. What does SimpleImputer(add_indicator=True) add to the output?

    The indicator columns tell the model which values were originally missing, so it can use the missingness pattern as information.
  3. 3. Why should an imputer be fitted on the training data only?

    A mean, median or neighbour model is learned from data. Computing it on all rows leaks information about the evaluation rows, so fit on training data and reuse the result.
  4. 4. Which estimator from this lesson accepts NaN in its input without any imputation?

    Histogram gradient boosting learns a default direction for missing values at each split. The other models need complete numeric input.

Summary

  • Missing values come in three flavours: MCAR (harmless), MAR (explained by other columns) and MNAR (depends on the hidden value, the hardest case).
  • SimpleImputer fills with a mean, median, most frequent value or a constant; KNNImputer and IterativeImputer use relationships between columns and win when columns are correlated.
  • add_indicator=True lets the model see where data was missing, which helps when missingness is informative.
  • Histogram gradient boosting handles NaN natively; most other models do not.
  • Fit imputers on training data only, inside a pipeline.

Next: Feature Creation and Transformation.