Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 76 ยท Cleaning and Preparing Data

Handling Missing Values

Decide how to handle missing data: MCAR, MAR and MNAR, dropping vs filling vs flagging, mean, median and group imputation, indicators and leakage.

Missing values are the most common finding in a data audit, and the most common place to make a quiet mistake. Filling every gap with the column mean feels harmless, yet it can shrink variation, hide relationships and change results. Dropping every incomplete row feels clean, yet it can throw away most of your data, or the very people you most need to understand.

The right choice depends on why the value is missing. This lesson gives you a vocabulary for that (MCAR, MAR, MNAR), compares the standard strategies on data where the truth is known, and shows the one rule that prevents leakage when you fill values before building a model. The mechanics of isna, fillna and dropna are covered in the pandas lesson on missing data; here the focus is on judgement.

Why is it missing? Three mechanisms

Statisticians describe missingness by what the chance of being missing depends on:

  • MCAR (missing completely at random): the chance is the same for every row. A sensor drops 2% of readings at random. Dropping these rows loses information but adds no bias.
  • MAR (missing at random): the chance depends on other columns you observed. Young customers skip the income question more often. Missing values can be predicted from the rest of the table, so group-wise fills can repair a lot of the damage.
  • MNAR (missing not at random): the chance depends on the missing value itself. High earners refuse to state their income. Nothing in the observed data fully corrects this; you can only reason about it and test how sensitive your result is.

The names are confusing, but the question is simple: does knowing the missing value (or other columns) tell me which rows are missing?

flowchart TD
    A["Some values are missing"] --> B{"Why?"}
    B -->|"No pattern"| C["MCAR<br/>drop or fill, little bias"]
    B -->|"Depends on other columns"| D["MAR<br/>fill using those columns"]
    B -->|"Depends on the value itself"| E["MNAR<br/>flag, model it, state the limit"]

You usually cannot prove which case you are in; you gather evidence and choose the safest handling.

Diagnosing from the data

You can test for MAR by comparing the missing rate across observed groups. If it differs a lot, the data is not MCAR. You cannot test for MNAR directly because the evidence is exactly what is missing. The simulation below hides about 20% of incomes in three different ways, so we know the mechanism of each.

Python Three missingness mechanisms on one column

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the three rows. For MCAR the missing share is the same in both age groups (about 20%) and the mean of the observed incomes is near the truth (45,771). For MAR the share is far higher among the young (36% against 11%), and the observed mean is too high, because the lower earners are the ones missing. The MNAR row shows an odd effect: the age groups differ, because income depends on age and the missingness depends on income. An analyst looking only at the table could mistake it for MAR. The observed mean is too low there, since high incomes disappeared more.

The lesson: a missing-by-group table is a good first diagnostic, but it cannot separate MAR from MNAR. Domain knowledge ("do rich people skip this question?") has to do that.

Three basic moves: drop, fill, flag

Drop the rows (or the column). This is safe when values are MCAR and few. It is risky when many rows are lost, or when the dropped rows differ from the kept ones. Dropping a column is reasonable when most of it is missing and it is not important.

Fill (impute) with a stand-in value. Common choices:

  • the mean (sensitive to outliers; shrinks the spread),
  • the median (robust, a good default for skewed numbers),
  • the most frequent value or a new "Unknown" category, for categories,
  • a group-wise statistic, such as the median income within the age band,
  • a model prediction from the other columns (more advanced; same cautions).

Flag the gap by adding an indicator column (income_missing, 0 or 1) before filling. Missingness is sometimes informative on its own: customers who skip a field may behave differently. A model can use the flag; a human can see which values were invented.

Comparing strategies on data with a known truth

Because we kept the true incomes, we can score each strategy. The data below is MAR: young people hide their income more often. We check three things per strategy: the estimated mean, the estimated spread (std), and the RMSE (the typical error) on the cells that were actually filled.

Python Drop vs mean vs median vs group median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table tells a clear story:

  • Dropping rows and filling with the overall mean or median both give a mean about 1,100 too high (46,835, 46,883 against 45,771), because the typical filled value is the average of the observed people, who are older and richer than the missing ones.
  • Mean and median fills also squash the spread: the std falls from 11,045 to about 9,730, because hundreds of identical values were pasted in. That makes later confidence intervals too narrow and correlations too weak.
  • Filling with the age-band median uses the observed column that explains the missingness. The mean is back on target (45,772), the spread is closer to the truth (10,636), and the error on the filled cells roughly halves (6,576 against 12,436).

Group-wise filling worked because the mechanism was MAR and the grouping column was the one that mattered. Under MNAR no such trick exists. In every case, simple filling gives a single guess per gap, so it still understates uncertainty. Advanced methods (multiple imputation) generate several plausible fills and combine the results; for most practical projects, the realistic goal is to avoid the worst bias and be open about the rest.

Note: Never fill the target you want to predict. Rows with a missing label should be set aside, not given an invented answer.

Avoiding leakage when you fill

If a model will be evaluated on unseen data, every statistic used to fill must be learned from the training data only. If you compute the median on the whole table and then split, information about the test rows has leaked into the training procedure. The effect is usually small for medians but can be serious for other transformations, and it breaks the principle that the test set stands in for the future.

The pattern: split first, fit the imputer on the training part, then transform both parts with that fitted imputer. In scikit-learn this is exactly what SimpleImputer does.

Python Fit the imputer on train only, add an indicator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training median (45,432) differs from the median of the whole table (45,961). The test rows are filled with the training value, exactly as a deployed model would have to do for a new customer. You will wire this into a full workflow in Pipelines and Hyperparameter Tuning, where the imputer is part of the model and cannot leak by construction.

A practical checklist

  1. Count and locate the gaps (the audit from Auditing Data Quality).
  2. Ask why: process failure, optional question, not applicable, privacy refusal? "Not applicable" (a ship date for a pending order) is not missing data at all; it is information.
  3. Compare missing rates across observed groups to look for MAR.
  4. If under a few percent and plausibly MCAR, dropping or a simple median fill is usually fine.
  5. If a grouping explains it, fill within groups; add an indicator column.
  6. If you suspect MNAR, say so, and test how the conclusion changes under different assumptions.
  7. Always fit fill-values on training data only, and record what you did.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Younger customers skip the income question more often, and age is recorded for everyone. What type of missingness is this?

    The chance of being missing depends on another observed column (age), which is the definition of missing at random.
  2. 2. People with very high incomes refuse to report them. Which statement is true?

    The missingness depends on the unobserved value itself. The observed data cannot fully reveal or correct that.
  3. 3. What is a downside of filling every gap with the column mean?

    Many identical values are pasted in, so the variation falls. Later intervals become too narrow and correlations too weak.
  4. 4. Why fit an imputer on the training set only?

    The test set must act as unseen future data. Statistics learned from it would let information from the test rows influence the model.
  5. 5. What does an indicator column such as income_missing add?

    Missingness can carry information, and the flag also records which values were invented.

Summary

  • Ask why values are missing: MCAR (no pattern), MAR (depends on other columns), MNAR (depends on the value itself).
  • Compare missing rates across observed groups to diagnose; only domain knowledge separates MAR from MNAR.
  • Three moves: drop (safe when few and MCAR), fill (median, group-wise median, "Unknown"), and flag with an indicator column.
  • Mean and median fills shrink variation; group-wise fills help when the grouping explains the missingness.
  • Learn fill values from the training data only, and never invent target values.

Next: Outliers, Errors and Impossible Values.