Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 111 ยท Feature Engineering and Pipelines

Outliers with Z-Scores and the IQR Rule

Find and handle outliers before modeling: point, contextual and collective types, z-score, modified z-score and IQR rules, capping, dropping and robust models.

An outlier is a value that sits far away from the rest of the data. Sometimes it is a mistake (a salary typed with an extra zero, a sensor that reports -999 when it fails). Sometimes it is the most interesting row in the table (a fraudulent payment, a machine about to break). Either way, outliers can drag averages, stretch scales and bend a fitted line, so every project needs a plan for them before the model sees the data.

This lesson covers the simple statistical rules you use during exploration: the z-score, the modified z-score and the interquartile range (IQR) rule. It also shows what to do once you have found something. For model-based detectors that work on many features at once, continue with anomaly and outlier detection.

Three kinds of outliers

  • Point outliers are single values that are unusual compared with the whole column. A customer age of 212 is one.
  • Contextual outliers are normal overall but unusual in their context. A temperature of 31 ยฐC is ordinary in July and extraordinary in February. You need the context (month, user, location) to see it.
  • Collective outliers are groups of points that are unusual together, although each point alone looks fine. A heartbeat signal that goes flat for twenty readings has no extreme value, yet the pattern is clearly wrong.

Simple column-by-column rules find only point outliers. For the other two kinds you must add context (group by month, compare with neighbors, look at windows).

The z-score rule

The z-score tells you how many standard deviations a value is from the mean:

TEXT
z = (x - mean) / standard deviation

For roughly bell-shaped data, about 99.7% of values lie within three standard deviations, so the common rule flags points with |z| > 3. The next example plants three bad values in 200 normal incomes and finds them with the z-score and the IQR rule.

Python Z-score and IQR rules side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both rules catch the three planted values. The IQR rule also flags two ordinary-looking values (81.9 and 26.5), because its fences are tighter. For a perfect bell curve the 1.5 ร— IQR fences cover about 99.3% of the data, so the rule flags a little more than the z-score rule does. That is by design: the IQR rule is a screening tool, and a flagged value is a reason to look, not proof that it is wrong.

The IQR rule

The IQR (interquartile range) is the distance between the 25th and 75th percentiles, Q3 - Q1. It measures the spread of the middle half of the data. The rule marks anything below Q1 - 1.5 ร— IQR or above Q3 + 1.5 ร— IQR as an outlier. This is exactly what the whiskers of a box plot show. Because quartiles ignore the extreme tails, the IQR does not depend on the outliers themselves, and it does not assume a bell curve. That makes it the safer default for skewed data such as prices or waiting times.

Use 3 ร— IQR instead of 1.5 ร— IQR when you want to flag only "far out" values.

Masking: why the z-score can miss outliers

The mean and standard deviation are themselves pulled by the outliers. Big outliers inflate the standard deviation and can hide each other, which is called masking. The modified z-score uses the median and the median absolute deviation (MAD), which barely move when a few values are extreme:

TEXT
modified z = 0.6745 ร— (x - median) / MAD       where MAD = median(|x - median|)

A common cutoff is |modified z| > 3.5.

Python Three large outliers mask each other

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three huge values inflate the standard deviation so much that none of them reaches |z| = 3, so the plain z-score flags nothing. The modified z-score flags all three. With a small sample, a z-score can never exceed (n - 1) / sqrt(n), so on tiny datasets the "3 standard deviations" rule cannot fire at all.

Contextual outliers: compare within a group

To find contextual outliers, compare each value with others in the same context. The example below has 20 years of monthly temperatures with one warm February.

Python A value that is only unusual in context

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the whole column, 31 ยฐC looks like a hot summer day. Inside its own month, it is many robust standard deviations from normal. Context made the difference.

What to do with an outlier

Detection is half the job. Decide per case:

  1. Check the source. If the value is impossible (negative age, sensor error code), fix it or mark it as missing and impute it (see missing values).
  2. Keep it if it is real and relevant. In fraud or fault detection, the outliers are the signal. Never delete them there.
  3. Cap (winsorize) it. Clip values to a chosen percentile, for example the 5th and 95th. You keep the row and limit its influence.
  4. Transform it. A log or power transform pulls long right tails in (see PowerTransformer).
  5. Use something robust. RobustScaler scales with the median and IQR. HuberRegressor and tree-based models are far less sensitive to extreme values than ordinary least squares.
  6. Drop it only with a reason you can write down. Removing points just because they hurt your score is a form of cheating.
Python How much does an outlier move a regression line?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ordinary least squares chases the three bad readings and reports a slope of 1.83 instead of 3. The Huber model gets 2.91 without deleting anything, and capping at the 5th and 95th percentiles recovers part of the damage (2.18). Notice that a plain |z| > 3 filter on y would have removed nothing here, because the bad readings are masked by the large spread of y: always look at outliers in the context of the relationship, for example through residuals.

Fit the rules on training data only

Percentiles, means and fences are statistics. Compute them on the training set and apply the same limits to validation and test data, exactly like scaling (see fit on the training data only). Computing fences on the whole dataset leaks information. Wrapping the clipping step in a pipeline transformer keeps this safe (see custom transformers).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A temperature of 31 ยฐC is normal in July but extreme in February. What kind of outlier is a February reading of 31 ยฐC?

    It is unusual only in its context (the month). Comparing it with the whole column would hide that.
  2. 2. Which statement about the IQR rule is true?

    Quartiles depend on the middle of the data, so the fences stay stable, and the rule needs no bell-shaped assumption.
  3. 3. Three huge values in a small sample are not flagged by the |z| > 3 rule. What is the best explanation?

    Masking: the outliers inflate the standard deviation, so no single point gets a z-score above 3. The modified z-score with the median and MAD resists this.
  4. 4. You are building a credit-card fraud model. What should you do with the extreme transaction amounts?

    In fraud detection the unusual rows carry the signal. Removing them removes the thing you want to find.

Summary

  • Outliers can be point, contextual or collective; column-wise rules find only point outliers.
  • The z-score rule (|z| > 3) assumes roughly bell-shaped data and can be masked by the outliers themselves.
  • The IQR rule and the modified z-score use quartiles or the median and resist masking.
  • Decide per case: fix, keep, cap, transform, use a robust model, or drop with a written reason.
  • Compute fences and caps on training data only, ideally inside a pipeline.

Next: turn raw signals and images into useful numbers with Feature Extraction.