Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 40 ยท Data Preparation

Outliers and Data Cleaning

Find outliers with the IQR rule, IsolationForest, LocalOutlierFactor and EllipticEnvelope, decide whether to remove or clip them, and clean duplicates and constant columns.

An outlier is a value far from the rest. It may be a typing mistake (age 420), a genuine rare event (a billionaire customer) or a sign of a different population. Outliers can pull a linear model off course and distort scalers, so a modelling workflow needs a way to find and handle them. This lesson covers detection, the choices for handling, and a few other cleaning chores.

The IQR rule

The simplest detector uses quartiles. A value is flagged if it lies more than 1.5 interquartile ranges (IQR) beyond the first or third quartile.

Python IQR rule with NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This works one column at a time. It will not spot a row that is unusual only because of combinations of values, such as a 12-year-old with an income of $200,000. For that you need multivariate detectors.

IsolationForest

IsolationForest builds random trees that split the data. Rare points are separated in fewer splits, so a short path means "outlier". predict returns -1 for outliers and 1 for normal points.

Python IsolationForest

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

contamination is your estimate of the share of outliers. If you do not know it, "auto" is a reasonable start, and you can instead use iso.score_samples(X) to rank rows by how unusual they are.

LocalOutlierFactor: unusual for its neighbourhood

LocalOutlierFactor compares a point's local density with that of its neighbours, so it can find points that are outliers relative to a dense cluster even when they would be normal in a sparse one. Because it has no predict for new data by default, use fit_predict on the data you are checking (or novelty=True for new data).

Python LocalOutlierFactor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

EllipticEnvelope: assume a bell shape

EllipticEnvelope fits a Gaussian (a smooth oval) to the central data and flags points far outside it. It is fast and effective when the data really is roughly bell-shaped, and poor otherwise.

Python EllipticEnvelope

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The planted point is not extreme on either axis alone; it breaks the correlation between the columns, and the detector sees that.

What to do with outliers

flowchart TD
    A["Outlier found"] --> B{"Real error?"}
    B -->|"yes"| C["Fix it or drop the row"]
    B -->|"no"| D{"Does it matter to the model?"}
    D -->|"no, tree model"| E["Keep it"]
    D -->|"yes, linear or distance model"| F["Clip, transform or use a robust method"]

Never delete points just because they hurt your score. Investigate first.

Three safe ways to reduce their influence:

  • Clip (winsorise) values to a percentile range.
  • Transform with a log or power transform (previous lesson).
  • Use robust methods: RobustScaler, the median, tree models or HuberRegressor.
Python Clipping to percentiles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compute the clipping limits on the training data and reuse them for test data, as with any other preprocessing.

Other cleaning chores

Python Duplicates and constant columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Also check that numbers were not read as text, that units are consistent (cm versus inches) and that impossible values (negative ages) are handled. Cleaning is not glamorous, but it often improves a model more than changing the algorithm.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the IQR rule, a point is an outlier if it lies beyond which limits?

    The conventional fences are 1.5 interquartile ranges below Q1 and above Q3.
  2. 2. What does IsolationForest.predict return for an outlier?

    Outliers are labelled -1 and inliers 1.
  3. 3. Why can multivariate detectors find rows that column-by-column checks miss?

    A row can be ordinary in every single column and still be unusual as a combination.
  4. 4. Which is the best first response to a suspected outlier?

    Removing real rare cases hides the truth. Decide from understanding of the data.

Summary

  • The IQR rule flags per-column outliers; IsolationForest, LocalOutlierFactor and EllipticEnvelope find multivariate ones.
  • Detectors return -1 for outliers.
  • Handle them by fixing, clipping, transforming or using robust models; do not delete blindly.
  • Remove duplicates and constant columns, and check units and types.

Next: Linear Regression