Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 39 ยท Anomaly Detection

Outlier Detection: Statistical Methods

Detect outliers and anomalies with statistics: types of outliers, z-score, modified z-score (MAD), the IQR rule, Grubbs test, Mahalanobis distance, Gaussian fitting and what to do with outliers.

An outlier (or anomaly) is a record that deviates so much from the others that it raises the suspicion it was generated by a different process. Sometimes outliers are errors to remove (a sensor glitch, a typo of 1,000 years of age). Sometimes they are the whole point: fraud, network intrusions, faulty machines, rare diseases. Outlier detection finds them, and it is one of the main data mining tasks.

Unlike classification, you rarely have many labeled anomalies. Anomalies are rare by definition, so most methods are unsupervised: define "normal" from the data, then flag what is far from normal.

Types of outliers

  • Global (point) outliers: a single value far from the rest, like a 9,000 USD coffee purchase.
  • Contextual outliers: normal in general but abnormal in context. A temperature of 30 C is fine in July and an outlier in January. Context can be time, place or user.
  • Collective outliers: a group of records that is abnormal together although each member might look normal, such as a flat line of identical heartbeat values.

The z-score

The simplest model assumes a roughly normal distribution. The z-score z = (x - mean) / std says how many standard deviations a value is from the mean. A common rule flags |z| > 3.

Python Z-score and its weakness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mean and standard deviation are themselves distorted by outliers, so a few of them can mask each other: in the small sample only one of the three obvious outliers crosses the line (and barely). That motivates robust statistics.

Robust methods: median, MAD and IQR

  • Modified z-score: use the median and the MAD (median absolute deviation) instead of mean and std: M = 0.6745 * (x - median) / MAD. Flag |M| > 3.5.
  • IQR rule (Tukey fences): with quartiles Q1, Q3 and IQR = Q3 - Q1, flag values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR (use 3 for "far out"). This is exactly how a box plot draws its whiskers. See visualizing data.
Python Compare z-score, MAD and IQR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

MAD and IQR are not fooled by the three outliers, because quartiles and the median ignore extreme values.

Grubbs' test

Grubbs' test is a hypothesis test for a single outlier in normal data. The test statistic is G = max|x - mean| / std; it is compared with a critical value from the t-distribution. If G exceeds the critical value, the most extreme point is declared an outlier at the chosen significance level; remove it and repeat if you want to look for more.

Python Grubbs' test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multivariate outliers: Mahalanobis distance

A record can be ordinary in every single column and still unusual in combination: a person 1.90 m tall weighing 45 kg. The Mahalanobis distance measures distance from the center in units that account for the spread and the correlation of the features: D^2 = (x - mu)^T S^-1 (x - mu). For normally distributed data D^2 follows a chi-square distribution with p degrees of freedom, so you can pick a cutoff such as the 99.9th percentile.

Python Mahalanobis versus Euclidean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The point is only about 2.1 units from the center, yet its Mahalanobis distance is large because it breaks the correlation pattern.

Parametric and non-parametric detection

  • Parametric: assume a distribution (normal, Gaussian mixture), fit it, and flag records with very low probability density. Works well when the assumption holds.
  • Non-parametric: make no distribution assumption; use histograms, kernel density estimates or the distance and density methods in the next lesson.

What to do with outliers

  1. Investigate first: is it an error, a rare valid case or the very thing you search for?
  2. Fix or remove data errors; keep real extremes or treat them with robust methods, caps (winsorizing) or transformations such as log.
  3. Document every removal. Deleting inconvenient data silently is a classic way to distort a result.

For cleaning workflows see data cleaning.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A temperature of 30 C in January in Oslo is which type of outlier?

    The value is normal in other contexts (summer) but abnormal in this one.
  2. 2. Why can the z-score fail with several large outliers?

    Mean and std are not robust, so extreme values change them and hide themselves.
  3. 3. Which values does the 1.5 * IQR rule flag?

    Tukey's fences are placed 1.5 IQRs beyond the first and third quartile.
  4. 4. What does Mahalanobis distance take into account that Euclidean distance does not?

    It rescales by the covariance matrix, so a point that goes against the correlation structure scores high.

Summary

  • Outliers can be global, contextual or collective; they are errors or valuable signals, so investigate before deleting.
  • Z-scores are simple but fragile; MAD and the IQR rule are robust; Grubbs' test checks one outlier statistically.
  • Mahalanobis distance finds multivariate outliers that are normal in every single column.

Next: Distance and Density Anomaly Detection