Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 40 ยท Anomalies and Applications

Anomaly Detection in Time Series

Find outliers and unusual behaviour in time series: global and rolling z-scores, robust MAD scores, residual-based detection after removing trend and seasonality, Isolation Forest and alert design.

An anomaly is an observation or a stretch of observations that does not fit the usual behaviour of the series: a sudden spike in server errors, a sensor stuck at one value, a sales drop on a day with a broken checkout. Detecting them early is one of the most valuable uses of time series methods: fraud, equipment failure, outages and data-quality problems all show up as anomalies.

The hard part is that "unusual" depends on context. A value of 900 may be normal on Friday evening and a disaster on Monday morning. The central idea of this lesson is therefore:

First model what is expected (level, trend, seasonality), then flag points whose residual is unusually large.

Common types of anomaly:

  • Point anomaly: a single odd value (a spike).
  • Contextual anomaly: a normal value at the wrong time (high traffic at 3 a.m.).
  • Collective anomaly: a sequence that is odd as a whole (a flat line for an hour).
  • Level shift: the series jumps to a new level and stays (see change point detection).

Global z-score

The simplest rule: flag points more than k standard deviations from the mean. It works only when the series has no trend or seasonality.

Python Global z-score on a flat series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the weakness: the outliers themselves inflate the mean and standard deviation, which hides smaller anomalies (masking).

Robust scores: median and MAD

Replace the mean with the median and the standard deviation with the median absolute deviation (MAD). These barely move when outliers are present. The "modified z-score" uses 0.6745 * (x - median) / MAD.

Python Robust z-score with MAD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The robust score flags 30 of the 31 abnormal points (one episode reading happens to sit close to normal). The classic score finds only the single spike at index 200: the 30-point episode has dragged the mean up and inflated the standard deviation, so the episode no longer looks extreme. This effect is called masking.

Rolling baselines for drifting series

If the level drifts, compare each point with a rolling median and spread computed from the recent past. Use only past data (shift(1)) so the point being tested does not influence its own baseline.

Python Rolling median and MAD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A global z-score would see the drift itself as anomalous; the rolling baseline adapts to it.

Residual-based detection (the standard approach)

For seasonal data, remove the expected pattern first, with STL decomposition or any forecasting model, then apply a robust score to the remainder or forecast errors.

Python STL residuals plus MAD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A spike of 60 on a value that swings by 25 every week would be invisible to a global threshold; against the STL remainder the two injected anomalies score about 24 and 30 while the next largest remainder scores only about 7. robust=True keeps the outliers from distorting the decomposition. Choose the threshold from the distribution of scores on clean history: here a limit of 4 would flag a handful of ordinary days, because STL's seasonal estimate is not perfect, while 10 isolates the real events.

Isolation Forest

Isolation Forest is a machine learning detector: anomalies are easier to isolate with random splits, so they end up with short path lengths. For time series you feed it features (the value, differences, rolling statistics, hour of day), not just the raw value.

Python Isolation Forest on window features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The contamination setting is the expected share of anomalies and decides how many points are flagged, so choose it from domain knowledge rather than leaving it to chance.

Designing alerts that people trust

  • Threshold trade-off. A lower threshold finds more real problems and more false alarms. Pick it from the cost of each.
  • Require persistence. Alert only if the score exceeds the limit for n consecutive points when single spikes are harmless.
  • Seasonal context. Compute baselines per hour-of-week for traffic-like series.
  • Handle known events. Holidays and campaigns should be regressors (ARIMAX) or excluded, not alerted.
  • Do not train on anomalies. Clean or down-weight past outliers before fitting a forecasting model, otherwise the "normal" is polluted.
  • Evaluate. With labelled incidents use precision, recall and detection delay, not accuracy (anomalies are rare).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a global z-score a poor detector for a seasonal, trending series?

    The expected value moves over time, so the deviation from a single global mean mixes normal variation with anomalies.
  2. 2. What is the advantage of median and MAD over mean and standard deviation?

    Outliers inflate the mean and standard deviation (masking), but hardly move the median and MAD.
  3. 3. In rolling baselines, why use shift(1)?

    Using only past values avoids leakage and mirrors real-time detection.
  4. 4. What does the contamination parameter of IsolationForest represent?

    It sets the score threshold so that roughly that share of points is flagged.

Summary

  • Model the expected behaviour first, then flag unusually large residuals.
  • Use robust statistics (median, MAD) and rolling baselines built only from past data.
  • STL or a forecasting model handles seasonality; Isolation Forest works on engineered features.
  • Tune thresholds to the cost of false alarms, require persistence and evaluate with precision and recall.

Next: Change Point Detection