Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 41 ยท Drift Detection and Retraining

Detecting Data Drift

Compare live data with a reference using the Kolmogorov-Smirnov test, Population Stability Index, chi-square and a domain classifier, and learn when each is the right tool.

To detect data drift you need two samples: a reference (usually the training data or a healthy period) and a current window (the last day or week of live inputs). Then you ask: do these look like they come from the same distribution? Statistics gives several ways to answer, each with strengths and traps.

This lesson implements the most common drift measures with numpy, pandas and scipy: the two-sample KS test, the Population Stability Index (PSI), the chi-square test for categories, and a multivariate method that trains a classifier to tell the two samples apart.

Choosing a method

Which test for which data

Numeric KS test or PSI per feature.
Categorical Chi-square or share differences.
Many features Domain classifier on all columns at once.
Predictions Compare score distributions too.

Always monitor the model outputs as well as the inputs.

1 / 2

The same measure is not right for every column. A univariate test looks at one feature at a time and misses changes in the relationships between features. Multivariate checks catch those, but are harder to interpret.

The KS test for numeric features

The Kolmogorov-Smirnov (KS) test compares two samples through their cumulative distribution functions. The test statistic is the largest vertical gap between the two curves. It makes no assumption about the shape of the distribution. scipy.stats.ks_2samp returns the statistic and a p-value; a small p-value means the samples are unlikely to be from the same distribution.

Python KS test per feature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only income is flagged. The statistic itself (0 to 1) is also useful: it is the size of the difference, while the p-value only says how surprising it would be under "no difference".

Large samples make everything significant. With millions of rows, even a trivial and harmless shift gets a tiny p-value. Decide on a minimum effect size (a statistic threshold such as KS > 0.1 or PSI > 0.2) and use p-values only as a secondary check. Conversely, with very few rows a real shift may not reach significance.

Population Stability Index (PSI)

PSI is popular in credit risk and widely used in MLOps because it gives a single interpretable number. Split the reference feature into bins (often 10 quantile bins), compute the share of rows in each bin for both samples, and sum

PSI = sum over bins of (current share - reference share) x ln(current share / reference share)

Common rules of thumb: below 0.1 no significant change, 0.1 to 0.25 moderate change (watch it), above 0.25 major change (investigate). These are conventions, not laws; tune them for your domain.

Python PSI from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two implementation details matter. First, bin edges come from the reference only, so the same boundaries apply to both samples. Second, the eps clip prevents infinities when a bin is empty in one sample. Without it, a new category or an empty bin makes PSI blow up.

Categorical features

For categories, compare the counts of each value with the chi-square test, or simply watch the shares. Also track new categories never seen in training, because the model has no idea what to do with them.

Python Chi-square and unseen categories

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The chi-square test says the mix changed; the table tells you how. The unseen tablet category is arguably the more urgent finding.

Multivariate drift with a domain classifier

Each feature can look unchanged while the combination changes. Example: age and income individually keep their distributions, but their correlation flips. A neat trick handles this: label reference rows 0 and current rows 1, train a classifier to tell them apart, and measure its ROC AUC on held-out data. An AUC near 0.5 means the classifier cannot distinguish the samples (no drift); well above 0.5 means drift, and the feature importances say where.

Python Domain classifier detects joint drift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The univariate tests see nothing, while the classifier separates the samples easily because it can use the relationship between f1 and f2. The cost: it is a model to train and maintain, and its AUC needs a threshold such as 0.6 or 0.65 to become an alert.

Practical tips

  • Compare the right windows. A weekly window against the training data is the usual base case; for strongly seasonal data compare with the same weekday or season.
  • Use enough data per window. A few hundred rows give noisy comparisons; aggregate more, or lengthen the window.
  • Many features, many tests. Testing 200 features at the 5 percent level flags about 10 by chance. Use stricter thresholds, correct for multiple tests, or rank by effect size and review the top few.
  • Weight by importance. A drift in a feature the model hardly uses matters less. Rank alerts by feature importance times drift size.
  • Monitor predictions too. The model output score distribution is a one-dimensional summary of all inputs and is often the best single drift signal.
  • Keep the reference current. Update the reference when you deploy a retrained model, otherwise you compare against an obsolete baseline.

Tools

You can write all of the above in an afternoon, as this lesson did. Mature open-source tools package it with reports and dashboards: Evidently, NannyML, WhyLogs, Alibi Detect and Great Expectations, and the cloud platforms have built-in monitors. See the tooling landscape. Understanding the measures first helps you read their reports critically.

Common mistakes

  • Trusting p-values on huge samples, so everything is "drift".
  • No minimum effect size, which leads to alert fatigue.
  • Binning with the current data's edges (compare using the reference edges only).
  • Ignoring unseen categories and missing values. Track the share of nulls and unknown values as features of their own.
  • Reacting to drift without checking performance. Drift is a cause for investigation, not a verdict.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a PSI above 0.25 conventionally suggest?

    Below 0.1 stable, 0.1 to 0.25 moderate, above 0.25 major, as common rules of thumb.
  2. 2. Why can a KS test with 10 million rows flag a harmless shift?

    Use an effect-size threshold, not only a p-value.
  3. 3. Which method can detect that the correlation between two features flipped while each feature kept its distribution?

    Only a multivariate approach sees changes in relationships.
  4. 4. Which bin edges should be used when computing PSI?

    The same boundaries must apply to both distributions for the shares to be comparable.

Summary

  • Compare a reference sample (training or a healthy period) with a current window.
  • KS tests and PSI work for numeric features; chi-square and share tables for categories; watch for unseen categories.
  • Use effect-size thresholds, since p-values flag trivial shifts on large data.
  • A domain classifier catches joint (multivariate) drift and points to the responsible features.
  • Monitor predictions as well as inputs, and rank alerts by feature importance.

Next: Monitoring Model Performance