Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 41 ยท Drift Detection and Retraining

Understanding Data and Concept Drift

Learn why models decay after deployment: covariate shift, label shift and concept drift, sudden versus gradual change, and simulations showing how each one hurts accuracy.

A model is a frozen summary of the past. The day it is deployed, the world keeps moving: customers change behavior, a sensor is replaced, a competitor launches, a pandemic starts, a data pipeline changes a unit. The relationship the model learned slowly (or suddenly) stops matching reality. This decay is called drift, and it is the main reason models that were excellent on launch day get worse without a single line of code changing.

Understanding the kinds of drift tells you what to measure and how to react. This lesson defines the types, shows when each one hurts, and simulates them so you can see the effect on accuracy.

Three things that can change

A supervised model learns a relationship between inputs X and a label y. Three parts of that picture can change:

Types of drift

Data drift The inputs X look different. Also called covariate shift.
Label drift The share of each class changes. Prior shift.
Concept drift The rule from X to y changes.

Only concept drift changes what the right answer is.

1 / 2
  • Data drift (covariate shift): the distribution of inputs changes, P(X). Example: your loan model was trained on applicants aged 25 to 45, and a marketing campaign now brings in many 18-year-olds.
  • Label drift (prior probability shift): the distribution of the target changes, P(y). Example: fraud was 0.5 percent of transactions and becomes 2 percent during an attack.
  • Concept drift: the relationship between inputs and target changes, P(y given X). Example: after a policy change, the same income and debt now lead to different default rates. The model's "rule" is simply out of date.

There are also upstream data problems that look like drift but are really bugs: a column renamed, a unit changed from dollars to cents, a feature that is suddenly always zero. These are covered by data validation and should be fixed at the source, not by retraining.

Not all drift is harmful. If inputs shift into a region where the model is still correct, accuracy may not change at all. Drift is a signal to investigate, not an automatic reason to retrain.

Data drift: does it matter?

The next simulation trains a model on data where the true rule is not a straight line. Then it scores two shifted populations. In one, inputs move to a region the model has already seen enough of; in the other, to a region where training had almost no data.

Python Covariate shift: sometimes harmless, sometimes not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here the rule never changes, yet accuracy collapses in the new region. The reason: a model only knows the part of the world it saw. This is why you monitor input distributions, not only accuracy.

Concept drift: the rule itself changes

Now keep the inputs identical and change the relationship. A fraud model learned that large night-time transactions are risky. Later, fraudsters adapt and the risky pattern becomes small daytime transactions. The inputs look the same; the right answer changed.

Python Concept drift with unchanged inputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The input statistics before and after are practically the same, so a monitor that watches only inputs sees nothing. Concept drift is invisible without labels (or a good proxy for them). That is why performance monitoring matters in addition to input monitoring.

Label drift

If the base rate of the target changes, a model that outputs calibrated probabilities becomes miscalibrated and a fixed decision threshold becomes the wrong one. Even with an unchanged rule, the mix of positives and negatives changes precision (the share of alerts that are real).

Python Label drift changes precision, not recall

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classifier is exactly the same in all three rows, and recall barely moves, but precision falls as positives become rare. A team watching only recall would miss that analysts are now drowning in false alarms.

Causes you should recognize

  • Real-world change: behavior, seasons, economy, regulation, competitors.
  • Feedback loops: the model's own decisions change the data (recommend more of one genre and people watch more of it).
  • Instrumentation changes: a new app version logs events differently, a sensor is recalibrated.
  • Pipeline bugs: a join breaks, a default value starts appearing, a field is dropped.
  • Population change: a new market, a new customer segment, a new device type.
  • Adversaries: fraudsters, spammers and attackers adapt to your model deliberately.

Where the evidence comes from

Different drift types need different signals:

Drift type Needs labels? Main signal
Data drift No Compare input distributions (training vs live)
Label drift Sometimes Predicted class share, or true labels when they arrive
Concept drift Yes (or a proxy) Live accuracy, precision, recall, business metrics

Labels often arrive late (a loan default is known months later). Until they do, you rely on input and prediction monitoring as an early warning. The next lessons show how to detect data drift and track performance when labels are delayed.

Common mistakes

  • Treating every shift as an emergency. Measure the effect on performance first.
  • Retraining on bugged data. Fix upstream data problems before retraining.
  • Watching only accuracy. It lags; input checks warn earlier.
  • Ignoring seasonality. Compare with the same period last year, not only last week.
  • No reference data. Keep the training distribution (or a snapshot) so you have something to compare against.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model trained on adults now receives many teenage users. Which drift is this?

    The distribution of the inputs changed, which is covariate shift.
  2. 2. Why can concept drift be hard to detect from inputs alone?

    You need labels or a proxy such as downstream business results.
  3. 3. Fraud rises from 0.5 percent to 2 percent of transactions. What changed?

    That is label (prior probability) drift, and it affects precision and calibration.
  4. 4. What should you do before retraining after you notice a sudden shift?

    Retraining on broken data bakes the bug into the new model.

Summary

  • Drift is the gap that grows between the data a model learned from and the world it serves.
  • Data drift changes inputs, label drift changes class proportions, concept drift changes the rule from inputs to labels.
  • Drift can be sudden, gradual, seasonal or a short incident; some drift is harmless.
  • Input monitoring warns early without labels; performance monitoring is needed for concept drift.
  • Distinguish true drift from data bugs before reacting.

Next: Detecting Data Drift