Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 41 ยท Drift Detection and Retraining

Retraining Triggers and Strategies

Decide when to retrain (schedule, drift, performance) and how (expanding, sliding window, weighted data), with simulations that compare the strategies in Python.

Models age. The question is not whether to retrain but when and on which data. Retrain too rarely and quality decays; retrain too often and you waste compute, add risk and chase noise.

This lesson compares the three families of triggers, then compares three ways to pick training data. Both comparisons are small simulations you can run and change.

When to retrain: three trigger families

Trigger Fires when Strength Weakness
Schedule A calendar event (nightly, weekly) Simple, predictable Retrains when unnecessary, misses sudden change
Drift signal Input or prediction distribution moves Works without labels Drift does not always hurt the model
Performance signal Measured metric falls below baseline Directly tied to value Needs labels, which arrive late

Real systems combine them: a weekly schedule as a safety net, drift alerts to react faster, and a performance floor as the final word. A trigger only starts the pipeline; the quality gate still decides whether the new model is deployed.

Python Comparing trigger policies on a drifting stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers are illustrative (this is a toy world), but the pattern is typical. The schedule retrained at day 30 when nothing had changed and only answered the day-50 shift ten days late. The threshold policy reacted on day 50 itself and then again at day 103 when slow decay crossed the line; the combined policy behaves the same here and adds a safety net for a world that never trips the threshold. The minimum gap (d - last >= 3) stops a trigger from firing every day while the metric stays low.

Which data to train on?

Once triggered, you must decide what the new training set contains.

flowchart TB
  S["Retraining data choices"] --> A["Expanding window: all history"]
  S --> B["Sliding window: last N days only"]
  S --> C["Weighted: all history, recent rows count more"]

Three common ways to assemble training data.

  • Expanding window: use everything. Best when the world is stable; slow to adapt when it is not.
  • Sliding window: use only the recent N days. Adapts fast but forgets rare events and has less data.
  • Weighted: keep all history but give newer rows larger sample_weight. A good compromise; the half-life is a parameter you can tune.
Python Expanding vs sliding vs weighted training data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the relationship rotates, old data is misleading, so the expanding window falls behind while the sliding and weighted strategies track the change. If the relationship were stable, the expanding window would win because it has the most data. There is no universal best: measure on your own history by replaying the past, retraining at each step and scoring on the following period. This is called backtesting.

Cost, risk and cadence

  • Compute and engineer time: an expensive model retrained daily may cost more than the quality it adds.
  • Risk: every new model is a release. Retraining often is only wise if the gates and rollouts are automated.
  • Label cost: if labels need human review, you may only be able to afford a monthly refresh.
  • Fine-tune or full retrain? Warm-starting from the old model is cheap but can carry old mistakes; a full retrain is clean but slow.

Fresh data alone is not a cure. If a data bug caused the drop, retraining on the broken data bakes the bug in; fix the source first.

A decision checklist

  1. Did a data or code bug cause the alert? Fix it, do not retrain.
  2. Is the shift temporary (holiday, outage)? Wait and watch.
  3. Is it persistent? Trigger the pipeline from Continuous Training Pipelines.
  4. Does the candidate pass the gates and win a canary? Promote it.
  5. Record the reason, metrics and data window for the audit trail.

Common mistakes

  • Retraining automatically on any drift alert with no quality gate.
  • Using a sliding window so short that rare but important cases vanish.
  • Never backtesting the retraining policy itself.
  • Forgetting that the new model's training data includes the old model's decisions (feedback loops), which can bias the data.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main weakness of a pure schedule trigger?

    A calendar knows nothing about the data, so it is either wasteful or late.
  2. 2. Why add a minimum gap (hysteresis) between triggered retrains?

    A retrain needs time to take effect and to produce fresh monitoring data.
  3. 3. When is a sliding window a better choice than an expanding window?

    Old rows describe an outdated relationship and can mislead the new model.
  4. 4. How can you choose a retraining policy for your own system?

    Backtesting measures the policy on your own data and drift pattern.

Summary

  • Triggers come from schedules, drift signals and performance signals; combine them with a minimum gap.
  • A trigger starts the pipeline; gates and rollouts still decide deployment.
  • Training data can be expanding, sliding or weighted; the right choice depends on how fast the world changes.
  • Backtest policies on history instead of guessing.
  • Fix data bugs before retraining on them.

Next: Automating the Retraining Loop