Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 41 ยท Drift Detection and Retraining

Automating the Retraining Loop

Close the MLOps loop in code: monitor a live model, trigger retraining, gate the candidate, promote or reject it, roll back on regression and keep an audit log.

So far you have seen the parts separately: monitoring, drift tests, triggers, pipelines, gates and rollouts. A mature MLOps system connects them into a closed loop that can notice trouble, build a fix, check the fix and ship it, with people supervising rather than pushing every button.

This lesson builds that loop end to end in a few dozen lines of Python, using a simulated stream of data whose relationship slowly changes. The goal is not production code but a clear picture of how the parts fit and where the safety valves go.

The loop

flowchart LR
  M["Monitor live model"] --> T{"Trigger fired?"}
  T -- no --> M
  T -- yes --> R["Retrain on recent data"]
  R --> G{"Quality gate"}
  G -- fail --> L["Log rejection, alert a human"]
  G -- pass --> P["Promote candidate to champion"]
  P --> W["Watch post-deploy metrics"]
  W -- regression --> B["Roll back"]
  W -- ok --> M
  L --> M
  B --> M

Every branch ends by writing to the audit log and returning to monitoring.

Four safety valves keep a self-updating system from making things worse: a cooldown between retrains, a gate before promotion, a rollback after promotion and an audit log of every decision.

A complete simulation

Python The closed retraining loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the audit log: each line says when, why and what happened. That log is as important as the model, because it is the only way to answer "why did the model change last Tuesday?"

Add a rollback

A candidate that passed offline gates can still fail in production. After promotion, watch live metrics for a short bake period; if they fall below the previous champion's recent level, return to it.

Python Rollback after a bad promotion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rollback must be cheap and automatic: keep the previous model artifact and its configuration available, and make switching a single registry change, not a rebuild.

What a human still decides

Automation should have limits. Common choices:

  • Fully automatic for low-risk models (recommendations) with strong gates and canaries.
  • Automatic training, manual promotion when decisions affect money or safety: the pipeline prepares the candidate with a report and a person approves.
  • Escalation rules: two rejected candidates in a row, or a rollback, should notify a human instead of retrying forever.

Guardrails checklist

  • Cooldown: a minimum time between retrains.
  • Budget: a cap on compute spend or number of runs per week.
  • Data checks first: never retrain on data that failed validation (see Data Validation in CI).
  • Gate and canary: see Model Quality Gates and Safe Rollouts.
  • Audit log: reason, data window, metrics, decision, who or what approved.
  • Kill switch: one switch pauses all automation.

Common mistakes

  • No cooldown, so a noisy metric causes a retraining storm.
  • Evaluating the candidate and champion on different data.
  • Promoting on a tiny improvement that is only noise.
  • Retraining on data produced by the model's own decisions without thinking about feedback loops.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the loop include both a gate and a post-deploy rollback?

    Offline metrics are a proxy. Real traffic can behave differently, so the loop keeps watching after promotion.
  2. 2. What is the purpose of the audit log?

    Without a record you cannot explain or debug past decisions.
  3. 3. A cooldown between retrains protects against:

    A retrain needs time to take effect; firing continuously wastes compute and adds risk.
  4. 4. When might you choose automatic training with manual promotion?

    High-impact models usually keep a human approval step even when the build is automatic.

Summary

  • A closed loop connects monitoring, triggers, retraining, gates, promotion and rollback.
  • Safety valves: cooldown, gate, bake-and-rollback, budget and an audit log.
  • Rollback should be one cheap registry change.
  • Choose how much to automate by the cost of a wrong decision.
  • Every decision, including rejections, belongs in the log.

Next: Model Cards and Documentation