Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 41 ยท Drift Detection and Retraining

Monitoring Model Performance

Track live model quality when ground truth arrives late: rolling metrics, label delay, proxy signals, slice monitoring and alert rules with worked Python examples.

Drift tests tell you that inputs look different. They cannot tell you whether the model is still right. Only performance monitoring, comparing predictions with what actually happened, answers that. The catch is that the true outcome often arrives days or months after the prediction.

This lesson shows how to compute live metrics when labels are delayed, what to watch while you wait, and how to turn the numbers into alerts that people trust.

The label delay problem

A fraud model scores a payment now, but the chargeback arrives six weeks later. A churn model predicts cancellation next month. A recommendation model gets clicks within seconds. Each domain has a different label delay, and it decides which monitoring strategy you can use.

flowchart LR
  A["Prediction logged"] --> B["Wait: label delay"]
  B --> C["True outcome arrives"]
  C --> D["Join on request id"]
  D --> E["Compute metric for that prediction window"]

Metrics are always computed for the window in which the prediction was made, not when the label arrived.

Three practical rules follow:

  1. Log a request id with every prediction so outcomes can be joined later (see Building a Prediction Service).
  2. Group by prediction date, not label date. Last Tuesday's accuracy is final only after Tuesday's labels are in.
  3. Report coverage next to the metric: "accuracy 0.91 on the 62 percent of Tuesday's predictions that already have labels."

Rolling metrics with delayed labels

Python Daily accuracy as labels arrive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On day 29 the newest days have no labels at all, and days 20 to 26 are only partly labelled. The quality drop that began on day 20 is real but only visible for days that already have labels. That is why teams add early-warning signals that do not need labels.

Signals you can see without labels

While labels are late, watch things that move first:

  • Prediction distribution: the share of positive predictions, the mean score, the histogram of scores.
  • Input drift: the tests from Detecting Data Drift.
  • Confidence: a calibrated model's mean confidence estimates its accuracy.
  • Business proxies: approval rate, click-through rate, manual-review rate, the share of requests that fall back to a default.
  • Data quality: missing values, out-of-range values, new categories.
Python Estimating accuracy from confidence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the three lines carefully. Confidence tracks accuracy well in normal conditions. Under concept drift the model is just as confident but wrong, so the estimate misses the damage entirely. Confidence-based estimates are an early hint, never a replacement for real labels.

Monitor slices, not only averages

An average can stay flat while one group collapses. Compute the same metric per slice: country, device, customer segment, product category, new versus returning users. Keep the list short and meaningful, and report the sample size of each slice, because small slices produce noisy numbers.

Turning metrics into alerts

  • Compare with a baseline, for example the metric measured at release or over the last stable 30 days, not with a fixed number you invented.
  • Use a window and a margin. "Seven-day accuracy more than 3 points below baseline for two consecutive days" is calmer than "accuracy below 0.9 today".
  • Add a minimum sample size so tiny windows cannot page anyone.
  • Alert on trends, too. A slow slide of half a point a week never crosses a threshold but costs money.
  • Route by severity. A page for a service outage, a ticket for a gradual decline.

Retraining decisions built on these alerts are the topic of Retraining Triggers and Strategies.

Common mistakes

  • Computing accuracy on whatever labels have arrived, which over-weights quick-to-label cases and makes recent days look better than they are.
  • Using accuracy for rare events; use precision, recall or a cost-weighted metric.
  • Alerting on a single noisy day.
  • Treating a stable proxy as proof the model is fine.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Metrics for delayed labels should be grouped by:

    You want to know how the model did on the predictions of a given period, regardless of when outcomes were reported.
  2. 2. Why report coverage next to a delayed metric?

    An accuracy computed from 5 percent of a day's predictions is far less reliable than one from 95 percent.
  3. 3. Under concept drift, why can confidence-based estimates fail?

    Inputs look normal, so the model is as sure as before, but its learned rule no longer matches reality.
  4. 4. Which alert rule is most robust against noise?

    Windows, margins, persistence and minimum sample sizes keep alerts meaningful.

Summary

  • Real performance needs ground truth, which often arrives late; join outcomes by request id and group by prediction date.
  • Report coverage with every delayed metric.
  • Watch label-free signals (score distribution, drift, confidence, business proxies) as early warnings, and remember they miss concept drift.
  • Compute metrics per slice with sample sizes.
  • Alert on baselines, windows, margins and minimum counts.

Next: Retraining Triggers and Strategies