Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 41 ยท Projects and Practice

Project: A Drift Monitoring Report

Build a weekly drift and performance report: PSI and KS per feature, prediction shift, delayed-label accuracy, alert rules and a chart you can run in the browser.

In this project you will build the kind of weekly report a model owner reads on Monday morning: which features moved, whether predictions shifted, what the delayed labels say about accuracy, and which alerts fired. The data is simulated so the story is known in advance (a sensor starts to drift at week 8 and the real relationship changes at week 12), which lets you check that your monitor finds what it should.

The pieces come from earlier lessons: Understanding Drift, Detecting Data Drift and Monitoring Model Performance.

Step 1: simulate a model and 20 weeks of traffic

Python Step 1: the model and the weekly stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: the drift measures

Python Step 2: PSI and KS functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rule of thumb: PSI below 0.1 is stable, 0.1 to 0.25 deserves a look, above 0.25 is a major shift. These cutoffs are conventions, not laws; tune them against false alarms in your own data.

Step 3: the weekly report

Python Step 3: run the monitor over 16 weeks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the printed alerts with the planted story. The temperature drift begins at week 8: PSI climbs into the "watch" range at week 9 and crosses the alert line at week 11, with no labels needed. The relationship change at week 12 leaves the input distributions untouched, so only the labelled AUC catches it, and only at week 14 because of the label delay (the alert says it is about week 12). That is the whole argument for monitoring both inputs and performance.

Step 4: a chart

Python Step 4: plot drift and accuracy together

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The chart uses illustrative numbers shaped like the report above; replace them with columns from rep to plot your real monitor.

Make it your own

  1. Add a KS test column and compare when it fires versus PSI.
  2. Add a minimum sample rule so tiny batches cannot alert.
  3. Require two consecutive weeks before an alert becomes a page.
  4. Add the prediction score distribution (PSI on proba).
  5. Connect the alert to the trigger policy in Retraining Triggers and Strategies.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why did input PSI not detect the week-12 change?

    Concept drift changes how inputs map to outcomes, not the inputs themselves.
  2. 2. Why is the AUC for a week computed two weeks later in the report?

    You can only evaluate predictions after their true outcomes are known.
  3. 3. A PSI of 0.4 on one feature most likely means:

    Above about 0.25 is conventionally a major shift.
  4. 4. What is the benefit of requiring two consecutive weeks before paging?

    Persistence filters out random blips.

Summary

  • A drift report combines per-feature PSI or KS, prediction shift and delayed-label performance.
  • Input monitoring warns early; labelled metrics confirm and catch concept drift.
  • Alert rules need thresholds, persistence and minimum sample sizes.
  • Simulated data with a known story is a good way to test a monitor.

Next: Practice: MLOps Exercises