Skip to content
Tutorials Center
My learning

Topics

Lesson 81 of 111 ยท Beyond the Basics

MLOps and Model Monitoring

Run models in production: the MLOps lifecycle, serving patterns, drift and performance monitoring with delayed labels, and when to retrain, in runnable Python.

Training a model is the middle of the job, not the end. A model that scored well in a notebook has to be packaged, deployed, watched and eventually replaced, and it must keep working while the world around it changes. MLOps (machine learning operations) is the set of practices that makes this repeatable: version everything, automate the steps, monitor what is live, and have a plan for when quality drops.

The earlier lesson Saving Models and Deploying covered the mechanics: pickle and joblib, a prediction function, training/serving skew, and the basic PSI and KS drift checks. This lesson builds on it and does not repeat it. We look at the whole lifecycle, the common serving patterns, and the part that surprises most newcomers: how to monitor a model when the true answers arrive weeks late, and how to decide when retraining is worth it. The runnable examples simulate a model running for several weeks while the data and the world drift.

The MLOps loop

Build Data, features, training, tests.
Ship Package and deploy safely.
Watch Monitor data, predictions, outcomes.
Retrain Replace the model when it decays.
1 / 2

The lifecycle

flowchart LR
    D["Data and features"] --> T["Train and evaluate"]
    T --> R["Model registry"]
    R --> S["Deploy and serve"]
    S --> M["Monitor"]
    M -->|"drift or decay"| D
    M -->|"healthy"| S

Production ML is a loop: monitoring feeds back into data collection and retraining.

Each stage has a habit that saves pain later:

  • Reproducible training. Keep the code in version control, record the data snapshot (a date, a hash or a table version), fix random seeds, and log the parameters and metrics of every run. Tools such as MLflow or Weights & Biases do this; a simple JSON file next to the model artifact is a fine start.
  • Model registry. A registry is a catalog of trained models with a version, its metrics, the data it was trained on, and a stage (staging, production, archived). Deployment then means "promote version 7", and rollback means "promote version 6 again".
  • Automated tests. Besides unit tests on feature code, test the model: it must beat a baseline, predictions must stay in a valid range, and a few known examples must give known answers.
  • CI/CD for ML. A pipeline retrains, validates and, if the new model passes the checks, deploys it with no manual steps. The same pipeline can run on a schedule or when monitoring raises an alert.
  • Feature consistency. The same code must compute features in training and in serving. A shared feature function or a Pipeline removes the main source of silent bugs.

Serving patterns

How a model is used decides how it is deployed.

Pattern How it works Typical use Watch out for
Batch A scheduled job scores many rows and stores the results Nightly churn scores, weekly recommendations Results are stale until the next run
Online (real-time) A web service scores one request in milliseconds Fraud checks, search ranking Latency, scaling, input validation
Streaming Events are scored as they flow through a message queue Sensor alerts, live monitoring Ordering, late events
Edge The model runs on the device Phone keyboards, cameras Model size, updates

Start with batch whenever the business allows it. It is the simplest to run and to debug, and the monitoring below is easier too. A read-only sketch of an online service looks like this:

Python
# Read-only: FastAPI is not available in the browser runner.
from fastapi import FastAPI
from pydantic import BaseModel
import joblib

app = FastAPI()
model = joblib.load("models/churn-v7.joblib")          # a registered, versioned artifact

class Customer(BaseModel):
    tenure_months: float
    monthly_charge: float
    support_calls: int

@app.post("/predict")
def predict(customer: Customer):
    score = float(model.predict_proba([[customer.tenure_months,
                                        customer.monthly_charge,
                                        customer.support_calls]])[0, 1])
    log_prediction(customer, score, model_version="v7")   # keep inputs and outputs for monitoring
    return {"churn_probability": score, "model_version": "v7"}

Notice log_prediction. If you do not store what the model saw and what it answered, you cannot monitor anything later. Keep the model version with every log row.

Releasing a new version safely

A new model that wins offline can still fail live. Three patterns reduce the risk:

  • Shadow mode. The new model receives real traffic and its predictions are logged, but users only see the old model's answers. You compare the two for free.
  • Canary release. Send a small share (say 5%) of traffic to the new model, watch the metrics, then increase.
  • A/B test. Split users randomly and compare a business metric, such as conversion, which is the final judge when predictions change behavior.

Always keep the previous version ready to roll back to.

What to monitor

Monitoring has layers, ordered from fastest and cheapest to slowest and most truthful:

  1. System health. Latency, errors, memory, throughput. Ordinary software monitoring.
  2. Data quality. Missing values, out-of-range values, new categories, schema changes. A broken upstream feed often shows up here first.
  3. Input drift. Has the distribution of the features changed since training? (Data drift.) Needs no labels.
  4. Prediction drift. Has the distribution of the model's outputs changed, for example the share of positive predictions? Needs no labels either.
  5. Performance. Accuracy, precision, error, using the true outcomes. The only direct measure of quality, but the labels often arrive late: a loan default is known months later, a churn a month later.

When labels are delayed, layers 3 and 4 are your early warning, and layer 5 is the confirmation. A subtle point is that the two kinds of drift behave differently. Data (covariate) drift changes the inputs; the model may or may not suffer. Concept drift changes the relationship between inputs and outcome (customers react to price differently after a competitor appears); the inputs can look perfectly normal while the model gets steadily worse. Only performance monitoring catches the second kind.

The next example simulates both. A logistic regression is trained on week 0. From week 3 one feature drifts upward (data drift); from week 6 the effect of another feature slowly fades and then reverses (concept drift). Each week we compute the PSI of every feature against the training data (the same psi function as in the previous lesson), the share of positive predictions, and the accuracy, as it would be known once the labels arrive.

Python Watching a model decay over 12 weeks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the alerts in the order they fire. The prediction shift alert appears first, in week 4, when the share of positive predictions has risen from 44% to 56%. The input drift alert follows in week 5 (PSI of feature 0 is 0.52, far above the 0.25 rule of thumb), while features 1 and 2 stay flat at about 0.01. Accuracy only crosses the 0.85 line in week 7, and in reality you would learn that only when the labels arrive, weeks after the drift began. So the label-free signals gave a two-to-three-week head start. Notice also what PSI cannot show: the weakening of feature 1 (the concept drift from week 6) leaves its histogram untouched, so the PSI for that feature is flat the whole time. In this run both kinds of drift happen together, so accuracy reflects the sum of them; in practice you would investigate to tell them apart.

Drift that single features cannot see

Per-feature checks such as PSI treat each column alone. They miss changes in the relationships between columns: two features that used to move together may become independent while each one keeps exactly the same histogram. A general method that does not care how the drift looks is the domain classifier (or adversarial validation): label the reference data as 0 and the new data as 1, then train a model to tell them apart with cross-validation. If it cannot (AUC near 0.5), the two samples look alike. If it can (AUC well above 0.5), something changed, and the model's feature importances tell you which features give it away.

In the example, the second data set has the same mean and spread for every feature, and the same marginal histograms, but the correlation between the first two columns has dropped from 0.8 to 0.

Python Domain classifier for multivariate drift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With nothing changed, the classifier cannot tell the samples apart (AUC 0.52, chance is 0.5). After the correlation was destroyed it separates them with an AUC of 0.73, and the importances show features 0 and 1, the pair whose relationship changed, carrying most of the signal while feature 2 carries least. Look at the per-feature KS statistics: they are smaller for the decorrelated data (about 0.03) than for the unchanged data (0.06 for the first two features), which is just sampling noise. Per-feature tests cannot see this drift at all, and the domain classifier can.

Deciding when to retrain

An alert is not an order. Ask three questions first: Is it a data problem (a broken feed, a unit change) that should be fixed upstream? Is it real drift that the current model can still handle? Or is it a real performance loss? Only the last two lead to retraining, and only the last is certain to need it.

Common retraining policies:

  • Never (manual): cheap, and fine for stable problems, but silent decay goes unnoticed.
  • Scheduled: retrain every week or month regardless. Simple and predictable; wastes effort when nothing changed and reacts slowly when something breaks.
  • Triggered: retrain when monitoring crosses a threshold, such as a performance drop or a large drift score. Efficient, but depends on good thresholds and on labels arriving in time.
  • Continuous / online: update the model with each batch of new data. Powerful, and risky, because bad data can poison it immediately.

A retrained model also needs validation before promotion: compare it with the current model on recent labeled data, and only promote it when it is clearly better.

The last example compares three policies over 20 simulated weeks of the same drifting world. Labels arrive with a one-week delay, and every retrain uses the most recent two weeks of labeled data. The trigger policy retrains when last week's accuracy falls under 0.85. We count both the quality (mean accuracy) and the cost (number of retrains).

Python Retraining policies compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Never retraining lets the model decay to a mean accuracy of 0.818 with a worst week of 0.770. Both active policies end at a mean of 0.875: retraining every 4 weeks takes 5 retrains, while the accuracy trigger needs only 3, because it retrains when something is actually wrong. The numbers depend on this particular simulation and on the 0.85 threshold, and a trigger that fires too late or too often costs more in real life, but the pattern is typical: some retraining beats none, and monitoring lets you retrain less often for the same quality.

Retraining on its own predictions or on data influenced by the model creates feedback loops: a recommender trained only on what it recommended learns to recommend more of the same. Keep some exploration, random traffic, or a holdout group scored by a different method.

A monitoring checklist

  • Log every request: inputs, output, model version, timestamp (respecting privacy rules; hash or drop personal data).
  • Track data quality, input drift and prediction drift on a schedule, per feature, and keep the history to spot slow creep.
  • Collect true outcomes as soon as they exist, join them to the logged predictions, and compute the real metric.
  • Set alert thresholds in advance and name an owner; an alert nobody reads is the same as no alert.
  • Write the rollback and retrain procedure down and rehearse it once.
  • Review fairness metrics on live data too: drift can hurt some groups more than others.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Labels for your model arrive six weeks late. Which monitoring signal is available immediately?

    Drift checks on inputs and on the model's own predictions need no true labels, so they work as an early warning. Performance metrics need the outcomes.
  2. 2. Inputs look unchanged, but the model's accuracy keeps falling. What is the most likely cause?

    If the feature distributions are stable but errors grow, the world changed what the features mean for the outcome, which is concept drift. Only performance monitoring catches it.
  3. 3. What does a domain classifier AUC of about 0.5 tell you?

    If a model cannot beat chance at telling old from new rows, the samples look alike. A high AUC means something changed, even if each feature alone looks the same.
  4. 4. What is shadow mode?

    Shadow mode lets you compare a new model with the live one on real inputs without any risk to users. Sending a share of traffic to the new model is a canary release.

Summary

  • MLOps makes ML repeatable: version data, code and models, use a registry, test models, automate deployment and keep the ability to roll back.
  • Choose batch, online, streaming or edge serving by business need; prefer batch when possible, and always log inputs, outputs and model version.
  • Release safely with shadow mode, canary releases or A/B tests.
  • Monitor in layers: system health, data quality, input drift, prediction drift, then performance once labels arrive.
  • Data drift and concept drift are different; only performance monitoring catches the second.
  • Per-feature checks miss changed relationships; a domain classifier catches multivariate drift.
  • Retrain by schedule or by trigger, validate before promotion, and beware of feedback loops.

You have reached the end of the "Beyond the Basics" chapter. Continue with Fairness and Ethics or test yourself with the projects.