Skip to content
Tutorials Center
My learning

Topics

Lesson 61 of 76 ยท Reproducibility and Production Habits

From Model to Decision and Deployment

Turn a trained model into something useful: batch vs real-time scoring, saving models with joblib, choosing a decision threshold by cost, and monitoring drift.

A model in a notebook helps nobody. It becomes valuable when its predictions change what someone does: a retention team calls the customers most likely to leave, a warehouse orders stock, a bank reviews a flagged payment. That step from "model that scores well" to "decision that gets made, and keeps working" is called deployment, and it is mostly not about servers. It is about three questions: how will predictions be delivered, what action follows from a score, and how will you notice when the model stops being right?

This lesson answers them with small, runnable examples. Real deployment infrastructure (web servers, containers, schedulers) cannot run in the browser, so those parts are shown as read-only sketches. The data-science parts, such as saving a model, picking a threshold and checking for drift, run live. Evaluation metrics are covered in Evaluating Models with the Right Metrics; here we use them to decide.

flowchart LR
    A["Train and validate"] --> B["Save model<br/>+ metadata"]
    B --> C["Score new data<br/>(batch or real time)"]
    C --> D["Apply decision rule<br/>(threshold, action)"]
    D --> E["Monitor inputs<br/>and outcomes"]
    E -->|"drift or decay"| A

Deployment is a loop: monitoring tells you when to retrain.

Batch or real time?

The first design choice is when predictions are needed.

  • Batch scoring runs on a schedule: every night, score all customers and write the results to a table. It is simple, cheap and easy to test, because the whole input is known in advance. Most business use cases (churn lists, demand forecasts, lead scoring) fit this pattern.
  • Real-time scoring answers one request at a time, in milliseconds: fraud checks at payment, recommendations on a page. It needs a running service, fast feature lookups and careful latency and error handling.

Start with batch unless a decision truly cannot wait. A nightly list is a hundred times easier to run, monitor and fix than a live service, and many "real-time" needs turn out to be "fresh enough by tomorrow morning".

A third pattern sits between them: precompute and look up. Score everything in batch, store the scores, and let the live system read a number from a table.

Saving a model

A trained model is an object in memory. To use it tomorrow, or on another machine, you save it to a file. For scikit-learn, the usual tool is joblib, which stores the whole fitted object, including a Pipeline with its preprocessing. Always save the entire pipeline, not just the final estimator, so that scoring applies exactly the same encoding and scaling as training.

Two warnings. First, a saved model is Python's pickle format in disguise (see pickle): loading a file runs code, so only load files you created or fully trust. Second, a model file only works with compatible library versions, so store the versions next to it.

Python Save a pipeline with joblib and load it back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The artifact stores the model, the exact feature list and the library version. At load time you can compare versions and refuse to run on a mismatch. Notice that the new customers go through loaded["features"]: the model must always receive the same columns in the same form as in training.

A batch scoring script

On a real project, scoring is a small script that a scheduler runs. The shape is always the same: load the model, read new data, validate it (see Testing Data Code), score, write results, and log what happened. Here is a read-only sketch:

Python
# score_batch.py  (runs every night, for example from cron or a workflow scheduler)
import joblib
import pandas as pd

artifact = joblib.load("models/churn_2025-06.joblib")
model, features = artifact["model"], artifact["features"]

customers = pd.read_parquet("data/customers_today.parquet")
check_input(customers, features)              # schema and range checks; stop on failure

customers["churn_score"] = model.predict_proba(customers[features])[:, 1]
customers["model_version"] = "churn_2025-06"
customers[["customer_id", "churn_score", "model_version"]].to_parquet("output/scores_today.parquet")

print(f"scored {len(customers)} customers")    # logs are how you debug at 3 a.m.

Storing the model version in every output row seems fussy until the first time you must explain why two weeks of scores look different.

A real-time version wraps the same predict_proba call in a web endpoint. The model code does not change; the new work is validation of each request, speed, timeouts and what to return when the input is bad.

Python
# service.py  (read-only sketch using FastAPI; not runnable here)
from fastapi import FastAPI
from pydantic import BaseModel
import joblib, pandas as pd

app = FastAPI()
artifact = joblib.load("models/churn_2025-06.joblib")   # loaded once at startup

class Customer(BaseModel):
    tenure_months: int
    monthly_fee: float
    plan: str

@app.post("/score")
def score(customer: Customer):
    row = pd.DataFrame([customer.model_dump()])[artifact["features"]]
    return {"churn_score": float(artifact["model"].predict_proba(row)[0, 1])}

From a score to a decision

A model gives a probability. A business needs an action: call or do not call, approve or review. The rule that connects them is the decision threshold: act when the score is at least t. The default of 0.5 is rarely the right choice, because the two kinds of mistake rarely cost the same.

Suppose a retention call costs 5 and saves an average customer worth 60 if they would really have left (and the call works half the time). Calling someone who would have stayed wastes 5; missing someone who leaves loses the value. Put your costs into numbers and let the data pick the threshold.

Python Choose a threshold by expected cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from top to bottom: at a low threshold you call almost everyone and pay for many useless calls; at a high threshold you call few people and miss most churners. Profit peaks in between, here at a threshold of 0.2 (profit 5,740, against 2,180 for calling everybody). Two lessons follow. The best threshold depends on costs and capacity, not on the model, so change those inputs and the answer moves. And "call everybody" and "call nobody" are baselines you must beat, just like a naive prediction is for the model itself.

Often capacity decides: the team can make 200 calls a week, so you take the 200 highest scores. That is a ranking decision, and it makes measures like precision at the top more relevant than overall accuracy.

Probabilities are only as trustworthy as the model's calibration. If a model's "30%" really means 15%, expected-cost calculations will mislead you. Check calibration plots from Evaluating Models before using scores as probabilities.

Monitoring: models decay

After deployment the world keeps moving. Customers change behavior, a price change shifts the mix of plans, an upstream system starts sending a field in different units. Two kinds of change matter:

  • Data drift: the inputs look different from the training data. You can detect this right away, because it needs no outcomes.
  • Concept drift (model decay): the relationship between inputs and outcome changes, so the model's accuracy drops. This needs true outcomes, which often arrive weeks later (did the customer actually leave?).

Monitor both. For inputs, compare today's distribution of each feature with the training distribution. Two common tools are the Population Stability Index (PSI), which compares bin shares, and a two-sample Kolmogorov-Smirnov test, which asks whether two samples could come from the same distribution (from SciPy). A rule of thumb for PSI is: below 0.1 stable, 0.1 to 0.25 worth watching, above 0.25 a real shift. These are conventions, not laws, so tune them to your data.

Python A live drift check: PSI and a KS test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only monthly_fee was shifted on purpose, and the check flags it with a PSI of about 0.4 while the other two features stay near zero. Be careful with the KS p-value: with thousands of rows, even a trivial difference becomes "significant", which is why many teams alert on a size measure like PSI and use the p-value only as a secondary signal. This is the same effect-size-versus-significance idea from Hypothesis Testing.

When true outcomes arrive, track the model's actual performance over time (for example monthly AUC or the share of churners found in the top group). Compare it with the number you reported at launch, and write down in advance what drop triggers action.

Retraining

Retraining is the usual remedy for drift, and there are three common policies:

  1. On a schedule (monthly, quarterly): simple and predictable.
  2. On a trigger: when a drift or performance alert fires.
  3. On demand, after a known change such as a new product or price list.

Whichever you choose, retraining is a project step, not a button. Rerun the validation checks, compare the candidate with the current model on the same recent holdout, and promote it only if it is better. Keep the old artifact so you can roll back. All of this works best when the pipeline is reproducible (Reproducible Analysis): the same code and a new data snapshot should give a new model file with a new version name.

A deployment checklist

  • The decision and the action are written down: who does what with a score, and who owns the outcome.
  • The baseline (the old process, or "do nothing") is known, and the model beats it in cost terms, not only in a metric.
  • The saved artifact holds the whole pipeline, the feature list, library versions and a training-data description.
  • Inputs are validated at scoring time; bad rows are rejected or flagged, never silently scored.
  • Drift and performance monitoring exist, with thresholds agreed in advance.
  • A rollback path and a retraining policy exist.
  • Fairness and privacy were considered for the people affected (see Ethics, Fairness and Responsible Data Use).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is batch scoring usually the best starting point?

    Batch jobs process a known input on a schedule, so they are easier to validate and debug. Choose real time only when a decision cannot wait.
  2. 2. What should you save with joblib?

    The preprocessing is part of the model. Saving the full pipeline guarantees scoring transforms data exactly as in training, and the versions help you detect incompatibility.
  3. 3. A model outputs well-calibrated churn probabilities. How do you choose a call threshold?

    The best threshold depends on what each kind of mistake costs and how many actions you can take, not on a universal default.
  4. 4. Which kind of drift can you detect without waiting for true outcomes?

    Comparing input distributions to the training data needs no labels. Detecting a changed relationship between inputs and outcome needs the real outcomes.
  5. 5. Why is a KS-test p-value alone a poor drift alarm on large data?

    Significance grows with sample size. A size-of-shift measure such as PSI tells you whether the change is large enough to matter.

Summary

  • Deployment is a loop: save, score, apply a decision rule, monitor, retrain.
  • Prefer batch scoring unless a decision truly cannot wait; precomputed scores are a good middle path.
  • Save the whole pipeline with joblib, plus the feature list and library versions; only load files you trust.
  • Choose the decision threshold from costs and capacity, and compare with "act on everyone" and "act on nobody".
  • Monitor input drift (PSI, KS) immediately and real performance when outcomes arrive; agree on alert levels in advance.
  • Retrain on a schedule or a trigger, compare with the current model, and keep a rollback.

Next: Ethics, Fairness and Responsible Data Use, because every deployed model affects real people.