Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 41 ยท CI/CD for ML

Model Quality Gates

Decide automatically whether a trained model may be promoted: absolute floors, champion comparison, slice checks, latency budgets and noise-aware thresholds.

A quality gate is a rule that a candidate model must satisfy before it moves forward. It turns "the new model looks good to me" into a repeatable, reviewable decision. Gates sit between training and registration in the pipeline, and a model that fails one never reaches production.

Good gates have three properties: they are automatic (no one forgets to run them), explainable (a failure says which rule and by how much), and calibrated (not so tight that random noise blocks every release, not so loose that regressions slip through). This lesson builds such a gate in Python.

What a gate can check

Gate rules

Absolute floor Recall must be at least 0.80.
Beat the champion No worse than the live model.
Slice checks No group falls far behind.
Budgets Latency and size stay in limits.

A model must pass all rules.

1 / 2
  • Absolute floor: a minimum on the metric that matters to the business (for example recall at least 0.80 for a safety model).
  • Relative to the champion: the model currently in production is the champion; the new one is the challenger. Require the challenger to be at least as good, within a small tolerance.
  • Slice checks: overall accuracy can rise while a region, device type or age group gets worse. Check important slices separately.
  • Operational budgets: inference latency, memory and file size. A more accurate model that is ten times slower may be unusable.
  • Behavioral tests: the suites from Testing ML Code also count as gates.

A first gate: floor plus champion comparison

Python Floor and champion comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The function returns a list of reasons rather than a boolean. When a gate fails at 3 a.m., a message like recall regressed by 0.041 vs champion is worth far more than False.

The test set must be data the model never trained on, and every candidate must be scored on the same test set. If you compare models on different data, a difference may come from the data, not the model.

Slice checks

Overall numbers hide groups. Suppose a model improves on average but gets worse for one customer segment. A slice gate computes the metric for each group and fails when any group (large enough to measure) falls below a floor or lags far behind the overall figure.

Python Per-slice gate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the min_rows guard. A slice with ten rows produces a very noisy metric, so it is better to report "not enough data" than to fail or pass on luck.

Beware of noise

Test-set metrics are estimates. If the challenger scores 0.842 and the champion 0.840, you cannot tell whether it is better or only luckier. Two practical remedies:

  1. Tolerance: allow a small regression (max_regression) so noise alone does not block releases.
  2. Bootstrap: resample the test set many times to see how much the metric wobbles, then demand that the difference be bigger than the wobble.
Python Bootstrap the difference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the interval contains zero, the honest conclusion is "no clear difference". Many teams then keep the simpler or cheaper model.

Operational budgets

Accuracy is not the only thing that can block a release. Add checks for the cost of using the model.

Python
import os, time, joblib

def operational_gate(model, X_sample, model_path, max_ms=20, max_mb=50):
    problems = []
    start = time.perf_counter()
    for row in X_sample[:200]:
        model.predict(row.reshape(1, -1))
    per_call = (time.perf_counter() - start) / 200 * 1000
    if per_call > max_ms:
        problems.append(f"latency {per_call:.1f} ms per call (limit {max_ms})")
    size_mb = os.path.getsize(model_path) / 1e6
    if size_mb > max_mb:
        problems.append(f"model file {size_mb:.1f} MB (limit {max_mb})")
    return problems

Timing depends on the machine, so measure on hardware similar to production and leave some headroom. See Latency, Throughput and Load for how to measure properly.

Wiring it into a pipeline

The gate script writes its reasons to the log and exits non-zero on failure. In CI it is the step right before registering the model:

YAML
      - run: python scripts/train.py --out artifacts/candidate.joblib
      - run: python scripts/quality_gate.py artifacts/candidate.joblib
      - run: python scripts/register_model.py artifacts/candidate.joblib

Because each run line stops the job on failure, register_model.py executes only when the gate passes.

Common mistakes

  • Tuning thresholds on the test set. Then the gate measures how well you tuned, not how well the model generalizes. Keep a final untouched holdout.
  • One global metric. Always add slice checks for groups that matter.
  • No floor, only "better than champion". A chain of tiny regressions, each within tolerance, slowly degrades the model. Keep an absolute floor too.
  • Hidden gate logic. Keep rules in a version-controlled file so changes are reviewed like code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In champion versus challenger testing, what is the champion?

    The champion is the current live model; the challenger is the candidate that wants to replace it.
  2. 2. Why return a list of failure reasons instead of just True or False?

    Explainable failures are fixable failures.
  3. 3. A challenger beats the champion by 0.002 and the bootstrap interval is [-0.02, 0.025]. What should you conclude?

    The interval includes zero, so the difference may be noise.
  4. 4. Why add slice checks?

    Averages hide group-level regressions.

Summary

  • A quality gate is an automatic, explainable rule set a candidate must pass before registration.
  • Combine an absolute floor, a comparison with the champion, slice checks and operational budgets.
  • Return reasons, not just pass or fail.
  • Use tolerances or bootstrap intervals so noise does not decide releases.
  • Keep an untouched holdout and keep gate rules in version control.

Next: Continuous Training Pipelines