Skip to content
Tutorials Center
My learning

Topics

Lesson 82 of 111 ยท Beyond the Basics

CI/CD for Machine Learning

Automate tests, data validation, training, evaluation gates and deployment for ML: a GitHub Actions workflow, drift checks and a champion-challenger gate.

In ordinary software, continuous integration (CI) runs the tests automatically on every change, and continuous delivery or deployment (CD) ships a change that passed them. Machine learning adds two moving parts that ordinary tests do not cover: the data and the model. A change to the code can break a model silently; a change in the data can break it without any change in the code; and a new model can be worse than the old one even though everything "ran without errors".

CI/CD for ML, a core part of MLOps, extends the usual pipeline to catch all three. This lesson shows what to automate, how to write the key checks (data validation, a promotion gate for new models), and how a pipeline file ties them together. The first two checks run here in the browser; the workflow file is shown read-only.

The stages of an ML pipeline

  1. Code checks: formatting, linting and unit tests for the functions you wrote (feature code, custom transformers, API handlers).
  2. Data validation: is the new data shaped like the old data (columns, types, ranges, missing values, categories)?
  3. Training: run the training script with a fixed seed and pinned dependencies, and log everything (experiment tracking).
  4. Evaluation and quality gate: compare the new (challenger) model with the current (champion) on held-out data, including important slices. Fail the pipeline if it is not clearly better or if it is worse for some group.
  5. Packaging: build the container or model artifact, and register it with a version.
  6. Deployment: a safe rollout (shadow mode, canary, gradual) of the new version.
  7. Monitoring and retraining: watch drift and performance (monitoring), and start continuous training (CT) when needed, either on a schedule or when drift alerts fire.

Stages 3 and 4 are what make ML pipelines different from software pipelines: the build is training, and the test is performance on data.

Stage 2: data validation

Most production failures of ML systems are data problems: a column renamed upstream, a unit changed from dollars to cents, a new category the model has never seen, a sensor stuck at zero. A validation step compares each new batch of data with a schema (an agreed description of the data) and stops the pipeline with a readable message.

The function below checks columns, numeric ranges, allowed categories and the share of missing values, and then computes the population stability index (PSI), a simple drift number: it compares how the values of a feature are distributed in a reference sample and in new data (values below 0.1 are usually "stable", above 0.25 "major shift").

Python Validate a data batch against a schema and measure drift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The clean batch passes; the broken one produces four precise messages (31 impossible ages, 10% missing incomes against a 5% limit, an unknown plan and a stray column). The PSI is around 0.005 when nothing changed and about 0.25 for a modest shift of the mean and spread, which is the "investigate" level. Libraries such as Great Expectations, pandera and TensorFlow Data Validation provide the same idea with more features.

Stage 4: the quality gate

A new model should replace the old one only when it is reliably better. A single comparison of two scores is noisy, so a good gate does three things:

  1. It requires a minimum gain (otherwise you redeploy for nothing).
  2. It checks that the gain is statistically believable, for example with a bootstrap confidence interval of the difference.
  3. It checks slices (regions, device types, customer groups) so that an overall improvement cannot hide a big loss for one group (fairness).
Python A champion-challenger gate with a bootstrap and slice check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random forest improves the AUC by about 0.09 (the lower end of the interval is still +0.06) and does not lose in either region, so it is promoted. The second candidate changes nothing, so the gate refuses it: retraining and redeploying a model that is not better only adds risk. In a real pipeline gate raises an exception or returns a non-zero exit code, which marks the pipeline run as failed and stops the deployment.

A pipeline file

CI services (GitHub Actions, GitLab CI, Jenkins, cloud build services) read a file that lists the steps. This GitHub Actions workflow is read-only here, because the site cannot run a CI server; every step corresponds to a stage above.

YAML
# .github/workflows/ml-pipeline.yml  (read-only sketch)
name: ml-pipeline
on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "0 3 * * 1"          # retrain every Monday at 03:00 (continuous training)

jobs:
  test-and-train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
          cache: pip
      - run: pip install -r requirements.txt
      - run: ruff check .                       # 1. code checks
      - run: pytest tests/unit                  #    unit tests
      - run: python scripts/validate_data.py    # 2. data validation (fails on bad data)
      - run: python scripts/train.py --seed 0   # 3. training, logs to the tracker
      - run: python scripts/evaluate_gate.py    # 4. gate: non-zero exit code if not better
      - uses: actions/upload-artifact@v4
        with:
          name: model
          path: artifacts/model.joblib

  deploy:
    needs: test-and-train
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production                      # can require a human approval
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: model
      - run: ./scripts/deploy_canary.sh          # 6. gradual rollout; secrets come from the CI settings

Notes on the file: the training job needs access to the data (often a small CI sample, with the full training done on a separate machine), secrets live in the CI system's secret store and never in the repository, and the environment: production setting lets you require a manual approval before the real deployment.

Writing unit tests for ML code

Tests for ML code do not check "is the accuracy high"; they check properties that must always hold:

  • A custom transformer returns the expected shape and keeps the column order (custom transformers).
  • The model's output is a probability between 0 and 1 and the probabilities of a row sum to 1.
  • Predictions do not change when the input rows are shuffled.
  • A tiny dataset can be overfit (a smoke test: if the model cannot memorize ten rows, something is broken).
  • The prediction API rejects invalid input with the right status code (serving).
  • Training is reproducible: the same seed and data give the same model.

Safe deployment patterns

  • Shadow mode: the new model receives real traffic and logs its predictions, but users still see the old model's answers.
  • Canary release: send 1 to 5% of traffic to the new model, watch errors, latency and business metrics, then increase.
  • A/B test: split users and compare the actual outcome (clicks, revenue), not only offline scores.
  • Rollback: keep the previous version deployed and make switching back one command or one alias change.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which two things does ML CI/CD have to check that ordinary software CI/CD usually does not?

    A model can fail because the data changed or because the new model is worse, even when all code runs without errors.
  2. 2. Why does the promotion gate use a bootstrap confidence interval of the gain and not only the difference of two scores?

    Test-set scores are noisy. The interval shows whether the gain would survive a different sample of data.
  3. 3. What is a canary release?

    A canary limits the blast radius of a bad release: only a few users see the new model until it proves itself.
  4. 4. A PSI of 0.25 for an input feature suggests what?

    As a rule of thumb, below 0.1 is stable, 0.1 to 0.25 is a moderate shift, and above 0.25 is a major shift.

Summary

  • ML CI/CD automates code tests, data validation, training, a model-quality gate, packaging, deployment and monitoring.
  • Validate data against a schema and watch drift (for example PSI) so that upstream changes stop the pipeline early.
  • Promote a challenger only if it is reliably better overall (a minimum gain with a confidence interval) and not worse for important slices.
  • Keep secrets out of the repository, pin dependencies, fix seeds, and deploy gradually with an easy rollback.

Next: when you want the computer to search for the model for you, see AutoML.