Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 41 ยท CI/CD for ML

CI/CD for Machine Learning

See how continuous integration and delivery change when a model is involved: code, data and models all trigger pipelines, and each stage has its own checks.

Continuous integration (CI) means every change is merged often and checked automatically. Continuous delivery (CD) means every change that passes the checks can be released with little manual work. Software teams have done this for years. Machine learning adds a twist: a change is not only a code edit. New data, a new hyperparameter or a retrained model can all change what your product does, and none of them show up in a plain git diff of source files.

In this lesson you will learn what an ML pipeline has to guard, what triggers it, and how to build the core idea (run only what changed, stop at the first failed gate) in plain Python. The next lessons fill in each stage: testing ML code, data validation and model quality gates.

This chapter uses plain Python to show the logic of CI/CD. Real pipelines run these same checks inside a CI service such as GitHub Actions or GitLab CI. Those services cannot run here, so YAML workflow files appear as read-only snippets.

What is different about ML

A normal service has one moving part: code. An ML system has three, and any of them can break it.

Three things change

Code Features, training loops, serving logic.
Data New rows, new columns, new sources.
Model New weights from a new training run.

A bug can hide in any of the three.

1 / 3

Because of this, ML teams talk about CI (test code, data and models), CD (deploy the model and the pipeline that makes it) and sometimes CT, continuous training, where the pipeline retrains the model automatically when something triggers it.

The shape of a pipeline

Each stage must pass before the next one runs. Cheap checks come first so that a typo fails in seconds, not after an hour of training.

flowchart LR
  A["Commit or new data"] --> B["Lint and unit tests"]
  B --> C["Data validation"]
  C --> D["Train on a small sample"]
  D --> E["Quality gate"]
  E --> F["Package and register"]
  F --> G["Deploy to staging"]
  G --> H["Smoke test, then production"]

A typical ML pipeline: cheap checks first, expensive steps later, a gate before anything is registered.

Two ideas make this practical:

  • Fail fast. Order stages by cost. Unit tests take seconds; full training can take hours.
  • Fail loudly. A failed stage stops the pipeline and says why. A silent pass of a broken model is the worst outcome.

A pipeline in thirty lines

You do not need a framework to understand the logic. A pipeline is an ordered list of named stages. Each stage returns normally or raises an error.

Python A tiny pipeline runner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the exit code. CI services decide red or green from the exit status of a command: 0 is success, anything else is failure. Every ML check you write ends up as a command that returns a proper exit code.

Run only what changed

Training the same model on the same data with the same code is a waste. Pipelines therefore compute a cache key from their inputs (code version, data fingerprint, parameters) and skip a stage when the key matches an earlier run. Tools such as DVC and many CI caches do exactly this. Here is the principle:

Python Cache keys for stages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only three of the four calls train. The fingerprint changes whenever data or parameters change, which is exactly the rule you want. In real tools the fingerprint covers file contents, not Python objects, but the idea is identical.

What CI usually looks like in a repository

A workflow file declares triggers and steps. This snippet is for GitHub Actions; it is read-only here because the site cannot run a CI service.

YAML
name: ml-ci
on:
  pull_request:
  schedule:
    - cron: "0 3 * * 1"      # weekly retraining check
jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: pytest tests/unit          # fast code tests
      - run: python scripts/validate_data.py
      - run: python scripts/train.py --sample 0.05
      - run: python scripts/quality_gate.py

Each run line is one command whose exit code decides whether the job continues.

Common mistakes

  • Training the full model on every pull request. Use a small sample for CI; save full training for a scheduled or release pipeline.
  • No data checks. Most production ML failures are data problems, not code bugs.
  • Non-deterministic tests. Seed random numbers, or your pipeline will fail "randomly" and people stop trusting it.
  • Deploying straight from a notebook. If the model cannot be rebuilt by a pipeline, it cannot be audited or fixed quickly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which is a trigger for an ML pipeline that a normal software pipeline usually lacks?

    Monitoring can detect that live data no longer matches training data and start a retraining pipeline. Pull requests and tags exist in software pipelines too.
  2. 2. Why should unit tests run before full model training?

    Ordering stages from cheap to expensive is the fail-fast principle.
  3. 3. How does a CI service decide a command failed?

    Exit code 0 means success; any other value marks the step and usually the whole job as failed.
  4. 4. What goes into a good stage cache key for training?

    If any input that affects the result changes, the key changes and the stage reruns; otherwise the earlier result is reused.

Summary

  • ML pipelines must guard three moving parts: code, data and models.
  • Triggers include code changes, schedules, new data and monitoring alerts.
  • Order stages cheap to expensive and stop at the first failure; use exit codes.
  • Cache keys built from code, data and parameters avoid repeated training.
  • Use small samples in CI; reserve full training for scheduled or release runs.

Next: Testing ML Code