Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 41 ยท CI/CD for ML

Continuous Training Pipelines

Wire data checks, training, evaluation and registration into one automated pipeline that skips unchanged work, stops on failure and records every run.

Classic CI/CD builds code when code changes. Machine learning has a second input that changes on its own: data. Continuous training (CT) means the pipeline that builds a model runs automatically whenever either input changes, or on a schedule, or when monitoring says the model has gone stale.

In this lesson you will build a small CT pipeline in Python: each stage is a function, the pipeline stops at the first failure, unchanged work is skipped, and every run leaves a record. The same shape maps directly onto GitHub Actions, GitLab CI, Airflow, Kubeflow or any other orchestrator.

Triggers: when does the pipeline run?

flowchart LR
  A["Code change"] --> P["CT pipeline"]
  B["New data lands"] --> P
  C["Schedule: nightly"] --> P
  D["Drift alert"] --> P
  E["Manual run"] --> P
  P --> R["Candidate model + report"]

Five common triggers feed the same pipeline.

  • Code change: a pull request touches the training code or the config. Usually runs a fast version on a data sample.
  • New data: a file or table partition appears. Trigger by event or by polling.
  • Schedule: nightly or weekly retraining is simple and works well when data changes slowly.
  • Drift or performance alert: monitoring asks for a rebuild (see Retraining Triggers and Strategies).
  • Manual: a person starts a run, for example after fixing a data bug.

The stages as plain functions

A pipeline is only a sequence of steps that each take inputs and return outputs. Writing them as ordinary functions keeps them testable on a laptop.

Python A pipeline that stops on failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each stage either returns what the next one needs or raises. The runner never reaches register after a failed validate, so a bad dataset cannot produce a model that looks legitimate.

Skip work that has not changed

Training can be expensive, so a CT pipeline asks first: has anything changed since the last successful run? The answer comes from a fingerprint of the inputs: data, code version and parameters.

Python Fingerprint cache: train only when inputs change

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Five runs, four trainings: only the repeated run was skipped. The same idea is built into tools such as DVC and many orchestrators.

The same pipeline as a CI workflow

On the site we cannot run a CI server, so the following GitHub Actions file is read-only. It shows how triggers and stages map to a real workflow.

YAML
name: train-model
on:
  push:
    paths: ["src/**", "configs/**"]
  schedule:
    - cron: "0 3 * * *"        # every night at 03:00 UTC
  workflow_dispatch: {}        # manual button
jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.lock
      - run: python -m pipeline.validate_data
      - run: python -m pipeline.train --config configs/prod.yml
      - run: python -m pipeline.evaluate --fail-below 0.85
      - run: python -m pipeline.register
        if: github.ref == 'refs/heads/main'

Notice that register is conditional on the main branch: pull requests train and evaluate but never publish a model.

Design rules for CT pipelines

  • Idempotent stages. Running a stage twice with the same inputs gives the same output and no duplicate side effects.
  • Immutable outputs. Write each run to a new, versioned location; never overwrite last week's model.
  • Fail loudly and early. Cheap checks (schema, row counts) come before expensive training.
  • Record everything. Keep the fingerprint, metrics, durations and the final status for every run, success or failure.
  • Human gate for risky steps. Training can be automatic while deployment to all users waits for approval or a canary (see Safe Rollouts).

Common mistakes

  • Retraining on a schedule without a quality gate, so a bad week of data silently ships a worse model (see Model Quality Gates).
  • Pulling "latest" data inside the training step, which makes runs impossible to reproduce.
  • Letting pull requests publish models.
  • Ignoring failed nightly runs because nobody is alerted.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What distinguishes continuous training from ordinary CI?

    In ML the data is an input too, so new data, a schedule or a drift alert can start a build.
  2. 2. Why raise an exception in a stage instead of printing a warning?

    Stopping at the first failure prevents a model built on invalid data from being registered.
  3. 3. Which inputs belong in a run fingerprint?

    Any of the three can change the result, so all must be part of the cache key.
  4. 4. In the workflow, why is the register step limited to the main branch?

    Publishing from experimental branches would let unreviewed code create production candidates.

Summary

  • Continuous training reruns the model build when code, data, a schedule or monitoring says so.
  • Write stages as plain functions; stop at the first failure.
  • Fingerprint data, parameters and code to skip unchanged work.
  • Keep outputs immutable and record every run.
  • Automate training, but gate deployment.

Next: Batch and Online Serving