Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 41 ยท ML Lifecycle and Pipelines

Pipeline Orchestration and DAGs

Understand tasks, dependencies and DAGs, build a tiny orchestrator with retries and caching in Python, and see how Airflow, Prefect and Kubeflow express the same ideas.

A training pipeline is a set of steps with an order: you cannot train before the data is prepared, and you cannot register a model before it is evaluated. Orchestration is the job of running those steps in the right order, on a schedule or a trigger, retrying failures, and recording what happened.

The standard way to describe the order is a DAG: a directed acyclic graph. Each node is a task, each arrow says "this must finish before that starts", and "acyclic" means there are no loops, so there is always a valid order.

flowchart LR
  L["load_data"] --> V["validate"]
  V --> F["make_features"]
  F --> T["train"]
  F --> B["baseline"]
  T --> E["evaluate"]
  B --> E
  E --> R["register"]

train and baseline do not depend on each other, so an orchestrator can run them in parallel.

Why not just one long script?

A single script is fine to start. But as pipelines grow you want:

  • Retries: a flaky network call should not fail the night.
  • Partial re-runs: re-train without re-downloading data.
  • Scheduling and triggers: run daily, or when new data lands.
  • Visibility: see which step failed and its logs.
  • Parallelism: run independent steps together.

Orchestrators provide these. Popular ones are Apache Airflow, Prefect, Dagster, Kubeflow Pipelines and cloud-native services (SageMaker Pipelines, Vertex AI Pipelines).

A DAG in plain Python

You can describe a DAG as a dictionary from task name to the tasks it needs. Python's standard library even includes a sorter.

Python Order tasks with graphlib

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Add a dependency that creates a cycle (for example "load_data": {"register"}) and TopologicalSorter raises CycleError. That is the "acyclic" rule in action.

A tiny orchestrator with retries and caching

The stand-in below is not Airflow, but it implements the same ideas: tasks as functions, dependencies, retries with a flaky task, and a cache so a successful task is not repeated.

Python A mini orchestrator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the first run gives up on load_data, the second run retries only what is missing. Real orchestrators add persistent state, a scheduler, workers and a web UI, but the logic is the same.

The same idea in real tools

These blocks are read-only: they need a scheduler or cluster, which the browser cannot run.

Airflow (a Python file defining a DAG):

Python
from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def churn_training():
    @task(retries=2)
    def load(): ...
    @task
    def train(data): ...
    train(load())

churn_training()

Kubeflow Pipelines v2 (components compiled to a YAML spec and run on Kubernetes):

Python
from kfp import dsl

@dsl.component(base_image="python:3.12")
def train(lr: float) -> float: ...

@dsl.pipeline(name="churn")
def pipeline(lr: float = 0.1):
    train(lr=lr)

Design tips

  • Make tasks idempotent. Running a task twice with the same input must give the same result and no duplicate side effects.
  • Pass references, not big data. Tasks exchange file paths or table names, not gigabytes in memory.
  • Keep tasks small and named by what they do, so failures are easy to locate.
  • Parameterize runs by date or data version rather than editing code.
  • Alert on failure and on slow runs.

Cron plus a Python script is a valid orchestrator for simple pipelines. Move to a full tool when you need retries, dependencies or visibility.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the 'acyclic' in DAG guarantee?

    Without cycles, tasks can always be put in an order that respects dependencies.
  2. 2. Two tasks that do not depend on each other can be...

    Independent tasks have no ordering constraint, so an orchestrator can run them at the same time.
  3. 3. What does it mean for a task to be idempotent?

    Idempotency makes retries and re-runs safe.
  4. 4. Why do orchestrators retry tasks?

    Many failures are temporary; automatic retries avoid manual restarts.

Summary

  • A pipeline is a DAG of tasks; orchestration runs it in order with retries, schedules and logs.
  • Independent tasks can run in parallel; cycles are not allowed.
  • Make tasks idempotent and pass references to data, not the data itself.
  • Airflow, Prefect, Dagster and Kubeflow express the same ideas; simple pipelines can start with cron.

Next: Configuration and Parameters