Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Orchestration and Quality

Orchestration: DAGs, Schedules and Retries

Understand orchestration: tasks and dependencies as a DAG, topological order, schedules, retries with backoff, sensors and failure handling, with a runnable mini scheduler.

A pipeline is rarely one script. It is a set of steps that depend on each other: extract from three sources, validate, join, load, refresh dashboards. Something has to run them in the right order, at the right time, retry the flaky ones, and tell you when a step fails. That something is an orchestrator: Apache Airflow, Dagster, Prefect, Mage, cloud schedulers such as AWS Step Functions or Azure Data Factory, and cron in its most basic form.

Airflow, Dagster and Prefect are services you install and run, which cannot run in a browser tab. The code below is a labelled stand-in: a tiny scheduler written with the Python standard library that demonstrates the real concepts (DAGs, dependencies, retries, failure propagation). Real Airflow code is shown in read-only blocks.

Tasks and the DAG

An orchestrator models a workflow as a DAG: a directed acyclic graph. Nodes are tasks, arrows are dependencies ("B needs A to finish first"). "Acyclic" means no loops, so there is always a valid order to run things in.

flowchart LR
  E1["extract_orders"] --> V["validate"]
  E2["extract_customers"] --> V
  V --> T["transform"]
  T --> L["load_warehouse"]
  L --> R["refresh_dashboard"]

extract_orders and extract_customers have no dependency on each other, so they can run in parallel. Orchestrators compute this automatically from the graph.

Topological order

Python Compute a valid run order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you accidentally create a cycle, the orchestrator refuses to load the DAG, which is exactly what you want.

A mini scheduler with retries

Real pipelines call APIs and databases, and these fail transiently. Retrying with exponential backoff (wait 1, 2, 4... seconds) absorbs most blips. A task that exhausts its retries is marked failed; its downstream tasks are marked upstream_failed and do not run.

Python Stand-in: run tasks, retry, propagate failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

audit_log still runs because it depends only on extract. Only the branch below the failure is blocked. This is how real orchestrators behave, and it is why you should design wide DAGs where independent branches do not wait on each other.

What a real DAG looks like

In Airflow, tasks are Python callables or operators, and dependencies use >>. This is read-only code (it needs an Airflow installation):

Python
from airflow.decorators import dag, task
from datetime import datetime, timedelta

@dag(
    schedule="0 3 * * *",                 # every day at 03:00 UTC
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={"retries": 3, "retry_delay": timedelta(minutes=5)},
)
def daily_sales():
    @task
    def extract(ds=None):
        ...                               # read the partition for logical date ds
    @task
    def transform(rows):
        ...
    @task
    def load(rows):
        ...
    load(transform(extract()))

daily_sales()

Scheduling concepts

  • Schedule: a cron expression or interval (@daily, every 15 minutes), or an event trigger (a file arrived, an upstream dataset updated).
  • Logical date (data interval): the period of data a run is about, not when it executes. The run for 2026-03-01 usually starts after that day ends. Writing tasks in terms of the logical date is what makes backfills possible (next lesson).
  • Catchup/backfill: whether the scheduler creates runs for past intervals it missed.
  • Sensors: tasks that wait until a condition holds (a file exists, a partition lands). Use timeouts and "reschedule" modes so waiting does not hog worker slots.
  • Pools and concurrency limits: cap how many tasks hit the same database at once.
  • SLAs and alerts: notify when a task runs late or fails, not only when it crashes.
Python Which days does a schedule produce?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Design guidance

  1. Keep tasks small and focused. One task per meaningful step makes retries cheap and failures easy to read.
  2. Let tasks be idempotent (the next lesson) so a retry never doubles data.
  3. Do not pass big data through the orchestrator. Tasks exchange small metadata (a path, a row count); the data stays in storage.
  4. Do heavy work elsewhere. The orchestrator triggers Spark, dbt, or warehouse SQL; it should not itself crunch gigabytes in a worker.
  5. Make failures loud and specific: alert on the task, include the logical date and a link to logs.
  6. Version DAGs as code and review them like any other code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must a workflow DAG be acyclic?

    If A depends on B and B on A, neither can start first.
  2. 2. A task fails after all retries. What happens to tasks that depend on it?

    Dependents of a failed task are blocked, while tasks on other branches can still run.
  3. 3. What does the logical date of a daily run represent?

    Tasks should read and write by logical date so the same run can be repeated for any past day.
  4. 4. Which is good orchestration practice?

    Orchestrators coordinate; storage and engines move the data.

Summary

  • Orchestrators model pipelines as DAGs and compute order, parallelism, retries and failure propagation.
  • Schedules and logical dates tie each run to a data interval, which enables backfills.
  • Keep tasks small, retry-safe and light on data; alert specifically.

Next: Idempotency, Backfills and Reprocessing