Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 37 ยท Concepts and Architecture

DAG Runs and Task Instances in Airflow

Learn how Airflow represents one execution of a DAG, how run ids and run types work, and how tasks become task instances with try numbers, pools and priorities.

A DAG file is a blueprint. Nothing runs until Airflow turns the blueprint into concrete records: a DAG run (one execution of the whole DAG) and a task instance (one task inside that run). Understanding these two objects explains most of what you see in the UI, in logs and in the database.

Real Airflow objects need the metadata database, so the model is shown with read-only code and rebuilt as a runnable stand-in with Python dataclasses.

From blueprint to records

Think of a recipe and a dinner:

  • The DAG is the recipe (one file, one definition).
  • A DAG run is one evening you cooked it.
  • A task instance is one step of that evening ("chop onions on Tuesday").

If a DAG has 5 tasks and has run on 30 days, the database holds 30 DAG runs and 150 task instances for it.

What identifies a DAG run

Every DAG run has:

  • dag_id: which DAG.
  • run_id: a unique string per DAG, like scheduled__2025-03-01T00:00:00+00:00 or manual__2025-03-01T10:15:42+00:00.
  • run_type: scheduled, manual, backfill, or asset_triggered (event-driven runs).
  • logical_date (nullable for some run types in Airflow 3): the point in time the run represents, see Data intervals.
  • state: queued, running, success or failed.
  • conf: an optional JSON dictionary passed when triggering manually.
  • start_date / end_date: real wall-clock times, different from the logical date.

A DAG run is success when every task finished in an acceptable state (success or skipped) and failed when a leaf task failed.

What identifies a task instance

A task instance is identified by the triple (dag_id, task_id, run_id) (plus a map_index for dynamic tasks). It carries:

  • state (see Task lifecycle)
  • try_number: how many attempts have started (1 for the first run)
  • max_tries: from the task's retries setting
  • pool and priority_weight: scheduling controls
  • hostname, pid, start_date, end_date, duration
  • queue and executor (where it was sent)

When you click Clear in the UI, Airflow resets the task instance (and optionally its downstream tasks) to none and the scheduler runs it again. The previous attempts keep their logs under try_number.

Controlling concurrency

Several settings limit how much runs at once, at different levels:

Python
from datetime import datetime
from airflow.sdk import DAG
from airflow.providers.standard.operators.bash import BashOperator

with DAG(
    dag_id="limited_dag",
    start_date=datetime(2025, 1, 1),
    schedule="@hourly",
    catchup=False,
    max_active_runs=1,       # only one DAG run at a time
    max_active_tasks=4,      # at most 4 task instances of this DAG at once
) as dag:
    BashOperator(
        task_id="heavy",
        bash_command="sleep 60",
        pool="warehouse",    # shared limit across DAGs
        priority_weight=5,   # higher runs first when slots are scarce
    )
  • max_active_runs: how many runs of this DAG may be active. Useful when each run reads and rewrites the same table.
  • max_active_tasks: parallel tasks within one DAG.
  • Pools: named slot groups shared across DAGs, for example "only 3 queries against the warehouse at the same time". Create them in Admin > Pools.
  • parallelism (global): the maximum number of running task instances in the whole system.

Manual runs and conf

You can start a run yourself and hand it parameters:

Terminal
airflow dags trigger report --conf '{"customer": "acme", "full_refresh": true}'

Inside a task, the parameters are available from the run's conf (through the context, dag_run.conf). This turns one DAG into a reusable job with inputs.

Stand-in: modelling runs and task instances

This stand-in creates DAG runs and task instances like the scheduler does, and shows that each run gets its own set of task instances.

Python Stand-in: DAG runs and task instances

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: max_active_runs

max_active_runs=1 makes later runs wait until the active one finishes. This stand-in shows the queue behaviour.

Python Stand-in: max_active_runs=1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical tips

  • Idempotent tasks. Because tasks are cleared and re-run so often, design them so that running twice for the same run gives the same result (use INSERT ... ON CONFLICT, overwrite one partition, etc.).
  • Use run_id in logs when you search for a problem; it uniquely names the execution.
  • Pools protect shared systems. If five DAGs each open 10 connections to one database, a pool of 10 slots is the polite fix.
  • Old runs accumulate. Plan periodic cleanup of the metadata database (airflow db clean) so the UI stays fast.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A DAG with 6 tasks has had 10 runs. How many task instances exist?

    Each run creates one task instance per task: 6 x 10 = 60.
  2. 2. Which triple identifies a (non-mapped) task instance?

    The task instance lives inside one DAG run: dag, task and run together identify it.
  3. 3. What does max_active_runs=1 do?

    It limits concurrent runs of that DAG, which protects shared resources between runs.
  4. 4. Why should tasks be idempotent?

    Clearing, retries and backfills re-run tasks, so repeats must be safe.

Summary

  • A DAG is the blueprint; a DAG run is one execution; a task instance is one task in one run.
  • Runs have a run_id, a run_type and a state; task instances have a state, a try number, a pool and more.
  • Limit concurrency with max_active_runs, max_active_tasks, pools and the global parallelism.
  • Make tasks idempotent because clearing and retrying are normal.

Next: Task lifecycle and states