Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Sensors, Branching and Dynamic DAGs

Dynamic DAG Generation in Airflow: DAG Factories and Config Files

Generate many similar Airflow DAGs from a config with a factory function: the globals() pattern, parse-time rules, config-driven pipelines and pitfalls. Runnable stand-in.

You have ten customers and each needs the same pipeline with different settings. Copying the DAG file ten times is a maintenance nightmare. Airflow DAG files are ordinary Python, so you can write one factory function that builds a DAG from a small configuration and call it in a loop. This is called dynamic DAG generation.

Airflow cannot run on this site; its code is read-only. The runnable examples are stand-ins that reproduce the factory pattern with a toy DAG class.

The factory pattern

Python
# dags/customer_pipelines.py
from airflow.sdk import DAG, task
import pendulum

CUSTOMERS = {
    "acme":   {"schedule": "@daily",  "bucket": "acme-data"},
    "globex": {"schedule": "@hourly", "bucket": "globex-data"},
    "initech": {"schedule": "0 6 * * 1", "bucket": "initech-data"},
}

def build_dag(name: str, cfg: dict) -> DAG:
    with DAG(
        dag_id=f"sync_{name}",
        schedule=cfg["schedule"],
        start_date=pendulum.datetime(2025, 1, 1),
        catchup=False,
        tags=["customer", name],
    ) as dag:

        @task
        def extract():
            print("reading from", cfg["bucket"])

        @task
        def load():
            print("loading", name)

        extract() >> load()
    return dag

for name, cfg in CUSTOMERS.items():
    globals()[f"sync_{name}"] = build_dag(name, cfg)

The scheduler imports the file and scans its module namespace for DAG objects. Assigning each DAG to a module-level variable (here through globals()) is what registers it; a DAG that only lives inside a local variable of a function is not found by the simple scanner. Each DAG needs a unique dag_id, or the later one overwrites the former.

flowchart TD
  C["config: customers"] --> F["build_dag(name, cfg)"]
  F --> D1["sync_acme"]
  F --> D2["sync_globex"]
  F --> D3["sync_initech"]

One definition, many DAGs.

A stand-in factory

This stand-in (not Airflow) uses a toy DAG class to show registration, unique ids and the effect of a duplicate id.

Python Stand-in: factory, globals and duplicate ids

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where the configuration comes from

Source Good for Watch out
A Python dict in the file Small, static sets Changes need a deploy
A YAML/JSON file next to the DAG Teams editing settings without touching code Must be shipped with the DAGs
Environment variables Per-environment values Hard to review
A database or API call Large, changing sets Slow and fragile at parse time

The last row is the trap. DAG files are parsed continuously (by default each file is re-parsed every 30 seconds or so, by several scheduler processes). Code at module level that calls an API, queries a database or reads a Variable runs on every parse. A slow or failing call there makes every DAG in that file disappear or the scheduler lag. Rules of thumb:

  1. Read local files at parse time; they are fast and shipped with the code.
  2. If the source of truth is remote, run a separate job that writes the config into a file or a small JSON that the DAG file reads (or a CI step that generates the DAG files).
  3. Keep imports minimal at the top level.
  4. Put real work inside tasks, never at module level.
Python Stand-in: build DAG specs from YAML-like config text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dynamic DAGs versus dynamic task mapping

Dynamic DAGs (this lesson) Task mapping
When it happens Parse time Run time
What varies The number and shape of DAGs/tasks The number of instances of one task
Data source Config known at parse Output of an upstream task
UI Separate DAGs, each with its own history One DAG, many mapped instances

Choose separate DAGs when each pipeline needs its own schedule, owner, alerts or access rights. Choose mapping for "one step repeated per item" inside one pipeline.

Good habits

  • Give generated DAGs clear tags and an owner so the UI stays searchable.
  • Keep the factory deterministic: the same config must always yield the same DAG ids and task ids.
  • Version the config in Git and review changes like code.
  • Add a DAG integrity test that imports the file and checks the expected number of DAGs.
  • Watch DAG counts: hundreds of generated DAGs per file are fine, but every DAG adds scheduler load and metadata.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does the scheduler find generated DAGs in a file?

    DAG objects must be reachable as module-level variables, for example by assigning to globals().
  2. 2. What happens if two generated DAGs share a dag_id?

    dag_id must be unique. A duplicate replaces the earlier DAG, which is a hard-to-spot bug.
  3. 3. Which is the riskiest source of configuration read at module level?

    Module-level code runs on every parse. Remote calls make parsing slow and failure-prone.
  4. 4. You need one pipeline per customer with different schedules and owners. Which tool fits?

    Different schedules and owners call for separate DAGs, which a factory creates from config.

Summary

  • A factory function plus a loop can create many DAGs from a config; assign each to a module-level name.
  • dag_id values must be unique and stable.
  • Keep parse-time code fast: local config only, no network or database calls.
  • Use generated DAGs for separate pipelines and task mapping for data-dependent fan-out.

Next: Providers and Provider Packages