Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท DAGs and Operators

Ways to Define an Airflow DAG: Context Manager, Decorator, Object

Compare the three ways to declare a DAG in Apache Airflow: the with statement, the @dag decorator and the constructor, plus DAG arguments, naming and file layout rules.

Airflow gives you several equivalent ways to declare a DAG. They produce the same object, so the choice is about readability. This lesson shows all three, explains the DAG arguments you will use daily, and gives rules for naming and organising DAG files so a team of ten can work in one repository.

Airflow code blocks are read-only. The stand-ins show how Python itself makes with DAG(...) and decorators work, which demystifies a lot of "magic" in Airflow.

Style 1: the context manager

The most common style. Tasks created inside the with block are attached to the DAG automatically.

Python
from datetime import datetime
from airflow.sdk import DAG
from airflow.providers.standard.operators.empty import EmptyOperator

with DAG(
    dag_id="style_with",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    start = EmptyOperator(task_id="start")
    end = EmptyOperator(task_id="end")
    start >> end

Style 2: the @dag decorator

A function becomes the DAG; its body defines tasks. Calling the function at module level registers the DAG. The function name is the default dag_id.

Python
from datetime import datetime
from airflow.sdk import dag
from airflow.providers.standard.operators.empty import EmptyOperator

@dag(start_date=datetime(2025, 1, 1), schedule="@daily", catchup=False)
def style_decorator():
    start = EmptyOperator(task_id="start")
    end = EmptyOperator(task_id="end")
    start >> end

style_decorator()      # <- do not forget this call

The decorator style pairs naturally with the TaskFlow API (next lesson) and makes it easy to produce several DAGs from one function with parameters.

Style 3: explicit object

Rarely used, but useful to know. Pass dag= to every task.

Python
my_dag = DAG(dag_id="style_object", start_date=datetime(2025, 1, 1), schedule=None)
start = EmptyOperator(task_id="start", dag=my_dag)

It is easy to forget dag= on one task, which makes that task orphaned and raises an error, so prefer the other two.

Important DAG arguments

Argument Purpose
dag_id Unique name (required).
start_date Earliest logical date for runs.
end_date Optional last date.
schedule Cron string, preset, timedelta, timetable, assets or None.
catchup Create runs for missed intervals?
default_args Arguments applied to every task (retries, owner...).
max_active_runs / max_active_tasks Concurrency limits.
tags UI labels.
description Shown in the UI.
doc_md Markdown documentation shown on the DAG page.
params Typed inputs for manual runs (see Params).
dagrun_timeout Maximum run duration.
owner_links, on_failure_callback, on_success_callback Ownership and alerts.

Add documentation next to the code, it pays off at 3 a.m.:

Python
with DAG(
    dag_id="documented",
    start_date=datetime(2025, 1, 1),
    schedule=None,
    doc_md="""
### Customer export
Owner: data-team. Runs the export and uploads a CSV.
Failure: page #data-oncall.
""",
) as dag:
    ...

One DAG per file, one purpose per DAG

Airflow can load many DAGs from one file, but a good rule is one DAG per file named after the dag_id. Keep DAGs focused: "sales_daily_etl" and "marketing_weekly_report" rather than "all_company_jobs". Smaller DAGs have clearer failure domains and shorter grid views.

Suggested project layout:

TEXT
dags/
  sales/
    sales_daily_etl.py
  marketing/
    weekly_report.py
include/            # SQL files, templates
plugins/            # shared custom code
tests/

Generating DAGs in a loop

Because DAG files are Python you can create similar DAGs from configuration. The decorator style makes it tidy:

Python
SOURCES = {"orders": "0 2 * * *", "users": "0 3 * * *"}

for name, cron in SOURCES.items():
    @dag(dag_id=f"load_{name}", start_date=datetime(2025, 1, 1), schedule=cron, catchup=False)
    def load():
        EmptyOperator(task_id="load")

    load()

Each loop creates a separate DAG. Keep the data tiny and local; fetching the list from a database at parse time is a classic performance trap.

Stand-in: how with DAG(...) knows its tasks

Behind with, Airflow keeps a "current DAG" in a context variable, and every new task registers itself with it. This stand-in implements the trick in about twenty lines.

Python Stand-in: a context-manager DAG

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: a decorator that registers a DAG

The @dag decorator turns a function into a DAG factory. This stand-in shows the same pattern with a registry.

Python Stand-in: @dag decorator pattern

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Forgetting to call the decorated function, so no DAG appears.
  • Two DAGs with the same dag_id in different files: one silently wins.
  • Heavy imports (pandas, big SDKs) at the top of the file slow every parse; import them inside task functions.
  • Using the current time to build task ids or start_date.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. With the @dag decorator, what makes the DAG register?

    The decorated function builds the DAG when it is called, so you must call it once.
  2. 2. Which is a good rule for DAG files?

    Focused DAGs have clearer failures; a duplicated dag_id causes one definition to override another.
  3. 3. What does doc_md do?

    doc_md is rendered in the UI to document owners, purpose and runbooks.
  4. 4. How does a task created inside with DAG(...) know which DAG it belongs to?

    The context manager stores the current DAG, and operators attach themselves to it when created.

Summary

  • Declare DAGs with with DAG(...), the @dag decorator or an explicit object; the first two are preferred.
  • Learn the core arguments: dag_id, start_date, schedule, catchup, default_args, concurrency limits, tags, doc_md.
  • One focused DAG per file keeps failures and reviews manageable.
  • Keep top-level code cheap and avoid dynamic values like datetime.now().

Next: The TaskFlow API