Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Scheduling and Dependencies

Airflow Task Groups: Organizing Large DAGs Visually

Organize big Apache Airflow DAGs with TaskGroup and @task_group: prefixes, nesting, default arguments, group-level dependencies, and why SubDAGs are gone.

A DAG with 60 tasks is hard to read. A task group is a purely visual and organisational container: it collapses related tasks into one expandable box in the graph view, gives them a shared name prefix and lets you wire the group as a unit. It does not change how tasks are scheduled; each task inside still runs on its own. Task groups replaced SubDAGs, which were removed in Airflow 3.

Task groups are Airflow constructs, so the DAG code is read-only. The stand-ins model prefixed ids and group wiring in plain Python so you can see what the group machinery does.

A first task group

Python
from datetime import datetime
from airflow.sdk import DAG, TaskGroup
from airflow.providers.standard.operators.empty import EmptyOperator

with DAG(dag_id="grouped", start_date=datetime(2025, 1, 1), schedule=None) as dag:
    start = EmptyOperator(task_id="start")

    with TaskGroup(group_id="extract") as extract:
        EmptyOperator(task_id="orders")
        EmptyOperator(task_id="users")

    with TaskGroup(group_id="load") as load:
        EmptyOperator(task_id="warehouse")

    end = EmptyOperator(task_id="end")

    start >> extract >> load >> end

What this gives you:

  • In the graph view, extract and load are boxes you can expand and collapse.
  • The tasks' real ids are prefixed with the group id: extract.orders, extract.users, load.warehouse. Use these full ids in logs, CLI commands and ti.xcom_pull(task_ids="extract.orders").
  • start >> extract makes every root task of the group (those without upstream inside the group) wait for start, and extract >> load makes every leaf of extract precede every root of load.

Pass prefix_group_id=False if you prefer to keep plain task ids (and make sure they stay unique):

Python
with TaskGroup("extract", prefix_group_id=False): ...

Dependencies inside and between groups

Python
with TaskGroup("transform") as transform:
    clean = EmptyOperator(task_id="clean")
    enrich = EmptyOperator(task_id="enrich")
    validate = EmptyOperator(task_id="validate")
    clean >> enrich >> validate

Inside the block you wire ordinary tasks. You can also point at a specific task of another group: extract.get_child_by_label("orders") >> clean.

The @task_group decorator

With TaskFlow, a decorated function becomes a group, and the function can take arguments and return values:

Python
from airflow.sdk import dag, task, task_group

@dag(start_date=datetime(2025, 1, 1), schedule=None)
def grouped_taskflow():

    @task
    def fetch(source):
        return f"data from {source}"

    @task
    def clean(raw):
        return raw.upper()

    @task_group
    def ingest(source):
        return clean(fetch(source))

    ingest("crm")
    ingest("billing")      # two groups: ingest and ingest__1

grouped_taskflow()

Calling a group twice creates two sibling groups, ingest and ingest__1. This is the easy way to repeat a mini pipeline for several inputs.

Nesting

Groups can contain groups:

Python
with TaskGroup("warehouse") as warehouse:
    with TaskGroup("staging"):
        EmptyOperator(task_id="load")
    with TaskGroup("marts"):
        EmptyOperator(task_id="build")

The ids become warehouse.staging.load and warehouse.marts.build. Keep nesting to two levels; deeper trees are hard to navigate.

Shared settings

Give a group default_args so that all its tasks inherit them:

Python
with TaskGroup("flaky_apis", default_args={"retries": 5}) as flaky:
    ...

Other useful arguments: tooltip (a hover text in the UI), ui_color and ui_fgcolor (colours of the collapsed box).

Task groups vs other tools

Need Use
Make a big graph readable Task group
Repeat the same sub-pipeline for N inputs known at parse time @task_group in a loop or called repeatedly
Repeat for inputs known only at run time Dynamic task mapping
Run a whole DAG as a step, with separate runs TriggerDagRunOperator
Separate failure domains and schedules Separate DAGs
SubDagOperator Removed: do not use it

SubDAGs ran a nested DAG with its own DAG run, which caused deadlocks and confusing UI. Task groups are just a view over normal tasks in one DAG run, so none of those problems exist.

Edge labels revisited

Labels make branching arrows self-explanatory, and they also work between groups: check >> Label("ok") >> load_group. Use them where a reader could wonder why a branch exists.

Stand-in: prefixed ids from nested groups

This stand-in implements TaskGroup as a context manager with a stack, producing the same full task ids Airflow does.

Python Stand-in: task group prefixes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: group-level wiring (roots and leaves)

start >> group connects start to every root of the group, and group >> end connects every leaf to end. This stand-in computes roots and leaves of a group's internal graph.

Python Stand-in: roots and leaves of a group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Using the short id (orders) instead of the prefixed id (extract.orders) in xcom_pull or CLI commands.
  • Expecting a group to run as a unit: there is no group-level state; each task has its own.
  • Putting dependencies to a group when you only meant one task inside it, which makes everything wait.
  • Nesting groups four levels deep.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the real task id of task 'orders' inside TaskGroup('extract')?

    By default the group id is added as a prefix with a dot.
  2. 2. Does a TaskGroup change how tasks are scheduled?

    Groups are a view and naming convenience within one DAG run.
  3. 3. What replaced SubDAGs?

    Task groups give the grouping without nested DAG runs and were the recommended replacement.
  4. 4. How do you repeat a mini pipeline for several known sources with TaskFlow?

    Each call of a @task_group function creates its own sibling group (ingest, ingest__1, ...).

Summary

  • TaskGroup and @task_group collapse related tasks in the graph and prefix their ids.
  • Group wiring connects the group's roots and leaves; each task still runs and fails on its own.
  • Groups can nest and share default_args; use dynamic task mapping when the number of inputs is only known at run time.
  • SubDAGs are gone; use groups for visual structure and separate DAGs for separate schedules.

Next: XComs: Passing Small Messages Between Airflow Tasks