Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 37 ยท Sensors, Branching and Dynamic DAGs

Branching in Airflow DAGs: Choosing a Path at Run Time

Learn how Airflow branching works: @task.branch, BranchPythonOperator, skipped states, joining branches with trigger rules, ShortCircuit and common branching bugs.

A DAG's shape is fixed when the file is parsed, but real pipelines must make decisions: process the full load on Mondays and a light one on other days, send an alert only when quality checks fail, stop early when there is nothing to do. Branching lets a task choose, at run time, which downstream path to follow. The paths not chosen are skipped.

Airflow cannot run on this site; its code is read-only. The runnable blocks are stand-ins that implement the same skip logic in plain Python.

@task.branch

A branch task returns the task_id (or a list of task ids) that should run next. Everything else directly downstream of it is marked skipped.

Python
from airflow.sdk import dag, task
from airflow.providers.standard.operators.empty import EmptyOperator
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def weekday_logic():
    @task.branch
    def choose(logical_date=None):
        if logical_date.weekday() == 0:           # Monday
            return "full_load"
        return "light_load"

    full = EmptyOperator(task_id="full_load")
    light = EmptyOperator(task_id="light_load")
    done = EmptyOperator(task_id="done", trigger_rule="none_failed_min_one_success")

    choose() >> [full, light] >> done

weekday_logic()

The classic form is BranchPythonOperator(python_callable=...); @task.branch is its decorator version. Return None to skip all downstream tasks. Return a list, such as ["a", "b"], to follow several branches at once.

flowchart LR
  C{"choose"} --> F["full_load"]
  C --> L["light_load"]
  F --> D["done"]
  L --> D

Only one of the two middle tasks runs; the other is skipped. done needs a special trigger rule to run anyway.

Why the join task needs a trigger rule

By default a task runs only when all upstream tasks succeeded (all_success). A skipped task does not count as success, so a plain join after a branch is skipped as well. That is the most common branching bug. Use none_failed_min_one_success on the join: it runs when nothing failed and at least one upstream task succeeded. More rules are in the next lesson.

A branching stand-in

This stand-in implements the logic: the branch returns ids, siblings are skipped, and the join evaluates its rule.

Python Stand-in: branch, skip and join

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Change rules to {} and run again: done becomes skipped, exactly like the real bug.

Short-circuiting

@task.short_circuit (or ShortCircuitOperator) is a simpler gate: it returns True to continue or False to skip all downstream tasks. Use it for "stop if there is no new data":

Python
@task.short_circuit
def has_new_rows(ti=None):
    return ti.xcom_pull(task_ids="count_new") > 0

By default the skip applies to everything downstream (ignore_downstream_trigger_rules=True). Set it to False to skip only the direct children and let later tasks with a more lenient trigger rule still run.

Python Stand-in: short circuit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other ways to decide

  • Branch on a Param or Variable (see this lesson): the branch function reads the value inside the task.
  • Branch on an XCom: pull the upstream result and compare.
  • BranchDayOfWeekOperator, BranchDateTimeOperator (standard provider): ready-made branches on calendar rules.
  • Latest-only (LatestOnlyOperator): skip downstream tasks for any run that is not the most recent, useful when catching up on history.
  • For many similar branches (one per file, one per customer), do not branch: use dynamic task mapping.

Common mistakes

  • Returning a task id that is not a direct child. The id must be a task directly downstream of the branch task.
  • A join with all_success: it is skipped. Use none_failed_min_one_success.
  • Making the "empty" path invisible. Add an EmptyOperator for the do-nothing path so the graph view shows both options.
  • Branching too deeply. A 6-level decision tree is hard to read; split it into several DAGs or move logic into one task.
  • Treating skipped as failure in alerts. A skipped task is normal after a branch.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must a branch task return?

    The ids returned are followed; the other direct downstream tasks are skipped.
  2. 2. Why does a join after a branch often get skipped?

    With all_success, one skipped parent blocks the join. Use none_failed_min_one_success.
  3. 3. What does a ShortCircuit task that returns False do?

    False means stop here: downstream tasks are marked skipped.
  4. 4. You need one processing task per customer file, and the number of files changes daily. What is the best tool?

    Mapping creates one task instance per item at run time, which is what branching cannot do cleanly.

Summary

  • @task.branch returns the id(s) of the path to follow; siblings are skipped.
  • A join after a branch needs none_failed_min_one_success (or similar) to run.
  • Short-circuit tasks gate everything downstream with True/False.
  • For repeated similar work use mapping, not branching.

Next: Trigger Rules