Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Scheduling and Dependencies

Airflow Task Dependencies: >>, chain, Fan-out and Fan-in

Wire Airflow tasks together with >>, <<, lists, chain() and cross_downstream(), build fan-out and fan-in patterns, avoid cycles, and use depends_on_past and edge labels.

A DAG is its tasks and the arrows between them. Dependencies decide the order, the parallelism and what happens after a failure. Airflow offers several syntaxes to express them, and a few recurring shapes (fan-out, fan-in, diamonds) cover most real pipelines. This lesson shows the syntax, the shapes and the traps.

Airflow code is read-only. The stand-ins implement the same operators (>> with lists, chain, cross_downstream) on a tiny graph, and a cycle detector, so you can try the patterns yourself.

The bitshift operators

Python
extract >> transform >> load       # a chain: left runs before right
load << transform << extract       # same graph, written backwards

a >> b means "b is downstream of a". It also returns b, which is why chains work from left to right.

Lists: fan-out and fan-in

Python
extract >> [clean_users, clean_orders]       # fan-out: both wait for extract
[clean_users, clean_orders] >> join          # fan-in: join waits for both
flowchart LR
    E[extract] --> U[clean_users]
    E --> O[clean_orders]
    U --> J[join]
    O --> J
    J --> R[report]

A diamond: fan-out after extract, fan-in at join.

A list on both sides is not allowed with >>: [a, b] >> [c, d] raises an error because it is ambiguous (did you mean all four arrows, or two pairs?). Use one of the helpers below.

cross_downstream and chain

Python
from airflow.sdk import chain, cross_downstream

# every task on the left before every task on the right (4 arrows)
cross_downstream([a, b], [c, d])

# a >> b >> c, with lists paired element by element
chain(a, b, c)
chain(start, [x1, x2], [y1, y2], end)   # start>>x1, start>>x2, x1>>y1, x2>>y2, y1>>end, y2>>end

chain with equal-length lists pairs them one-to-one (x1>>y1, x2>>y2), which is handy for parallel lanes. For lists of different lengths it raises an error, so be precise. In Airflow 2 these helpers lived in airflow.models.baseoperator.

Dependencies with TaskFlow

With @task, passing a result creates the dependency. Add explicit ones with >> when there is no data flow:

Python
@dag(...)
def pipeline():
    @task
    def a(): ...
    @task
    def b(x): ...
    @task
    def audit(): ...

    result = b(a())    # a >> b via data
    result >> audit()  # explicit ordering

Dependencies inside loops

Python
tables = ["users", "orders", "items"]

previous = None
for t in tables:
    load = BashOperator(task_id=f"load_{t}", bash_command=f"echo {t}")
    if previous:
        previous >> load       # sequential: users, then orders, then items
    previous = load

Remove the previous logic and the three tasks run in parallel. Choosing between them is a design decision: parallel is faster, sequential is gentler on a shared database (or use a pool).

Edge labels

Label an arrow to explain why two tasks are connected. The label shows in the graph view:

Python
from airflow.sdk import Label

check >> Label("data ok") >> load
check >> Label("data missing") >> alert

(In Airflow 2: from airflow.utils.edgemodifier import Label.)

What a dependency really means

With default settings a task runs when all upstream tasks have succeeded. That default is the trigger rule all_success. Other rules (run when any upstream failed, run regardless, run if at least one succeeded) are described in Trigger rules. A failed upstream gives the downstream upstream_failed.

Dependencies between runs

By default each DAG run is independent. Two arguments connect runs:

  • depends_on_past=True on a task: it runs only if the same task succeeded in the previous run. One failure blocks every later day until fixed. Use for accumulating computations (running totals).
  • wait_for_downstream=True: the task waits until the downstream tasks of the previous run finished, useful when a downstream step reads what this step writes.

Cross-DAG dependencies

Separate DAGs can depend on each other. Options, from loosest to tightest coupling: schedule the downstream DAG on assets the upstream produces; use TriggerDagRunOperator to start another DAG; or use an external task sensor. The later chapters cover these.

Cycles

If A leads to B and B leads back to A, no valid order exists. Airflow raises AirflowDagCycleException while loading the file, and the DAG does not appear. Typical accidental cause: building dependencies in a loop with the wrong variable.

Stand-in: building a graph with >>

This stand-in supports task >> task, task >> [list] and [list] >> task with the same semantics, then prints each task's upstream set.

Python Stand-in: fan-out and fan-in with the bitshift operator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: chain and cycle detection

This stand-in implements chain and a depth-first cycle check, like the check Airflow performs when it loads a file.

Python Stand-in: chain() and cycle check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Design guidelines

  • Keep graphs shallow and readable; split a 200-task DAG into several DAGs or task groups.
  • Make dependencies reflect real data or ordering needs. Extra arrows reduce parallelism for no reason.
  • Add an EmptyOperator join node if many tasks must feed many tasks, instead of an arrow explosion.
  • Do not make a task "depend" on a run by sleeping; use sensors or assets.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does [a, b] >> [c, d] do?

    Airflow refuses list >> list; cross_downstream makes all pairs and chain pairs equal-length lists.
  2. 2. What does depends_on_past=True do?

    It links runs: a failure in one day blocks the same task on later days.
  3. 3. Which is the usual state of a task whose upstream failed (default trigger rule)?

    all_success requires every parent to succeed, otherwise the child becomes upstream_failed.
  4. 4. What happens when the dependencies form a cycle?

    A cycle has no valid order, so Airflow refuses the DAG at parse time.

Summary

  • a >> b makes b downstream of a; lists give fan-out and fan-in.
  • Use chain and cross_downstream for patterns that bitshift cannot express.
  • Default behaviour needs all upstream tasks to succeed; depends_on_past links runs together.
  • Cycles are rejected; keep graphs readable and avoid unneeded arrows.

Next: Task groups and edge labels