Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท XComs, Variables and Connections

The TaskFlow API: Passing Data Between Tasks Cleanly

Use Airflow's TaskFlow API (@task) to pass values between tasks without manual XCom calls: multiple outputs, mixing with classic operators and common mistakes.

The TaskFlow API is the modern way to write Python tasks. You decorate an ordinary function with @task, call it like a function inside a DAG, and Airflow turns the calls into tasks, the arguments into XComs and the call order into dependencies. You write what looks like normal Python, and you get a normal Airflow graph.

This lesson explains what happens behind the scenes, how to return several values, and how to connect TaskFlow tasks to classic operators.

Airflow itself cannot run here, so the real code is read-only. The runnable examples are stand-ins in plain Python that mimic the ideas.

A complete TaskFlow DAG

Python
from airflow.sdk import dag, task
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def prices():
    @task
    def fetch() -> dict:
        return {"apple": 1.2, "pear": 1.8}

    @task
    def add_tax(prices: dict, rate: float = 0.2) -> dict:
        return {name: round(p * (1 + rate), 2) for name, p in prices.items()}

    @task
    def show(prices: dict):
        for name, p in prices.items():
            print(name, p)

    show(add_tax(fetch()))

prices()

Parse this file and Airflow sees three tasks (fetch, add_tax, show) with fetch >> add_tax >> show. The call add_tax(fetch()) does not run fetch at parse time. fetch() returns an XComArg, a placeholder for "the output of the task named fetch". When the task actually runs, the placeholder is replaced by the real value pulled from the database.

flowchart LR
  F["fetch()"] -- "XComArg" --> T["add_tax()"] -- "XComArg" --> S["show()"]

The function calls draw the arrows. The values travel through XCom at run time.

A stand-in that works the same way

This stand-in (not Airflow) keeps the key idea: calling a decorated function returns a reference, and only the final "run" resolves the references in dependency order.

Python Stand-in: @task builds a graph, run executes it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can see why TaskFlow code reads so naturally: the structure of your function calls is the DAG.

Returning several values

Return a dictionary for several named outputs. With multiple_outputs=True (also inferred automatically when the return annotation is a dict), each key becomes its own XCom, so downstream tasks can use result["key"] and depend only on what they need:

Python
@task(multiple_outputs=True)
def summarize(rows: list[dict]) -> dict:
    return {"count": len(rows), "total": sum(r["amount"] for r in rows)}

stats = summarize(rows)
report(stats["count"], stats["total"])

stats["count"] is also an XComArg that points at the key count. You can index it, but you cannot use it as a normal Python value while the DAG is being parsed (for example if stats["count"] > 0: fails). Do the computation inside a task, or use branching.

Python Stand-in: multiple outputs and indexing a reference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error shows why you must not do arithmetic or if checks on task outputs outside a task.

Mixing TaskFlow with classic operators

Classic operators such as BashOperator work in the same DAG. To pass a TaskFlow output into an operator, use the XComArg in a templated field; to use an operator's output in a TaskFlow task, read its .output attribute:

Python
from airflow.providers.standard.operators.bash import BashOperator

@task
def make_name() -> str:
    return "orders_2025"

say = BashOperator(task_id="say", bash_command="echo {{ ti.xcom_pull(task_ids='make_name') }}")
make_name() >> say

# the other direction
get_date = BashOperator(task_id="get_date", bash_command="date +%F", do_xcom_push=True)

@task
def use(date: str):
    print("date:", date)

use(get_date.output)

BashOperator pushes the last line of its output as an XCom when do_xcom_push=True (the default).

Typing and decorators

Type hints are optional but documented in the UI and enable the dict shortcut. Besides @task, there are variants for other environments: @task.virtualenv, @task.docker, @task.kubernetes, @task.branch and @task.sensor. They share the same data-passing rules. Arguments and results still travel through XCom, so they must be serializable and small.

Common mistakes

  • Treating a call result as a value at parse time. It is a reference until the task runs.
  • Heavy imports at the top of a DAG file. Move them inside the task function so the scheduler parses the file quickly.
  • Returning a DataFrame. Write it to storage and return the path.
  • Forgetting to call the DAG function (prices() at the bottom). Without that call no DAG is registered.
  • Calling a task twice with the same name. Airflow adds a numeric suffix (show__1); use .override(task_id="...") for clear names.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does fetch() return inside a TaskFlow DAG body?

    The call only registers a task and returns a placeholder; the value is pulled from XCom when the downstream task runs.
  2. 2. How do the arrows of the DAG get drawn in TaskFlow code?

    Using an XComArg as an argument both passes the data and makes the downstream task depend on the upstream one.
  3. 3. What does multiple_outputs=True do?

    Each dictionary key becomes its own XCom, so downstream tasks can reference result['key'].
  4. 4. Which line fails in a DAG file?

    At parse time stats['count'] is an XComArg, not a number, so Python cannot compare it. Decide inside a task or use branching.

Summary

  • @task turns a function into a task; calling it builds the graph and returns a placeholder.
  • Arguments and return values travel through XCom automatically.
  • Use multiple_outputs=True or a dict annotation for several named outputs.
  • Classic operators join in via templates (xcom_pull) or .output.
  • Never compute with placeholders at parse time; do it in a task.

Next: Variables and Params