Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท DAGs and Operators

The Airflow TaskFlow API: @task and Passing Data

Write Airflow DAGs with plain Python functions using the TaskFlow API: the @task decorator, return values as XComs, multiple outputs, dependencies and mixing in operators.

The TaskFlow API lets you write tasks as ordinary Python functions. Instead of creating an operator object and wiring data by hand, you decorate a function with @task, call it, and pass its return value to the next function. Airflow turns the calls into tasks and the arguments into dependencies and XCom transfers. For Python-heavy pipelines it is the cleanest style and the one used in most modern examples.

TaskFlow code runs only inside Airflow, so it is read-only here. A stand-in decorator below reproduces the core trick (turning function calls into a dependency graph) so you can see how it works and play with it.

A complete TaskFlow DAG

Python
from datetime import datetime
from airflow.sdk import dag, task

@dag(start_date=datetime(2025, 1, 1), schedule="@daily", catchup=False)
def sales_pipeline():

    @task
    def extract() -> list[dict]:
        return [{"item": "pen", "qty": 3, "price": 2.5},
                {"item": "book", "qty": 1, "price": 12.0}]

    @task
    def transform(rows: list[dict]) -> float:
        return sum(r["qty"] * r["price"] for r in rows)

    @task
    def load(total: float) -> None:
        print(f"Daily revenue: {total:.2f}")

    load(transform(extract()))

sales_pipeline()

Read the last line from the inside out: extract() runs first, its result feeds transform, whose result feeds load. Airflow creates three tasks (extract, transform, load) with dependencies extract >> transform >> load and moves the data between them using XComs (see XComs). You never call xcom_push or xcom_pull.

Rules to remember

  1. Calling a task function inside the DAG does not run it. It returns a lightweight reference (an XComArg) that stands for "the future result of that task". The real function runs later on a worker.
  2. Return values must be serializable. Numbers, strings, lists, dictionaries and other JSON-friendly types work. Large objects (data frames of millions of rows) do not belong in XCom; write to storage and pass the path.
  3. The task id defaults to the function name. Override with @task(task_id="clean"). Repeated calls of the same function get suffixes _1, _2.
  4. You can pass normal Python values too: load(total=100) is fine.

Several outputs

Return a dictionary and set multiple_outputs=True (it is automatic when the return annotation is a dict/TypedDict) so downstream tasks can use individual keys:

Python
@task(multiple_outputs=True)
def stats(rows):
    return {"count": len(rows), "total": sum(r["qty"] for r in rows)}

s = stats(extract())
report(count=s["count"], total=s["total"])

s["count"] is again a reference; Airflow pulls only that key.

Task options

Python
from datetime import timedelta

@task(
    retries=2,
    retry_delay=timedelta(minutes=1),
    execution_timeout=timedelta(minutes=5),
    pool="warehouse",
)
def risky_step():
    ...

Anything you can set on an operator can be set on @task, because it creates a PythonOperator-like task.

Mixing with classic operators

Not everything is Python. Combine TaskFlow tasks with operators using >>:

Python
from airflow.providers.standard.operators.bash import BashOperator

@task
def prepare():
    return "ready"

notify = BashOperator(task_id="notify", bash_command="echo done")

prepare() >> notify

You can also feed a TaskFlow result into an operator's templated field: bash_command="echo {{ ti.xcom_pull(task_ids='prepare') }}", or simply pass the reference: BashOperator(task_id="n", bash_command=prepare()) for templated parameters.

Other decorators

Besides @task, there are task-flavoured decorators for specific needs:

  • @task.virtualenv runs the function in a fresh virtual environment with its own packages.
  • @task.branch picks which downstream path runs (see the branching chapter).
  • @task.sensor turns a function into a waiting task.
  • @task.bash runs the string the function returns as a shell command.
  • @task.docker and @task.kubernetes run the function in a container (with the right provider installed).

Stand-in: functions that become a graph

This stand-in's task decorator does not run the function immediately. It returns a node and records which other nodes were passed as arguments, which is how dependencies are inferred. A run function then executes nodes in dependency order and moves return values, like XCom.

Python Stand-in: a mini @task decorator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: multiple outputs

Selecting one key from a dictionary result is the same idea as multiple_outputs:

Python Stand-in: using one key of a result

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Classic vs TaskFlow

Classic operators TaskFlow
Best for Shell, SQL, cloud services Python logic
Passing data XCom push/pull or templates Return values and arguments
Boilerplate More Less
Mix? Yes, freely Yes, freely

Use whichever reads best. Large real DAGs mix both.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does calling a @task function inside a DAG function do?

    The call builds the graph; the real function runs later on a worker.
  2. 2. How does the result of one @task reach the next one?

    TaskFlow stores the return value as an XCom and delivers it as the argument of the next task.
  3. 3. What should you do with a very large data set between tasks?

    XCom lives in the metadata database and is meant for small values.
  4. 4. How can you give a @task function a different task id?

    Pass task_id to the decorator to override the default function name.

Summary

  • @task turns a Python function into a task; calling it returns a reference to its result.
  • Passing references as arguments builds dependencies and moves data via XCom automatically.
  • Use multiple_outputs for dictionary results, and keep passed values small.
  • Mix TaskFlow with classic operators using >>; variants like @task.virtualenv and @task.branch cover special cases.

Next: Operators: Bash, Python and Empty