Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 37 ยท XComs, Variables and Connections

XComs: Passing Small Messages Between Airflow Tasks

Learn how Airflow XComs let tasks share small values: push and pull, return_value, keys, templates, size limits and when not to use XCom. Runnable stand-ins included.

Every task in an Airflow DAG runs as its own process, often on a different machine. They cannot share Python variables. When task B needs a small fact from task A (a file name, a row count, an id), Airflow gives you XComs, short for "cross-communication". An XCom is a small value that a task writes to Airflow's metadata database and another task reads back later.

XComs are for messages, not for data. Pass the path of a file, never the file. This lesson shows how XComs are stored, how to push and pull them, and where the limits are.

Apache Airflow cannot run on this site. The Airflow code below is read-only. Each lesson also has a small, clearly labelled stand-in written in plain Python that you can run, so you can see the idea working. The stand-ins are not Airflow.

What an XCom is

An XCom row has these fields: a key, a value, the dag_id, the run_id, the task_id and (for mapped tasks) a map_index. So an XCom belongs to one task in one DAG run. A later run starts with a clean slate and never sees the values of an earlier run unless you ask for them explicitly.

flowchart LR
  A["extract task"] -- "push: return_value = '/tmp/orders.csv'" --> DB[("metadata DB<br/>xcom table")]
  DB -- "pull" --> B["transform task"]

Tasks never talk directly. The metadata database sits in the middle.

Pushing and pulling

The most common way to push is simply to return a value from a task. Airflow stores it under the key return_value. Pulling uses ti.xcom_pull (the task instance, ti, is available in the task context):

Python
from airflow.sdk import dag, task

@dag(schedule=None)
def xcom_demo():
    @task
    def extract() -> str:
        return "/tmp/orders.csv"          # stored as key "return_value"

    @task
    def transform(path: str) -> int:
        print("reading", path)
        return 42

    @task
    def explicit_pull(ti=None):
        path = ti.xcom_pull(task_ids="extract", key="return_value")
        print("manual pull:", path)

    transform(extract())
    explicit_pull()

xcom_demo()

Three things to notice:

  • transform(extract()) passes the XCom for you: the TaskFlow API (see the next lesson) wires the push, the pull and the dependency in one go.
  • ti.xcom_pull(task_ids=..., key=...) is the manual form. You can also call ti.xcom_push(key="rows", value=42) to push under a custom key.
  • xcom_pull with a list of task ids returns a list of values.

A stand-in you can run

Below is a stand-in (not Airflow) that keeps XComs in a dictionary keyed by the same fields the real table uses. Notice how the run id keeps two runs apart.

Python Stand-in: an XCom store

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each run sees its own value. A missing task simply gives None: in real Airflow a failed xcom_pull also returns None unless you ask for include_prior_dates=True or the TaskFlow API raises because the upstream task has no output.

Serialization and size limits

The value must be serializable. By default Airflow stores it as JSON (and can handle a few extra types such as datetime and dataclasses through its serializer). Anything else, such as an open file, a database cursor or a DataFrame, fails. Even valid values are limited by the database column: a typical limit is 64 KB on MySQL (a BLOB), and bigger on PostgreSQL, but you should never get near it.

Python Stand-in: reject values that are too big or not JSON

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you need to move real data, write it to object storage or a database and pass its location. The rule of thumb: an XCom should fit comfortably in a tweet.

XComs in templates

Operators with templated fields can read XComs through Jinja, which is handy with classic operators such as BashOperator:

Python
from airflow.providers.standard.operators.bash import BashOperator

report = BashOperator(
    task_id="report",
    bash_command="echo 'file is {{ ti.xcom_pull(task_ids=\"extract\") }}'",
)

Templates are rendered just before the task runs, when the upstream value exists.

Custom XCom backends

If large values are unavoidable, Airflow lets you replace the storage with a custom XCom backend: a class that writes the value to, say, S3 and stores only a reference in the database, then reads it back transparently. The provider packages ship such backends, configured with xcom_backend in the [core] section. Treat this as an advanced option: pipelines are easier to debug when the data location is visible in your own code.

Common mistakes

  • Returning big objects (DataFrames, lists of a million rows). Save them elsewhere and return a path.
  • Assuming XComs survive between runs. They are tied to a run id.
  • Pulling from a task that is not upstream. It may not have run yet, so you get None or a stale value. Always declare the dependency.
  • Pushing secrets. XComs are visible in the UI. Use connections for credentials.
  • Using XComs as state across runs. Use an Airflow Variable or a database table.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a task that simply returns a value do in Airflow?

    Returned values are pushed automatically under the key return_value, tied to that task instance in that run.
  2. 2. Which is the best thing to pass through an XCom?

    XComs carry small, serializable messages. A path or id is perfect; big data and secrets are not.
  3. 3. Which fields identify one XCom row?

    An XCom belongs to a task instance in a specific DAG run, so run_id and task_id are part of its identity.
  4. 4. Why should you declare a dependency before pulling an XCom from another task?

    Dependencies guarantee that the producer finishes first, so the value exists when the consumer reads it.

Summary

  • An XCom is a small message stored in the metadata database for one task instance in one run.
  • Returning a value pushes return_value; ti.xcom_pull and ti.xcom_push give manual control.
  • Values must be serializable and small; pass paths and ids, not data.
  • Templates can read XComs; custom backends can offload large values.
  • Do not use XComs for secrets or for state across runs.

Next: The TaskFlow API and Data Passing