Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Sensors, Branching and Dynamic DAGs

Dynamic Task Mapping in Airflow: One Task per Item

Learn dynamic task mapping in Airflow: expand, partial, mapped results, reduce steps, expand_kwargs, limits and when to prefer it over loops. Runnable map-reduce stand-in.

Sometimes you do not know, when writing the DAG, how many tasks you need. Today there are three files to process, tomorrow three hundred. Airflow's dynamic task mapping creates one task instance per item at run time, based on the output of an earlier task. You write the task once and Airflow fans it out, then lets you fan back in.

Airflow cannot run on this site. Airflow code is read-only; the runnable blocks are stand-ins in plain Python that act out the fan-out and fan-in.

expand()

Call .expand() on a task and give it the argument that should be mapped over:

Python
from airflow.sdk import dag, task
import pendulum

@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1))
def per_file():
    @task
    def list_files() -> list[str]:
        return ["a.csv", "b.csv", "c.csv"]      # could be any length at run time

    @task
    def process(path: str) -> int:
        print("processing", path)
        return len(path)

    @task
    def total(sizes: list[int]):
        print("sum:", sum(sizes))

    sizes = process.expand(path=list_files())
    total(sizes)

per_file()

When list_files finishes, the scheduler creates three process task instances (map indexes 0, 1, 2). They run in parallel up to your concurrency limits. In the UI the single process box shows a count of mapped instances. If the list is empty, no instances are created and the mapped task is marked skipped.

flowchart LR
  L["list_files"] --> P0["process [0]"]
  L --> P1["process [1]"]
  L --> P2["process [2]"]
  P0 --> T["total"]
  P1 --> T
  P2 --> T

Fan-out from one list, fan-in to a single reducer.

The reducer total receives a lazy sequence of the mapped outputs, ordered by map index. Treat it like a list; for big results iterate rather than materialising everything.

A map-reduce stand-in

This stand-in models mapped task instances as a list of results with map indexes.

Python Stand-in: expand and reduce

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

partial() for fixed arguments

Arguments that are the same for every instance go into .partial(); the ones that vary go into .expand():

Python
@task
def convert(path: str, fmt: str, compress: bool):
    ...

convert.partial(fmt="parquet", compress=True).expand(path=list_files())

You can expand over several arguments. Airflow then creates the cross product: 3 paths and 2 formats make 6 instances.

Python
convert.partial(compress=True).expand(path=list_files(), fmt=["parquet", "orc"])

If you want matching pairs instead of a product, use expand_kwargs with a list of dictionaries:

Python
convert.expand_kwargs([
    {"path": "a.csv", "fmt": "parquet", "compress": True},
    {"path": "b.csv", "fmt": "orc", "compress": False},
])
Python Stand-in: cross product versus expand_kwargs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mapping classic operators and task groups

Operators support the same methods: BashOperator.partial(task_id="x", ...).expand(bash_command=[...]). Since Airflow 2.5 you can map over whole task groups with @task_group and .expand(), which repeats a small multi-task pipeline per item, for example download then validate then load for every file.

Where the list comes from

The mapped input can be:

  • a literal list or dict in the DAG file,
  • the return value of an upstream task (it must be a list or dict; this is the usual case),
  • an XComArg transformed with .map(), .zip() or .concat() (lazy transformations, for example files.map(lambda p: p.upper())).

Mapped task instances are individually visible: each has its own log, its own retries and its own state, so one bad file does not stop the others. You can retry just the failed instance (map index 1) from the UI.

Limits and costs

  • max_map_length (default 1024) caps how many instances one mapped task may create; larger input fails the run.
  • Very large fan-outs create many task instances and put load on the database. For tens of thousands of tiny items, batch them: map over batches of 100 items and loop inside the task.
  • max_active_tis_per_dag and max_active_tis_per_dagrun limit how many instances run at once; pools can throttle shared resources.
  • Mapped outputs travel through XCom: keep each result small.
  • You cannot map over the output of a task in the same call chain at parse time; the list must exist when the upstream task finishes.

Mapping versus loops in the DAG file

A Python for loop in the DAG file creates tasks at parse time, so the list must be known then (dynamic DAG generation). Mapping creates tasks at run time from data. Use a loop when the set is static (three environments), and mapping when the set depends on data (today's files).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When are mapped task instances created?

    The number of instances depends on the upstream output, so it is known only once that task has finished.
  2. 2. What does .partial() do?

    partial fixes constant arguments; expand supplies the varying ones.
  3. 3. You expand a task over 3 paths and 2 formats. How many instances?

    Multiple expanded arguments create a cross product: 3 x 2 = 6.
  4. 4. What happens if the input list is empty?

    With nothing to map over, there are zero instances and the task is marked skipped.

Summary

  • .expand() creates one task instance per item at run time; .partial() holds constant arguments.
  • Several expanded arguments give a cross product; expand_kwargs gives explicit pairs.
  • Downstream tasks receive a lazy sequence of mapped results.
  • Respect max_map_length and batch tiny items.
  • Use mapping for data-dependent fan-out and loops for static sets.

Next: Dynamic DAG Generation