Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Scheduling and Dependencies

Airflow Catchup, Backfills and Re-running Past Data

Learn what catchup does in Apache Airflow, how backfills re-process historical dates, how to clear and re-run tasks, and how to keep reruns safe with idempotent tasks.

Pipelines have a past. You deploy a DAG with start_date in January, but today is March: should Airflow run all the missing runs? A bug is found in last month's logic: how do you re-process it? Catchup and backfills answer these questions, and used carelessly they can launch hundreds of unplanned runs. This lesson makes them predictable.

These are Airflow behaviours with commands that need a running system, so they are read-only. The stand-ins compute which runs a catchup or backfill would create, which is the most useful thing to know before you press the button.

Catchup

Imagine start_date = 2025-01-01, schedule="@daily", and you unpause the DAG on 5 January.

  • catchup=True: Airflow creates a run for every missed interval from the start date: 1 Jan, 2 Jan, 3 Jan, 4 Jan, and so on, then continues daily. They run according to max_active_runs.
  • catchup=False: Airflow only schedules from "now": the most recent complete interval (or the next trigger), ignoring the older ones.
Python
with DAG(
    dag_id="daily_load",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,      # the default in Airflow 3 (it was True in Airflow 2)
) as dag:
    ...

When to use catchup=True: when each run handles one period of data and you want the complete history processed automatically, for example loading a year of partitions on the first deployment. Combine it with max_active_runs to avoid a stampede:

Python
with DAG(
    dag_id="history_load",
    start_date=datetime(2024, 1, 1),
    schedule="@daily",
    catchup=True,
    max_active_runs=3,        # process three days at a time
) as dag:
    ...

When to use catchup=False: for jobs that only care about the latest state (sending a report, refreshing a cache, calling a health endpoint).

Backfills

A backfill deliberately creates runs for a date range, usually in the past, even when catchup is off. Use it to load history, or to repair data after a fix.

In Airflow 3 backfills are a first-class feature that you start from the UI (the DAG's Backfill button) or the CLI:

Terminal
airflow backfill create \
  --dag-id daily_load \
  --from-date 2025-01-01 \
  --to-date 2025-01-31

Options you can choose when creating one:

  • Reprocess behaviour: skip dates that already succeeded, re-run only failed ones, or re-run everything.
  • Run order: oldest first (default) or newest first.
  • Max active runs for this backfill.

Backfill runs have run_type = backfill. The UI lists backfills with progress, and you can pause or cancel them. In Airflow 2 the command was airflow dags backfill, which ran inside your terminal process; Airflow 3 moved it to the scheduler so it is more reliable.

Clearing and re-running

For fixing a handful of runs, you rarely need a backfill. In the grid view, select a task instance and click Clear:

  • Only this task, or also downstream (usual choice), upstream, or the future and past runs.
  • Airflow resets the instances to none, the scheduler runs them again, and the earlier tries remain visible as old logs.

Clearing a whole DAG run re-runs it from scratch. You can also clear from the CLI: airflow tasks clear daily_load -s 2025-02-01 -e 2025-02-07.

Making reruns safe

Catchup, backfill and clear all re-run tasks, so idempotence is mandatory. Run twice, same result:

  • Overwrite or replace a partition rather than appending: DELETE FROM sales WHERE day = '{{ ds }}' followed by INSERT, or MERGE/upsert.
  • Write to a path that includes the date (s3://bucket/sales/{{ ds }}/data.parquet).
  • Never use now(); derive dates from the interval (see Data intervals).
  • Avoid side effects that cannot be undone (emails, payments) inside backfillable tasks; guard them or put them in a separate DAG with catchup=False.
  • Use depends_on_past=True only when a task truly needs the previous day's output, because one failure then blocks all later days.

Controlling the load

A backfill of 500 days will happily overwhelm your database. Control it with:

  • max_active_runs on the DAG or the backfill.
  • Pools for shared resources.
  • Running backfills off-peak or with newest-first if recent data matters most.

Stand-in: which runs does catchup create?

This stand-in lists the daily runs that would exist for a given start date and "today", with and without catchup.

Python Stand-in: catchup True vs False

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: a backfill plan with limited concurrency

This stand-in plans a backfill over a range, skipping dates already successful and processing them in batches of max_active_runs.

Python Stand-in: planning a backfill

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Idempotent write stand-in

Delete-then-insert for one date means two runs give the same table. This stand-in proves it with SQLite in Python.

Python Stand-in: an idempotent daily load

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. With catchup=True, start_date 1 Jan, daily, and the DAG unpaused on 5 Jan, what happens?

    Catchup creates every missed interval so the full history is processed.
  2. 2. Which is the correct tool to deliberately re-process January after fixing a bug?

    Backfills create runs for a chosen range; clearing re-runs existing runs.
  3. 3. Why must tasks be idempotent for backfills?

    Re-running is the whole point of backfills, so a repeat must be harmless.
  4. 4. What protects your database from a 500-day backfill running all at once?

    Limit concurrent runs and shared resources with max_active_runs and pools.

Summary

  • catchup=True creates runs for every missed interval; catchup=False (default in Airflow 3) schedules only recent ones.
  • A backfill creates runs for a date range on demand; Airflow 3 runs it in the scheduler (airflow backfill create).
  • Clear task instances to re-run a few runs.
  • Replace partitions, avoid now() and limit concurrency so repeats are safe.

Next: Task dependencies and fan-out patterns