Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 37 ยท Concepts and Architecture

Airflow Task Lifecycle: States, Retries and Timeouts

Follow a task through every Airflow state from scheduled to success, and learn how retries, retry_delay, exponential backoff, timeouts and callbacks shape failures.

Tasks fail. Networks drop, APIs return 503, disks fill up. A good Airflow pipeline expects failure and tells Airflow what to do about it. In this lesson you learn the states a task instance passes through, the arguments that control retries and timeouts, and how to react with callbacks.

The DAG snippets are real Airflow 3 code and read-only. The stand-ins model a state machine and a retry loop with backoff, so you can experiment with the numbers.

The states

A task instance is always in exactly one state. The common ones:

State Meaning
none Created but nothing evaluated yet
scheduled The scheduler decided it should run
queued Given to the executor, waiting for a worker slot
running Executing on a worker
success Finished without error
failed Failed and has no retries left
up_for_retry Failed, a retry is planned after retry_delay
up_for_reschedule A sensor in reschedule mode is sleeping between checks
deferred Waiting on the triggerer
skipped Intentionally bypassed
upstream_failed A required upstream task failed
removed The task no longer exists in the DAG file
stateDiagram-v2
    [*] --> scheduled
    scheduled --> queued
    queued --> running
    running --> success
    running --> up_for_retry: error and retries left
    up_for_retry --> scheduled: after retry_delay
    running --> failed: error and no retries
    running --> deferred: waits on triggerer
    deferred --> scheduled: trigger fired
    scheduled --> skipped: branch or skip
    scheduled --> upstream_failed: parent failed

Simplified path of a task instance. Retries loop back to scheduled.

Retries

Most tasks should retry, because many failures are transient. Set defaults for the whole DAG with default_args, and override on individual tasks:

Python
from datetime import datetime, timedelta
from airflow.sdk import DAG
from airflow.providers.standard.operators.bash import BashOperator

default_args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
    "max_retry_delay": timedelta(minutes=30),
}

with DAG(
    dag_id="resilient_pipeline",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,
    default_args=default_args,
) as dag:
    call_api = BashOperator(
        task_id="call_api",
        bash_command="curl --fail https://example.com/data.json -o /tmp/data.json",
        execution_timeout=timedelta(minutes=10),
    )

    quick_check = BashOperator(
        task_id="quick_check",
        bash_command="test -s /tmp/data.json",
        retries=0,   # no point retrying a deterministic check
    )

    call_api >> quick_check
  • retries: how many extra attempts after the first failure. retries=3 allows 4 tries in total.
  • retry_delay: pause between attempts.
  • retry_exponential_backoff: roughly doubles the delay each time, capped by max_retry_delay. Great for rate-limited or recovering services.

Timeouts

  • execution_timeout: maximum time one try may run. When exceeded, the task fails (and can retry). Always set it on tasks that could hang.
  • dagrun_timeout (on the DAG): maximum time for the whole run.
  • Sensors have their own timeout for how long they may wait (see the sensors chapter).

Callbacks and alerts

Callbacks are Python functions that Airflow calls when something happens:

Python
def alert(context):
    ti = context["ti"]
    print(f"Task {ti.task_id} failed on try {ti.try_number}")
    # send a message to chat, create a ticket, page someone...

with DAG(
    dag_id="with_alerts",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    default_args={"on_failure_callback": alert},
    catchup=False,
) as dag:
    ...

Available callbacks include on_success_callback, on_failure_callback, on_retry_callback and on_execute_callback. Place them in default_args to cover all tasks, or set them per task. DAG-level callbacks fire when the whole run succeeds or fails.

Choosing retry numbers

A common starting point for tasks that call external services: retries=3, retry_delay=5 minutes, exponential backoff on. For quick local steps (formatting a file) retries rarely help. For tasks that must not run twice (sending an email, charging a card) either make them idempotent or set retries=0 and handle the failure manually.

Stand-in: the state machine

This stand-in models the allowed transitions. Trying an illegal jump (for example queued to success) is rejected, which mirrors why the UI never shows impossible histories.

Python Stand-in: task state machine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: retries with exponential backoff

This stand-in computes the waiting times for a flaky task and simulates two failures before a success. Times are simulated, nothing actually sleeps.

Python Stand-in: retry delays with backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Setting a huge retries for a bug that is not transient: you just delay the alert.
  • No execution_timeout: one hung call blocks a worker slot for days.
  • Non-idempotent tasks with retries, producing duplicate data.
  • Putting on_failure_callback on a DAG and expecting a per-task message (DAG callbacks run once per run).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A task has retries=3. How many times can it start in total?

    One first attempt plus three retries equals four tries.
  2. 2. What does the state up_for_retry mean?

    up_for_retry marks a failed try that will run again.
  3. 3. Which argument stops a single hung try?

    execution_timeout fails a try that runs longer than allowed.
  4. 4. A task has upstream_failed. What happened?

    With the default all_success rule a failed parent makes the child upstream_failed.

Summary

  • A task instance moves through states: scheduled, queued, running, then success, failed, retry, deferred, skipped or upstream_failed.
  • retries, retry_delay and exponential backoff handle transient failures; execution_timeout handles hangs.
  • Callbacks (on_failure_callback and friends) let you alert or clean up.
  • Design tasks to be safe to repeat.

Next: What changed in Airflow 3