Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Testing, Monitoring and Deployment

Airflow Alerting: Retries, Callbacks, Notifiers and Deadlines

Make Airflow tell you when something breaks: retries with backoff, on_failure_callback, notifiers (Slack, email), timeouts, SLA replacements and alert design that avoids noise.

Pipelines fail for boring reasons: a network blip, a late file, a full disk. Good operations means two things: recover automatically from transient problems (retries) and tell a human quickly about real ones (alerts). This lesson covers retries, callbacks, notifiers, timeouts and how to avoid alert fatigue.

Airflow cannot run on this site. Airflow code is read-only. The runnable examples are stand-ins that simulate retries and callbacks with deterministic "flaky" functions.

Retries

Set retries in default_args (applied to every task of the DAG) or on a task:

Python
from datetime import timedelta
from airflow.sdk import dag, task
import pendulum

default_args = {
    "owner": "data-team",
    "retries": 3,
    "retry_delay": timedelta(minutes=2),
    "retry_exponential_backoff": True,       # 2, 4, 8 minutes ...
    "max_retry_delay": timedelta(minutes=30),
    "execution_timeout": timedelta(hours=1),  # kill a task that hangs
}

@dag(schedule="@daily", start_date=pendulum.datetime(2025, 1, 1),
     catchup=False, default_args=default_args)
def resilient():
    @task(retries=5)
    def call_api():
        ...

    call_api()

resilient()

Retry only what can succeed on a second attempt: network calls, locks, temporary outages. A bug in your code will fail every time, so retries only delay the alert. To fail without retrying (bad input), raise AirflowFailException. To skip, raise AirflowSkipException.

Python Stand-in: retries with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Callbacks

Callbacks are Python functions Airflow calls on lifecycle events. They exist at task level (on_failure_callback, on_success_callback, on_retry_callback, on_skipped_callback) and DAG level (on_failure_callback, on_success_callback). The function receives the context dictionary with the task instance, the exception and the run id.

Python
def notify_failure(context):
    ti = context["ti"]
    exc = context.get("exception")
    send_to_chat(
        f"FAILED {ti.dag_id}.{ti.task_id} run={context['run_id']} "
        f"try={ti.try_number}: {exc}"
    )

@dag(schedule="@daily", start_date=pendulum.datetime(2025, 1, 1),
     on_failure_callback=notify_failure)
def reported():
    ...

Callbacks run after retries are exhausted (for on_failure_callback), so you are not paged for a transient blip that the third attempt fixes. Keep callbacks short and safe: an exception inside a callback is logged but does not change the task state.

Python Stand-in: callbacks after the final failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notifiers

Rather than writing the HTTP call yourself, use notifiers from providers: reusable classes that you pass to a callback argument.

Python
from airflow.providers.slack.notifications.slack import SlackNotifier

@dag(
    schedule="@daily", start_date=pendulum.datetime(2025, 1, 1),
    on_failure_callback=SlackNotifier(
        slack_conn_id="slack_alerts",
        text="Failed: {{ ti.dag_id }}.{{ ti.task_id }} ({{ run_id }})",
        channel="#data-alerts",
    ),
)
def notified():
    ...

Notifier fields are templated, and credentials live in a connection. Email (SmtpNotifier), PagerDuty, Teams and others work the same way. The built-in email_on_failure option still exists but needs SMTP configuration and is less flexible.

Timeouts and lateness

  • execution_timeout kills one task that runs too long. dagrun_timeout limits a whole DAG run. Sensors have their own timeout (sensors).
  • Airflow 2 had SLAs (sla=timedelta(...)) with a miss callback. They were removed in Airflow 3. Replacements are a freshness check (alert if there was no success by a given time), your monitoring tool, or the newer deadline alerts that later Airflow 3 releases add. Check the documentation of your version.

A simple, robust "did it finish on time?" design: a small separate DAG or an external monitor that checks the latest successful run time of the important DAGs.

Python Stand-in: freshness check instead of an SLA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Designing alerts people trust

  1. Alert on the final failure, after retries, not on each attempt.
  2. Include context: DAG, task, run, attempt, error text and a link to the log.
  3. Route by owner. A team's DAGs alert that team's channel.
  4. Separate severity. Page for revenue-critical data; post to chat for the rest.
  5. Avoid duplicates. One failed run of a 50-task DAG should not send 50 messages; prefer the DAG-level callback.
  6. Test your alert path by running a deliberately failing DAG once.
  7. Delete alerts nobody acts on. Noise teaches people to ignore the real ones.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does on_failure_callback of a task run?

    It fires on the final failure. on_retry_callback is the one that fires when a retry is scheduled.
  2. 2. Which argument kills a task that runs too long?

    execution_timeout stops the task after the given time and marks it failed (or retried).
  3. 3. How do you fail a task immediately without any retry?

    AirflowFailException marks the task failed and skips remaining retries.
  4. 4. What replaced Airflow 2 SLA misses in practice for most teams?

    SLAs were removed in Airflow 3; checking the age of the last success or using deadline alerts covers the need.

Summary

  • Use retries, retry_delay and exponential backoff for transient errors; fail fast on permanent ones.
  • Callbacks and notifiers report final failures with context; keep them short.
  • Use execution_timeout and freshness checks to catch hangs and missing runs.
  • Design alerts with context, routing and low noise.

Next: Deploying Airflow