Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Sensors, Branching and Dynamic DAGs

Airflow Sensors: Waiting for Files, Time and Other Events

Understand Airflow sensors: poke and reschedule modes, poke_interval, timeout, soft_fail, deferrable sensors and common built-in sensors, with a runnable poke-loop stand-in.

Many pipelines start with a wait: the partner's file has not arrived, the upstream table is not ready, a job on another system is still running. A sensor is a special operator whose only job is to wait until a condition becomes true, then succeed so that downstream tasks can run.

Airflow cannot run on this site. The Airflow snippets are read-only; the runnable examples are stand-ins that simulate a sensor's polling loop in plain Python.

The basic idea

A sensor repeatedly calls a method named poke(). If poke returns True, the sensor succeeds. If it returns False, the sensor waits poke_interval seconds and tries again. If the total waiting time exceeds timeout, the sensor fails.

Python
from airflow.sdk import dag
from airflow.providers.standard.sensors.filesystem import FileSensor
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def wait_for_file():
    FileSensor(
        task_id="wait_for_orders",
        filepath="/data/incoming/orders.csv",
        poke_interval=60,        # check every minute
        timeout=60 * 60 * 6,     # give up after six hours
        mode="reschedule",
    )

wait_for_file()

Important settings every sensor shares:

Parameter Meaning
poke_interval Seconds between checks (default 60)
timeout Total seconds before the sensor fails (default 7 days)
mode poke or reschedule (see below)
soft_fail If true, a timeout marks the task skipped instead of failed
exponential_backoff Make the gap between pokes grow

Poke versus reschedule mode

  • poke mode keeps the task running and occupying a worker slot the whole time, sleeping between checks. It is simple and fine for short waits.
  • reschedule mode frees the worker slot between checks. The task goes into a up_for_reschedule state and the scheduler wakes it up at the next interval. Use it for waits longer than a few minutes, or you can starve the cluster: fifty sensors in poke mode are fifty busy slots doing nothing.

A third option is deferrable sensors (deferrable=True on sensors that support it). They hand the waiting to a lightweight async process called the triggerer, so they use no worker slot at all and can wait for thousands of events at once. This needs a triggerer process running.

flowchart LR
  P["poke mode<br/>slot held the whole time"] --- R["reschedule mode<br/>slot freed between pokes"] --- D["deferrable<br/>waits in the triggerer"]

From simplest to most efficient for long waits.

A poke loop you can run

This stand-in simulates the loop with a fake clock, so it runs instantly and the output is deterministic. The file "appears" at second 250.

Python Stand-in: poke, interval and timeout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

soft_fail=True is useful when "no file today" is acceptable and downstream tasks should simply be skipped.

Writing your own sensor

Subclass BaseSensorOperator and implement poke(context), or use the decorator form for quick checks:

Python
from airflow.sdk import task
from airflow.sdk.bases.sensor import PokeReturnValue

@task.sensor(poke_interval=30, timeout=3600, mode="reschedule")
def api_is_ready() -> PokeReturnValue:
    status = call_status_endpoint()          # your own function
    return PokeReturnValue(is_done=(status == "ready"), xcom_value=status)

With PokeReturnValue the sensor can also push a value as an XCom when it finishes.

Python Stand-in: a custom condition with a value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common built-in sensors

  • FileSensor and TimeDeltaSensor/TimeSensor/DateTimeSensor (standard provider).
  • ExternalTaskSensor: wait for a task or DAG run in another DAG. The two DAGs must share the same logical date or you set execution_delta.
  • S3KeySensor, GCSObjectExistenceSensor, SqlSensor, HttpSensor: provider sensors for storage, databases and APIs (see providers).
  • Asset-based scheduling (data-aware scheduling) can replace many sensors: a DAG simply runs when an upstream task updates an asset.

Practical advice

  1. Always set a sensible timeout. The default of seven days can hide a broken upstream for a week.
  2. Use reschedule or deferrable mode for anything longer than a few minutes.
  3. Make sensors cheap: a HEAD request or file exists, not a full download.
  4. Combine with retries and alerts so humans learn when data never arrives.
  5. Remember pools: sensors in poke mode can be limited with a pool so they cannot take every slot.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens when a sensor's poke() returns False?

    Sensors keep checking at poke_interval until the condition is true or the timeout is exceeded.
  2. 2. Which mode frees the worker slot between checks?

    In reschedule mode the task leaves the worker between pokes and the scheduler brings it back later.
  3. 3. What does soft_fail=True change when a sensor times out?

    A skipped sensor lets you treat 'no data today' as acceptable, skipping dependent tasks without raising a failure.
  4. 4. What do deferrable sensors need to run?

    Deferrable tasks hand their waiting to the triggerer, an async process, freeing the worker completely.

Summary

  • A sensor waits for a condition by calling poke() every poke_interval until success or timeout.
  • Use reschedule or deferrable mode for long waits so slots are not wasted.
  • soft_fail turns a timeout into a skip.
  • Built-in sensors cover files, time, other DAGs, storage and APIs; custom sensors are easy to write.

Next: Branching in DAGs