Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Scheduling and Dependencies

Airflow Data Intervals and the Logical Date Explained

Understand Airflow data intervals, logical_date, data_interval_start and end, and why a daily run covers yesterday's data. Includes interval and trigger timetables and examples.

The most confusing question for new Airflow users is: "Why did my daily job for 5 March run on 6 March?" The answer is the data interval. Airflow was designed for pipelines that process a period of data, and a period can only be processed after it is over. This lesson explains data intervals, the logical date and the differences between interval-based and trigger-based schedules.

This is a concept lesson with read-only Airflow code. The stand-ins compute data intervals and logical dates with plain datetime, so you can check your understanding on real numbers.

The idea of a data interval

Suppose you have a daily job that loads yesterday's sales. A run that starts on 6 March at 00:00 processes the period from 5 March 00:00 to 6 March 00:00. Airflow stores those two bounds on the DAG run:

  • data_interval_start: 2025-03-05 00:00
  • data_interval_end: 2025-03-06 00:00

The run starts at the end of its interval, because that is when the data is complete.

flowchart LR
    A["5 Mar 00:00<br/>interval start"] -- "data for 5 Mar" --> B["6 Mar 00:00<br/>interval end = run starts"]
    B --> C["run processes 5 Mar"]

A daily interval DAG processes day D only after D has finished.

What is the logical date?

For an interval-based run, the logical date equals the start of the interval (5 March in the example). It is the "label" of the run: the date the run is about. The run's real start time (start_date of the run) is later, 6 March.

Value Example
data_interval_start 2025-03-05 00:00
logical_date 2025-03-05 00:00 (same, for interval timetables)
data_interval_end 2025-03-06 00:00
Run started (wall clock) 2025-03-06 00:00:07

{{ ds }} gives the logical date, 2025-03-05. That is why a daily export that writes /data/{{ ds }}.csv on 6 March creates the file for 5 March.

Interval vs trigger timetables

Two families of cron-based timetables exist:

Interval (data-interval) timetable. Written CronDataIntervalTimetable("0 0 * * *") and the default in Airflow 2. Each run covers [previous cron time, this cron time).

Trigger timetable. Written CronTriggerTimetable("0 0 * * *") and the default for cron strings in Airflow 3. A run is created at the cron time and describes that moment. It has no meaningful period, and data_interval_start equals data_interval_end.

Python
from airflow.timetables.interval import CronDataIntervalTimetable
from airflow.timetables.trigger import CronTriggerTimetable

# "process yesterday" style
schedule = CronDataIntervalTimetable("0 0 * * *", timezone="UTC")

# "do something at midnight" style
schedule = CronTriggerTimetable("0 0 * * *", timezone="UTC")

Choose by what the task needs:

  • A report for the period that just ended or a partitioned table load: interval.
  • A job that just needs to fire at a time (clean up, health check, snapshot): trigger.

You can use the intervals even in a trigger world, because a task can compute its own range: data_interval_start is also available with trigger timetables, set from the schedule and previous run.

Manual and asset-triggered runs

When you press Trigger in the UI or an asset event creates a run, there is no schedule slot. In Airflow 3 such runs may have logical_date = None. Tasks should not assume {{ ds }} exists for every run; a safe pattern is to derive the period from data_interval_start/data_interval_end, or use run_after (when the run was created), or pass the period in conf and fall back to a default. Airflow 3 does let you ask for a logical date when triggering manually if the DAG needs one.

Using the interval inside tasks

Python
@task
def load_sales(data_interval_start=None, data_interval_end=None):
    start = data_interval_start.strftime("%Y-%m-%d")
    end = data_interval_end.strftime("%Y-%m-%d")
    print(f"SELECT * FROM sales WHERE day >= '{start}' AND day < '{end}'")

Half-open ranges (>= start AND < end) avoid double counting a row exactly on the boundary.

Or in SQL templates:

TEXT
WHERE created_at >= '{{ data_interval_start }}'
  AND created_at <  '{{ data_interval_end }}'

Use the interval, never "yesterday" computed from now(): re-running last week's run must process last week's data.

Idempotent, partitioned loads

Interval thinking leads to a robust pattern: each run owns one slice of the data and replaces it completely. Examples: delete-then-insert for the interval, INSERT OVERWRITE PARTITION (day = '{{ ds }}'), write to a path that includes the date. Running twice gives the same result, so retries, clears and backfills are safe.

Stand-in: compute intervals for a cron

This stand-in builds daily and weekly intervals and shows the logical date and the moment the run starts.

Python Stand-in: data intervals for daily runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: interval vs trigger timetable

The same cron slot produces different run labels in the two styles. This stand-in computes both for midnight runs.

Python Stand-in: interval vs trigger semantics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Reading {{ ds }} as "today" and being surprised it is yesterday's date on an interval DAG.
  • Computing dates with datetime.now(), which breaks reruns.
  • Assuming a logical date exists for every run in Airflow 3.
  • Writing inclusive end bounds (<= end), which double counts boundary rows across runs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A daily interval DAG has logical_date 2025-03-05. When does that run start?

    The run covering the interval of 5 March starts when that interval ends, at 6 March 00:00.
  2. 2. What is the difference between CronTriggerTimetable and CronDataIntervalTimetable?

    Interval timetables attach a data interval; trigger timetables just fire at the cron time.
  3. 3. Which SQL filter correctly selects one interval?

    A half-open range [start, end) never counts a boundary row twice.
  4. 4. Why not compute 'yesterday' with now() inside the task?

    Dates must come from the run's interval so any run can be repeated accurately.

Summary

  • A data interval is the period a run is about; interval runs start when the period ends.
  • For interval timetables logical_date equals the interval start, and {{ ds }} is that date.
  • Airflow 3 cron strings default to trigger semantics; pick CronDataIntervalTimetable when you want periods.
  • Manual and asset runs may have no logical date; use the interval fields or conf.
  • Process data by half-open intervals so every run is repeatable.

Next: Catchup and backfills