Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Testing, Monitoring and Deployment

Airflow Logging and Monitoring: Logs, Metrics and Health Checks

Learn where Airflow task logs live, remote logging, structured log messages, StatsD and OpenTelemetry metrics, health endpoints and what to watch in production.

A pipeline that fails at 3 a.m. should tell you what failed and why without you reading source code. Airflow gives you three kinds of visibility: logs (what a task said), metrics (numbers over time: how long, how many, how late) and health checks (is the platform itself alive). This lesson covers all three and what to watch.

Airflow cannot run on this site. Configuration and DAG snippets are read-only. The runnable examples are stand-ins: a log formatter, a metrics collector and a health summary written in plain Python.

Task logs

Each task attempt writes its own log file. In the UI open a task instance and choose Logs; every retry has a separate tab. Anything you print() or log with the standard logging module (or self.log in operators) lands there:

Python
import logging
from airflow.sdk import task

log = logging.getLogger(__name__)

@task
def load(rows: int):
    log.info("loading %s rows", rows)
    if rows == 0:
        log.warning("no rows to load")
    try:
        write_to_warehouse()
    except Exception:
        log.exception("load failed")       # includes the traceback
        raise                              # keep the task failed

By default logs are files on the worker's disk under $AIRFLOW_HOME/logs, organised by dag_id/run_id/task_id/attempt. That is fragile: when a Kubernetes pod disappears so do its files. Production setups enable remote logging so logs are uploaded to object storage (S3, GCS, Azure Blob) when the task ends, and the UI reads them from there:

Terminal
export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://my-airflow-logs/prod
export AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=aws_default

Also set up log retention (lifecycle rules on the bucket) and be careful not to log secrets, personal data or whole payloads.

Write logs that help

Python Stand-in: structured log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Include the identifiers you would search for: dag, task, run, row counts, durations, the file or partition processed. Avoid vague messages like "done".

Metrics

Airflow can emit numbers to StatsD (enable with [metrics] statsd_on = True, host and port) or to OpenTelemetry ([metrics] otel_on = True), which you can forward to Prometheus, Datadog or Grafana. Useful metrics include:

Metric Why watch it
scheduler.heartbeat Is the scheduler alive? Alert when it stops
dagrun.duration.success.<dag_id> Run time trends, slow creep
dagrun.schedule_delay.<dag_id> Start lateness, a sign of overload
executor.queued_tasks, executor.open_slots Queue build-up and free capacity
pool.starving_tasks.<pool> / pool.open_slots.<pool> Resource shortages
ti.failures, ti.successes Overall error rate
dag_processing.total_parse_time Slow DAG files hurt the scheduler

This stand-in collects counters and timings with the same naming style and computes a simple report.

Python Stand-in: a metrics collector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dashboards turn these into trends; alerts on them catch problems that no single task failure reveals (a DAG that is slowly getting twice as slow).

Health checks

The webserver (API server) exposes a health endpoint (/api/v2/monitor/health in Airflow 3, /health in older versions) that reports whether the metadata database, scheduler, triggerer and DAG processor have recently sent a heartbeat. Point your load balancer and uptime monitor at it. In Kubernetes use liveness probes for each component, and the CLI check airflow jobs check --job-type SchedulerJob.

Python Stand-in: summarise component heartbeats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What to monitor in practice

  1. Task and DAG failures, with the error text in the alert (alerting lesson).
  2. Lateness: did the daily DAG finish by 07:00? Alert on missing success, not only on failure.
  3. Scheduler health and DAG parse time, the heart of the system.
  4. Queue length and open slots (executors).
  5. Database size: the metadata DB grows with every run. Clean old data on a schedule with airflow db clean.
  6. Data quality: row counts, null rates and freshness checks as tasks inside the DAG, so a "successful" run with empty data still raises an alarm.

Common mistakes

  • Relying on local log files in containers.
  • Logging passwords, tokens or personal records.
  • Alerting on every transient failure instead of the final failure after retries.
  • Having no alert for a DAG that never ran.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why enable remote logging in production?

    Uploading logs to object storage keeps them after workers are replaced and makes them available in the UI.
  2. 2. Which metric indicates the scheduler may be down?

    The scheduler emits regular heartbeats; their absence is the simplest liveness signal.
  3. 3. A DAG silently stopped being scheduled and nothing failed. What would catch this?

    Failure alerts need a failure. A check on the last successful run (freshness) detects runs that never happen.
  4. 4. Which is a good log message?

    Good logs state what happened with the identifiers and numbers you would search for.

Summary

  • Task logs are per attempt; enable remote logging and retention in production.
  • Write informative, secret-free log lines with identifiers and counts.
  • Export metrics through StatsD or OpenTelemetry and watch heartbeats, delays, queues and failure rates.
  • Use health endpoints and add freshness and data-quality checks.

Next: Alerting, Callbacks, Retries and SLAs