Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Concepts and Architecture

Airflow Architecture: Scheduler, Executor, Workers and Database

Understand how Apache Airflow works inside: scheduler, DAG processor, API server, triggerer, executor, workers and the metadata database, and how a task travels between them.

Airflow is not one program but a small team of processes that cooperate through a shared database. Knowing who does what turns many mysteries ("why is my task stuck in queued?", "why did my DAG change not show up?") into quick diagnoses. This lesson names every component and follows one task from schedule to success.

You cannot run these processes on this site. The diagram and descriptions are read-only; the stand-in at the end is a toy scheduler loop in Python that demonstrates the idea of components cooperating through a shared state store.

The components

flowchart LR
    DF["DAG files<br/>(dags/ folder)"] --> DP["DAG processor"]
    DP --> DB[("Metadata database")]
    S["Scheduler"] <--> DB
    S --> EX["Executor"]
    EX --> W["Workers"]
    W --> DB
    API["API server<br/>(UI and REST)"] <--> DB
    TR["Triggerer"] <--> DB
    W -. "logs" .-> LG[("Log storage")]

Every component talks to the metadata database; workers run your task code.

DAG processor

Parses your Python files on a loop, builds the DAG structure and serializes it into the metadata database. Because the scheduler and the UI read the serialized copy, they never have to import your code. In Airflow 2 this ran inside the scheduler; since Airflow 3 it is a separate process (it can run on its own, which helps security and scaling).

Scheduler

The brain. It decides which DAG runs should be created, which task instances are ready (all upstream tasks succeeded, trigger rule satisfied, a free pool slot) and hands them to the executor. It also handles retries and timeouts. It does not run your task code.

Executor

A pluggable piece inside the scheduler that decides how and where a ready task runs: in a local subprocess (LocalExecutor), on a Celery worker (CeleryExecutor), in a Kubernetes pod (KubernetesExecutor), and so on. See Executors in a later chapter.

Workers

The processes or pods that actually execute task code. In Airflow 3 a worker talks to Airflow through the API server using the Task SDK, instead of connecting to the database directly. That is a notable change: user code in a task no longer needs database credentials.

API server

Serves the web UI and the REST API. It is the only component users talk to directly, and in Airflow 3 it is also the gateway workers use to report state.

Triggerer

Runs deferred tasks. A task that would only wait for something (a file, a query result, a time) can hand its waiting to the triggerer, which keeps thousands of waits in a single async event loop and frees the worker slot. We use this with sensors later.

Metadata database

The source of truth: DAG definitions (serialized), runs, task instances and states, connections, variables, XComs, pools, users. Use PostgreSQL (or MySQL) in production. SQLite is only for single-process learning.

Log storage

Task logs go to files on the worker, then are served by the API server, or are shipped to remote storage such as S3 or GCS so they survive pod restarts.

Following one task through the system

  1. You save etl.py in dags/.
  2. The DAG processor imports it and writes the serialized DAG into the database.
  3. At the scheduled time the scheduler creates a DAG run and one task instance per task (state scheduled).
  4. A task whose upstreams are done moves to queued: the scheduler gives it to the executor.
  5. The executor starts the task on a worker; state becomes running.
  6. The worker runs the operator's code and reports success or failed back (through the API server in Airflow 3).
  7. The scheduler sees the new state and releases downstream tasks.
  8. The UI (API server) shows all of this live by reading the database.

Everything is state in the database. Components never call each other directly with commands such as "run this now"; they update rows and look at rows. That design makes Airflow resilient: if the scheduler restarts it simply reads the database and carries on.

Scaling and high availability

  • You can run several schedulers at once. They coordinate with database row locks, so you get both speed and failover.
  • Workers scale horizontally: add Celery workers or let Kubernetes create pods.
  • The database becomes the central point; give it real hardware, connection pooling and backups.

Where to look when something is wrong

Symptom Likely component
DAG missing or stale in the UI DAG processor (import error, parse interval)
Tasks stay scheduled Scheduler: parallelism limits, pools, no executor slots
Tasks stay queued Executor or workers: not enough workers, broker problem
Task fails with a Python error Your code: read the task log
UI is slow API server or database
Sensor never wakes up Triggerer not running (deferrable sensors)

Stand-in: a toy scheduler and worker

This stand-in has a dictionary acting as the "metadata database", a scheduler_tick function that marks tasks ready, and a worker_run function that runs them. The two never call each other; they only read and write the shared state, as in Airflow.

Python Stand-in: components cooperating through state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that each tick releases only tasks whose upstream tasks have already succeeded in the database, the same rule the real scheduler applies.

Stand-in: executor slots

The executor limits how many tasks run at once (the parallelism setting). This stand-in shows a queue draining with 2 slots.

Python Stand-in: limited parallelism

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component creates DAG runs and decides which tasks are ready?

    The scheduler creates runs and task instances and releases tasks whose conditions are met.
  2. 2. Where does your task's Python code actually run?

    Workers execute operator code; the scheduler only decides and hands work to the executor.
  3. 3. Your DAG file was edited but the UI still shows the old graph. Which component do you suspect first?

    The DAG processor parses files and writes the serialized DAG. If it fails to import the file, the old version stays.
  4. 4. What is the main job of the triggerer?

    Deferrable tasks hand their waiting to the triggerer so workers are free.

Summary

  • Airflow = DAG processor + scheduler + executor + workers + API server + triggerer + metadata database (+ log storage).
  • Components coordinate by reading and writing state in the database, not by calling each other.
  • The scheduler never runs your code; workers do. In Airflow 3 workers report through the API server and the Task SDK.
  • Knowing which component owns which symptom speeds up debugging.

Next: DAG runs and task instances