Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Providers and Executors

Airflow Executors: Local, Celery, Kubernetes and Parallelism

Understand Airflow executors: LocalExecutor, CeleryExecutor, KubernetesExecutor, multiple executors, parallelism, pools and slots, with a runnable scheduling simulation.

The scheduler decides which task should run next. The executor decides where and how it runs. The executor is a pluggable part of Airflow: one choice runs tasks as subprocesses on one machine, another sends them to a fleet of workers, another starts one Kubernetes pod per task. Picking the right one, and tuning concurrency, is a central operations decision.

Airflow cannot run on this site. Configuration snippets are read-only. The runnable example is a stand-in that simulates how limited worker slots stretch the total run time.

What the executor does

flowchart LR
  S["Scheduler<br/>finds runnable tasks"] --> E["Executor<br/>queues and launches"]
  E --> W1["Worker / process / pod"]
  E --> W2["Worker / process / pod"]
  W1 --> M[("Metadata DB<br/>task states")]
  W2 --> M

The scheduler hands tasks to the executor, which runs them somewhere and reports back through the metadata database.

You choose it in airflow.cfg or with the environment variable AIRFLOW__CORE__EXECUTOR.

The main executors

Executor How tasks run Best for Notes
LocalExecutor Subprocesses on the scheduler machine Small and medium installs, development Needs a real database (PostgreSQL/MySQL); limited by one machine
CeleryExecutor A message broker (Redis/RabbitMQ) feeds long-running worker processes Steady, high throughput, many machines Workers are always on, so tasks start fast; you manage workers and a broker
KubernetesExecutor One pod per task instance Bursty load, per-task images and resources Pod start-up adds seconds; strong isolation; scales to zero
CeleryKubernetesExecutor / multiple executors Mix of two executors Fast tasks on Celery, heavy tasks on Kubernetes Since Airflow 2.10 you can configure several executors and choose one per task with the executor argument
Provider executors (ECS, Batch, Lambda) Cloud container services AWS-centered shops Installed with the matching provider

The old SequentialExecutor (one task at a time, SQLite) was only for tutorials and is gone in Airflow 3; for a laptop use LocalExecutor or airflow standalone.

Terminal
# environment-variable style configuration
export AIRFLOW__CORE__EXECUTOR=CeleryExecutor
export AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0

Choosing one per task (multiple executors configured):

Python
from airflow.sdk import task

@task(executor="KubernetesExecutor")
def heavy_training():
    ...

Concurrency: the limits that apply

Even with many workers, several nested limits cap how much runs at once. The tightest one wins:

Limit Scope Setting
parallelism the whole installation [core] parallelism (default 32)
max_active_runs DAG runs of one DAG DAG argument (default 16)
max_active_tasks task instances of one DAG DAG argument
max_active_tis_per_dag instances of one task task argument
Pool slots any group of tasks pool="name", slot count set in the UI
Worker concurrency per Celery worker worker_concurrency

Pools are named groups with a fixed number of slots, used to protect shared resources. A pool called warehouse with 5 slots guarantees that at most five tasks hit the warehouse at once, however many are runnable. Tasks can take more than one slot (pool_slots=2) and priorities (priority_weight) decide who goes first.

Python
extract = PythonOperator(task_id="extract", python_callable=fn,
                         pool="warehouse", pool_slots=1, priority_weight=5)

Simulating slots

This stand-in (not Airflow) schedules 8 equal tasks of 10 "minutes" each on 1, 2, 4 and 8 slots, and reports the finish time. It shows why pool and worker sizes dominate wall-clock time, and why more slots stop helping once every task already runs.

Python Stand-in: slots versus total run time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the last line: when one task is much longer than the rest, extra slots cannot shorten the run. Split long tasks or use mapping.

Priorities and fairness

When more tasks are queued than slots exist, the executor must choose. The scheduler orders queued tasks by priority_weight (higher first), and the weight rule (downstream, upstream or absolute) controls how weights add up along the graph. Without care, a huge backfill can starve urgent daily jobs; give critical DAGs a dedicated pool or higher priority.

Python Stand-in: a pool and priorities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical choices

  • Learning or one server: LocalExecutor with PostgreSQL.
  • Many fast tasks all day: Celery, tuned worker_concurrency, autoscaled workers.
  • Different dependencies or resources per task, uneven load: Kubernetes executor, or @task.kubernetes for just a few tasks.
  • Pods start slowly: keep tiny tasks on Celery or batch them.
  • Always monitor queue length and slot usage (monitoring lesson).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an executor do?

    The scheduler decides which tasks are ready; the executor launches them on a process, worker or pod.
  2. 2. Which executor starts a fresh pod for every task instance?

    KubernetesExecutor creates one pod per task instance, giving isolation and per-task resources.
  3. 3. What is a pool?

    Pools cap concurrency for tasks sharing a resource, whatever the cluster size.
  4. 4. Eight 10-minute independent tasks run on 16 slots. How long do they take?

    With enough slots all tasks run together, so the run lasts as long as the longest task.

Summary

  • The executor runs the tasks the scheduler selects: Local for one machine, Celery for worker fleets, Kubernetes for pod-per-task, and multiple executors can be mixed.
  • Concurrency is limited at several levels; pools protect shared resources.
  • Wall-clock time depends on slots and on the longest task.
  • Pick the executor from workload shape, isolation needs and operational skills.

Next: Sharing Code Across DAGs