Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Testing, Monitoring and Deployment

Deploying Airflow: Docker, Helm, CI/CD and Configuration

Learn how to deploy Airflow to production: components, Docker images, Helm on Kubernetes, DAG delivery with Git sync, AIRFLOW__ configuration, secrets and a CI/CD checklist.

Running airflow standalone on a laptop is a great way to learn. A production Airflow is a small distributed system with several components, a real database, shared logs and a delivery process for DAG files. This lesson maps those parts and shows the common ways to deploy them.

Airflow cannot run on this site. Commands and files below are read-only examples. The runnable stand-ins show how AIRFLOW__ environment variables become configuration and how a CI gate checks a deployment.

The components

flowchart LR
  Dev["Developers<br/>Git"] --> Sync["DAG delivery<br/>git-sync / image / bucket"]
  Sync --> DP["DAG processor"]
  DP --> DB[("Metadata DB<br/>PostgreSQL")]
  SCH["Scheduler"] --> DB
  API["API server / UI"] --> DB
  SCH --> EX["Executor"]
  EX --> W["Workers or pods"]
  TR["Triggerer"] --> DB
  W --> LOGS[("Remote logs")]

Every component reads and writes the metadata database; none of them talk to each other directly except through it (and the executor's queue).

Component Role
API server / webserver UI and REST API
Scheduler Creates runs, queues tasks
DAG processor Parses DAG files (a separate process in Airflow 3)
Triggerer Runs deferrable tasks asynchronously
Workers Execute tasks (Celery workers or Kubernetes pods)
Metadata database PostgreSQL or MySQL (never SQLite in production)
Broker Redis or RabbitMQ, only for CeleryExecutor

You can run several schedulers for high availability. The database is the one thing you must back up and size carefully.

Ways to deploy

  1. Docker Compose: the official docker-compose.yaml is fine for development and small single-host setups, not for serious production.
  2. Official Helm chart on Kubernetes (helm install airflow apache-airflow/airflow): the most common production route. It starts every component, supports CeleryExecutor and KubernetesExecutor, git-sync and probes.
  3. Managed services: Amazon MWAA, Google Cloud Composer, Astronomer. You give up some control and gain operations help.
  4. Virtual machines with systemd: possible, but you own everything.

Whatever you choose, build a custom image from the official one:

DOCKERFILE
FROM apache/airflow:3.0.0
COPY requirements.txt /requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt \
    --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.12.txt"
COPY dags/ /opt/airflow/dags/

(Use the Airflow and Python versions you actually run; constraint files exist per pair.) Pinned providers and libraries in the image guarantee that every component runs identical code. See providers.

Getting DAG files to every component

Method How Good for
Baked into the image COPY dags/ and redeploy on change Strict, reproducible releases
git-sync sidecar A container pulls a Git branch every N seconds Fast iteration, Helm default option
Shared volume / bucket sync A bucket or network disk mounted or synced Managed services, simple setups

Every scheduler, DAG processor and worker must see the same DAG version. With git-sync, tag releases and sync a tag or a protected branch, not a developer's feature branch.

Configuration with environment variables

Airflow reads airflow.cfg, but in containers you configure it with environment variables named AIRFLOW__<SECTION>__<KEY>. A double underscore separates the parts:

Terminal
export AIRFLOW__CORE__EXECUTOR=CeleryExecutor
export AIRFLOW__CORE__PARALLELISM=64
export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:***@db/airflow
export AIRFLOW__LOGGING__REMOTE_LOGGING=True
Python Stand-in: turning AIRFLOW__ variables into config

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Secrets such as the database URL, the Fernet key (encrypts connection passwords in the database), the API secret key and the JWT secret must be generated once and identical on all components. Store them in your platform's secret store (Kubernetes Secrets, cloud secret manager), not in Git. A secrets backend supplies connections and variables (variables).

Upgrades and database migrations

Airflow upgrades change the metadata schema. The procedure: back up the database, stop or scale down the schedulers, run airflow db migrate, start the new version. In the Helm chart a migration job does this for you. Never skip major versions without reading the upgrade guide, and rehearse on a staging copy.

Periodically clean the database: airflow db clean --clean-before-timestamp ... removes old runs and task instances.

A CI/CD gate

A simple, effective pipeline on every pull request:

  1. Lint (Ruff, including the AIR rules that flag deprecated Airflow imports).
  2. Build the image with pinned dependencies.
  3. Run DAG integrity and unit tests inside the image.
  4. On merge, push the image/tag and roll out; git-sync or the new image delivers the DAGs.
  5. Smoke test: trigger one canary DAG and check it succeeds.
Python Stand-in: a pre-deploy checklist

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Production checklist

  • PostgreSQL with backups and connection pooling (PgBouncer for large installs).
  • At least two schedulers (or one with automatic restart) and probes on every component.
  • Remote logging, log retention and monitoring.
  • Role-based access, HTTPS in front of the UI, SSO if possible.
  • Resource requests and limits set for workers, and pools for shared systems.
  • A staging environment that mirrors production.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which database should a production Airflow use for metadata?

    SQLite does not support concurrent writers; production needs PostgreSQL or MySQL.
  2. 2. How is the setting [core] parallelism written as an environment variable?

    The pattern is AIRFLOW__SECTION__KEY with double underscores.
  3. 3. Why must the Fernet key be identical on all components?

    A different key cannot decrypt stored secrets and tasks would fail to read connections.
  4. 4. What is the main benefit of a custom image with pinned constraints?

    Pinned, pre-built images give reproducible environments across dev, CI and production.

Summary

  • Production Airflow = API server, scheduler, DAG processor, triggerer, workers, metadata database (and a broker for Celery).
  • Prefer the Helm chart or a managed service, built on a custom pinned image.
  • Deliver DAGs through the image or git-sync so all components see the same version.
  • Configure with AIRFLOW__SECTION__KEY variables and keep secrets in a secret store.
  • Gate deployments in CI: lint, build, test, canary.

Next: Project: A Daily ETL Pipeline