Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

Installing Apache Airflow: pip, Standalone and Docker

Install Apache Airflow step by step: virtual environment, constraint files, airflow standalone, Docker Compose, AIRFLOW_HOME, and the first checks to run.

There are three common ways to get Airflow running on your own machine: a Python virtual environment with pip, the all-in-one airflow standalone command for learning, and Docker Compose for a setup that looks more like production. This lesson walks through each one, explains AIRFLOW_HOME, and shows how to verify the install.

Airflow is not installed on this site, so every command below is read-only. The runnable programs at the end are stand-ins that help you check the Python side of an installation (version rules and the constraint-file URL).

Requirements

  • A supported Python 3 version. Each Airflow release lists the versions it supports; check the release notes for the exact range.
  • Linux, macOS or Windows through WSL 2. Airflow does not run natively on Windows; use WSL or Docker.
  • About 4 GB of RAM for a comfortable standalone setup.
  • For the Docker route: Docker Engine or Docker Desktop and Docker Compose v2.

Option 1: pip in a virtual environment

Always use a virtual environment so Airflow's many dependencies do not collide with your other projects.

Terminal
python3 -m venv airflow-venv
source airflow-venv/bin/activate

export AIRFLOW_HOME=~/airflow

AIRFLOW_VERSION=3.0.6            # choose the current release
PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Why a constraint file?

Airflow depends on hundreds of packages. A constraint file pins every dependency to a combination the Airflow maintainers tested. Without it, pip may pick newer versions of some library and Airflow can break in surprising ways. The URL contains both the Airflow version and your Python version, so it must be built for your pair.

AIRFLOW_HOME

AIRFLOW_HOME is the folder where Airflow keeps its files. The default is ~/airflow. After the first run you will find:

TEXT
airflow/
  airflow.cfg          # configuration
  dags/                # your DAG files go here
  logs/                # task logs
  airflow.db           # SQLite metadata database (standalone mode)

Set the variable in every shell where you run Airflow commands, otherwise you may create a second, empty home folder by accident.

Option 2: airflow standalone

For learning, one command starts everything:

Terminal
airflow standalone

It initializes the metadata database, creates a login, and starts the scheduler, the DAG processor, the triggerer and the API server (web UI) in a single terminal. Open http://localhost:8080. In Airflow 3 the default login manager prints a generated admin password in the terminal output; keep that window open the first time.

Standalone uses SQLite and the LocalExecutor-style execution, which is fine for learning but never for production: SQLite allows only one writer at a time.

Option 3: Docker Compose

The Airflow project publishes a ready-made docker-compose.yaml for each release. It starts PostgreSQL, Redis, the Celery executor, the scheduler, the API server and workers.

Terminal
mkdir airflow-docker && cd airflow-docker
# download the compose file that matches your chosen Airflow version
curl -LfO "https://airflow.apache.org/docs/apache-airflow/${AIRFLOW_VERSION}/docker-compose.yaml"

mkdir -p ./dags ./logs ./plugins ./config
echo "AIRFLOW_UID=$(id -u)" > .env      # Linux: avoid root-owned files

docker compose up airflow-init          # create the database and the first user
docker compose up -d                    # start everything
docker compose ps                       # check that containers are healthy

The dags/ folder on your machine is mounted into the containers, so saving a file there is enough to publish a DAG. To stop and delete the data volumes use docker compose down --volumes.

The official compose file is meant for learning and local development. For production you need your own images, secrets management, backups of the metadata database, and usually Kubernetes or a managed service. We cover this later in the course.

Verify the installation

Terminal
airflow version          # prints the installed version
airflow config list      # shows the effective configuration
airflow dags list        # lists DAGs the system has found

If airflow dags list shows the bundled example DAGs, the scheduler and the database are talking to each other. Example DAGs can be turned off with AIRFLOW__CORE__LOAD_EXAMPLES=False, which you should do before you start building your own.

Configuration through environment variables

Every option in airflow.cfg can be overridden with an environment variable named AIRFLOW__{SECTION}__{KEY} (two underscores on each side):

Terminal
export AIRFLOW__CORE__LOAD_EXAMPLES=False
export AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags
export AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=30

This is how containers are configured, because you do not need to edit a file inside the image.

Stand-in: building the right constraint URL

Mixing up the Airflow or Python version in the constraint URL is the most common install mistake. This stand-in builds the URL the same way the shell snippet does, and checks a version range.

Python Stand-in: constraint URL builder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: environment-variable names

A second small helper that turns config sections into the environment variable names Airflow reads:

Python Stand-in: AIRFLOW__SECTION__KEY names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common problems

  • airflow: command not found: the virtual environment is not activated.
  • No DAGs show up: the file is not in the dags/ folder of the same AIRFLOW_HOME the scheduler uses, or it has an import error (see the UI import-error banner).
  • Dependency conflicts after installing extra packages: re-run pip install with the same constraint file; add providers the same way, for example pip install apache-airflow-providers-postgres --constraint ....
  • Permission errors in Docker on Linux: set AIRFLOW_UID as shown.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do you install Airflow with a constraint file?

    Constraint files pin hundreds of transitive dependencies to tested versions, avoiding surprising breakage.
  2. 2. Which statement about airflow standalone is correct?

    Standalone is a convenience for local learning, with SQLite as the database.
  3. 3. What is the environment variable that sets core.load_examples to False?

    The pattern is AIRFLOW__SECTION__KEY with two underscores around the section.
  4. 4. How do you run Airflow on a Windows machine?

    Airflow is not supported natively on Windows; WSL 2 and Docker are the usual routes.

Summary

  • Use a virtual environment, an AIRFLOW_HOME, and the official constraint file for your Airflow and Python versions.
  • airflow standalone is the quickest way to learn; Docker Compose looks closer to a real deployment.
  • Both are for development: production needs PostgreSQL or MySQL, secrets handling and backups.
  • Any config option can be set with AIRFLOW__SECTION__KEY.

Next: Your first DAG