Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Providers and Executors

Airflow Providers: Packages for AWS, Google, Databases and More

Learn how Airflow provider packages add operators, hooks, sensors and connection types for services like AWS, GCP, Postgres and Slack, how to install and pin them and find the right one.

Airflow's core knows how to schedule and run tasks. It does not contain code to talk to Amazon S3, BigQuery, Snowflake or Slack. That code lives in separate, individually versioned Python packages called providers. This keeps the core small and lets integrations release fixes without waiting for a new Airflow version.

Airflow and its providers cannot run on this site. Airflow code is read-only. The runnable example is a stand-in that models how a provider registry turns a connection type into the right hook.

What a provider contains

A provider package, named apache-airflow-providers-<name>, can ship:

  • Operators (S3CopyObjectOperator) and sensors (S3KeySensor),
  • Hooks that wrap the vendor's client library (hooks),
  • Connection types (appear in the UI's connection form),
  • Triggers for deferrable operators, transfer operators (S3ToRedshiftOperator), notifiers, log handlers and secrets backends,
  • Extra metadata: version, minimum Airflow version, and dependencies.

Examples:

Package Adds
apache-airflow-providers-standard Bash, Python, branch, empty operators, file/time sensors (included in Airflow 3 by default)
apache-airflow-providers-amazon S3, Redshift, EMR, Athena, Lambda and more
apache-airflow-providers-google BigQuery, GCS, Dataproc, Vertex AI
apache-airflow-providers-postgres, -mysql, -snowflake Database hooks and operators
apache-airflow-providers-http, -ssh, -sftp Generic protocols
apache-airflow-providers-slack, -smtp Notifications
apache-airflow-providers-cncf-kubernetes Kubernetes pods and the Kubernetes executor

Installing providers

Providers are installed with pip next to Airflow:

Terminal
pip install apache-airflow-providers-postgres==6.1.0 apache-airflow-providers-amazon

Rules that save pain:

  1. Pin versions in a requirements or constraints file so every environment (laptop, CI, production) has the same code.
  2. Install providers on every component that runs DAG code: scheduler, DAG processor, workers and triggerer. The webserver needs them to display connection forms.
  3. Use the official constraints files (pip install ... --constraint constraints.txt) published for each Airflow version to avoid dependency conflicts.
  4. In Docker, bake providers into your custom image (FROM apache/airflow:<version> then pip install). Do not install packages at container start.
  5. Check the CLI: airflow providers list shows what is installed and airflow providers get <name> shows details.

Using a provider in a DAG

Python
from airflow.sdk import dag, task
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow.providers.postgres.hooks.postgres import PostgresHook
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def s3_to_postgres():
    wait = S3KeySensor(
        task_id="wait_for_file",
        bucket_name="incoming",
        bucket_key="orders/{{ ds }}.csv",
        aws_conn_id="aws_default",
        mode="reschedule",
        poke_interval=300,
    )

    @task
    def load():
        hook = PostgresHook(postgres_conn_id="warehouse_db")
        hook.run("INSERT INTO audit(msg) VALUES ('loaded')")

    wait >> load()

s3_to_postgres()

The import path tells you where the class lives: airflow.providers.<name>.<kind>.<module>. Provider code also supplies the default connection ids (aws_default, google_cloud_default, postgres_default).

How the registry finds hooks

When you create a connection with conn_type="postgres", Airflow has to know which hook goes with it. Each installed provider declares its connection types in package metadata, and Airflow's ProvidersManager reads them at startup. This stand-in models that lookup.

Python Stand-in: a provider registry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The failing case matches a real error message you will meet: if you forget to install the Snowflake provider, the connection type is missing from the dropdown and the DAG fails to import the operator.

Choosing and checking versions

  • Each provider has its own version (6.1.0) and states the minimum Airflow version it supports. Read the changelog before upgrading; major versions may drop old arguments.
  • Provider releases are independent, so upgrade a provider to get a bug fix without upgrading Airflow.
  • Deprecated import paths (for example old airflow.operators.bash) move into providers; Airflow 3 expects airflow.providers.standard.... Ruff's AIR rules can flag such imports for you.
  • The documentation page of each provider lists every operator, hook, sensor and connection type with example DAGs.
Python Stand-in: check provider versions against requirements

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Writing your own provider (or not)

If your team has many internal DAGs using the same system, a small internal Python package with a hook, an operator and perhaps a sensor works well, and it can be packaged as a real provider (declaring provider_info entry points) when you want a connection type in the UI. Most teams start with a plain shared package. See custom operators and hooks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are integrations such as S3 or BigQuery shipped as providers?

    Separate packages keep the core small and let integrations release fixes independently.
  2. 2. On which components must providers used by your DAGs be installed?

    Any process importing the DAG or executing tasks needs the same provider packages.
  3. 3. Which command lists installed providers?

    airflow providers list shows installed provider packages and versions.
  4. 4. What is the best way to keep environments consistent?

    Pinned, constrained, pre-built images avoid surprises between laptop, CI and production.

Summary

  • Providers are separate packages adding operators, hooks, sensors, connection types and more.
  • Install and pin them on every component that touches DAG code, ideally in a custom image.
  • Connection types are registered by providers so Airflow can pick the right hook.
  • Read changelogs and minimum Airflow versions before upgrading.

Next: Executors: How Tasks Actually Run