Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท XComs, Variables and Connections

Airflow Connections and Hooks: Talking to External Systems

Learn how Airflow Connections store host, login and secrets, how URIs and environment variables define them, and how Hooks use them to reach databases and APIs.

Almost every pipeline talks to something outside Airflow: a database, an object store, an HTTP API, a message queue. Putting hosts and passwords inside DAG code is unsafe and awkward. Airflow separates the how to reach it (a Connection) from the what to do with it (a Hook, used by operators and your own tasks).

Airflow and its providers are not available on this site, so Airflow code is read-only. The runnable examples are stand-ins that show how a connection URI is parsed and how a hook uses it.

What a Connection holds

A connection is a named record with these fields:

Field Meaning
conn_id The name you use in code, e.g. warehouse_db
conn_type Kind of system: postgres, mysql, http, aws, sftp...
host, port, schema Where to connect and which database/path
login, password Credentials (password is encrypted in the database)
extra A JSON object for everything else (SSL mode, region, timeouts)

You create connections in the UI (Admin, Connections), with the CLI (airflow connections add), in a secrets backend, or with an environment variable.

Defining a connection with an environment variable

The variable AIRFLOW_CONN_<CONN_ID> holds a URI. The connection id is the part after the prefix, lower-cased:

Terminal
export AIRFLOW_CONN_WAREHOUSE_DB='postgresql://loader:p%[email protected]:5432/sales?sslmode=require'

Special characters in the password must be URL-encoded (@ becomes %40). Query-string pairs become the extra JSON. Newer versions also accept a JSON representation, which avoids encoding problems.

Python Stand-in: parse a connection URI

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the password is masked when displayed. Airflow does the same in the UI and logs.

Hooks: the code that uses a connection

A hook is a Python class that knows how to open a connection of one type and exposes useful methods. Hooks take a conn_id, look the connection up, build a client and hand it to you. Operators (such as SQLExecuteQueryOperator) are thin wrappers that call a hook.

Python
from airflow.sdk import task
from airflow.providers.postgres.hooks.postgres import PostgresHook

@task
def count_orders() -> int:
    hook = PostgresHook(postgres_conn_id="warehouse_db")
    rows = hook.get_records("SELECT count(*) FROM orders")
    return rows[0][0]

The DAG file never contains a host or password. Change the connection in the UI or the secrets backend and every task picks it up. The same code can run against a test database in development and the real one in production just by defining a different warehouse_db in each environment.

Python Stand-in: a hook that reads a connection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Testing a connection and keeping it safe

  • The UI has a Test button, and airflow connections test <conn_id> runs the hook's test method. Testing from the UI is disabled by default in production for safety.
  • Use a secrets backend (AWS Secrets Manager, Google Secret Manager, HashiCorp Vault, Azure Key Vault) so credentials never sit in the Airflow database or in your Git repository.
  • Give each connection the least privileges it needs, for example a read-only user for extract tasks.
  • Never print a connection object or URI in a task log: the password may appear.
  • Never put credentials in extra unless the provider's documentation says so. Those fields are shown less carefully than passwords.

Reading a connection yourself

Sometimes you only need the fields, for instance to call a REST API with the requests library:

Python
from airflow.sdk import task, Connection

@task
def call_api():
    conn = Connection.get("crm_api")
    print(conn.host, conn.login)          # never print conn.password

Hooks for HTTP (HttpHook) already do the same and add retries and sessions. Prefer an existing hook from a provider package (next chapter) before writing your own; when none exists, write one (custom operators and hooks).

Common mistakes

  • Wrong conn_type: Airflow cannot find the right hook (or the right form in the UI).
  • Unencoded special characters in a URI password. Use %40, %2F, or the JSON format.
  • Creating the connection in only one environment, so the DAG fails elsewhere with "conn_id isn't defined".
  • Opening connections at parse time. Create hooks inside tasks only.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which environment variable defines a connection with the id warehouse_db?

    Connections use the AIRFLOW_CONN_ prefix followed by the upper-cased connection id. AIRFLOW_VAR_ is for Variables.
  2. 2. What is a hook?

    Hooks wrap the client library for one kind of system and are used by operators and tasks.
  3. 3. Where does the extra field of a connection store its data?

    extra is a JSON object holding anything that doesn't fit the standard fields.
  4. 4. Why should you create hooks inside tasks rather than at module level?

    The scheduler parses DAG files all the time. Network access during parsing slows everything and can break imports.

Summary

  • A Connection stores host, login, secret and extras under a conn_id; define it in the UI, CLI, environment or a secrets backend.
  • AIRFLOW_CONN_<ID> holds a URI; encode special characters.
  • Hooks turn a connection into a client; operators and tasks use hooks.
  • Keep credentials out of code, logs and extra, and open hooks only inside tasks.

Next: Sensors