Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท XComs, Variables and Connections

Airflow Variables and Params: Configuring DAGs Without Editing Code

Learn Airflow Variables, environment-variable overrides, Jinja access, JSON values and DAG Params for run-time input, plus the parse-time cost trap. Runnable stand-ins.

Pipelines need settings: a bucket name, a threshold, a list of customers. Hard-coding them means editing and redeploying code for every change. Airflow offers two tools. Variables are global key-value settings stored in Airflow. Params are inputs attached to a DAG that can be changed for each run, for example when you press "Trigger DAG w/ config".

Airflow cannot run on this site, so Airflow code is read-only. The runnable blocks are stand-ins in plain Python.

Variables

A Variable is a key and a string value (often JSON text) in the metadata database. You can create it in the UI (Admin, Variables), with the CLI (airflow variables set bucket my-data) or through an environment variable.

Python
from airflow.sdk import Variable

bucket = Variable.get("bucket")                       # raises if missing
retries = Variable.get("retry_count", default=3)      # default when missing
config = Variable.get("etl_config", deserialize_json=True)   # JSON text -> dict
Variable.set("last_run_note", "all good")

Environment variables win

If an environment variable named AIRFLOW_VAR_<NAME> exists, Airflow reads it before the database. AIRFLOW_VAR_BUCKET=my-data defines the variable bucket. Environment variables are fast (no database query), easy to manage as code and never appear in the UI. Airflow also supports configured secrets backends (such as a cloud secret manager) that are searched before the database.

The lookup order is: secrets backends, then environment variables, then the metadata database.

Python Stand-in: variable lookup order and JSON

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading variables the cheap way: Jinja

Variable.get() at the top of a DAG file runs on every parse, which happens every thirty seconds or so for every file. That means a database query each time, and a failure to import the DAG if the variable is missing. Prefer reading variables inside tasks or in templated fields, which are evaluated only when the task runs:

Python
from airflow.providers.standard.operators.bash import BashOperator

# good: evaluated at run time
copy = BashOperator(
    task_id="copy",
    bash_command="aws s3 sync /tmp/out s3://{{ var.value.bucket }}/daily/",
)

# JSON variable access
show = BashOperator(
    task_id="show",
    bash_command="echo batch size {{ var.json.etl_config.batch }}",
)

var.value.<key> gives the raw string, var.json.<key> parses JSON and lets you walk into it. Avoid this at module level:

Python
BUCKET = Variable.get("bucket")   # runs on every scheduler parse: avoid

Params: input per run

Params belong to a DAG (or a task). They have a default and optionally a schema, and are available in templates as {{ params.name }} and in Python as context["params"]. Anyone triggering the DAG can override them.

Python
from airflow.sdk import dag, task, Param

@dag(
    schedule=None,
    params={
        "country": Param("DE", type="string", enum=["DE", "FR", "ES"]),
        "limit": Param(100, type="integer", minimum=1, maximum=1000),
    },
)
def report():
    @task
    def build(params=None):
        print("country", params["country"], "limit", params["limit"])

    build()

report()

The UI shows a form generated from this schema. Airflow validates the values with JSON Schema before the run starts, so a wrong value (limit = 5000) is refused. Unlike Variables, Params are set per run: two runs of the same DAG can use different values at the same time.

Python Stand-in: Params with defaults and validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Variables or Params or XComs?

Need Use
A setting shared by many DAGs and rarely changed Variable (or env var)
A different input each time a run is triggered Param
A value produced by one task for the next in the same run XCom
A credential or host Connection

Tips

  • Store structured settings as one JSON variable, not twenty tiny ones, to reduce database reads.
  • Variables whose names contain words such as password, secret, api_key or token are masked in the UI and logs. Still, real secrets belong in a secrets backend.
  • Do not use a Variable as a cross-run counter. Concurrent runs can overwrite each other.
  • Production deployments often set AIRFLOW_VAR_* in the container environment so settings travel with the deployment.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does Airflow look for a variable first when AIRFLOW_VAR_BUCKET is set?

    Secrets backends come first, then environment variables, then the database. Environment variables avoid a database query.
  2. 2. Why is Variable.get() at the top of a DAG file discouraged?

    The scheduler parses DAG files constantly. Read variables inside tasks or via Jinja so they are read only at run time.
  3. 3. Which tool lets someone choose a value each time they trigger a DAG?

    Params are per-run inputs with defaults and optional JSON Schema validation.
  4. 4. How do you read a JSON variable inside a template?

    var.json parses the stored JSON text so you can walk into it with dots.

Summary

  • Variables are global settings; environment variables (AIRFLOW_VAR_*) and secrets backends take priority over the database.
  • Read variables in tasks or Jinja templates, not at module level.
  • Params are per-run inputs with defaults and schema validation.
  • Choose Variable, Param, XCom or Connection by who sets the value and when.

Next: Connections and Hooks