Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 37 ยท DAGs and Operators

Airflow Operators: BashOperator, PythonOperator and EmptyOperator

Learn what Airflow operators are and use the core ones: BashOperator, PythonOperator, EmptyOperator and others, with arguments, return codes, op_kwargs and a map of provider operators.

An operator is a template for one kind of task. "Run a shell command", "call a Python function", "run a SQL statement", "copy a file to S3" are all operators. When you create an operator inside a DAG, you get a task. When the DAG runs, you get a task instance. This lesson covers the core operators that ship with Airflow and the arguments every operator shares.

Operators need Airflow to execute, so the Airflow code is read-only. The stand-ins implement tiny versions of BashOperator-style and PythonOperator-style behaviour in plain Python.

Operators, tasks and task instances

TEXT
BashOperator          <- the class (template)
  task_id="say_hi"    <- a task: an operator configured inside a DAG
    ... run on Mar 5  <- a task instance: that task in one DAG run

Every operator has an execute(context) method. A worker calls it when the task runs. Anything the operator does is inside that method.

Arguments all operators accept

Argument Meaning
task_id Unique within the DAG (required).
retries, retry_delay Failure handling.
execution_timeout Maximum time per try.
trigger_rule When to run, relative to upstream outcomes.
pool, priority_weight Resource control.
queue Which worker queue (CeleryExecutor).
depends_on_past Wait for the previous run's same task to succeed.
owner, email_on_failure Ownership and alerts.
doc_md Task documentation in the UI.

EmptyOperator

Does nothing. Use it as a visual start/end marker or a join point in a graph.

Python
from airflow.providers.standard.operators.empty import EmptyOperator

start = EmptyOperator(task_id="start")
join  = EmptyOperator(task_id="join", trigger_rule="none_failed_min_one_success")

BashOperator

Runs a shell command. The task is success when the command exits with code 0; any other exit code is a failure. Exit code 99 means skipped by default.

Python
from airflow.providers.standard.operators.bash import BashOperator

backup = BashOperator(
    task_id="backup",
    bash_command="tar -czf /backups/data_{{ ds }}.tgz /data",
    env={"BACKUP_LEVEL": "full"},
    cwd="/tmp",
)

check = BashOperator(
    task_id="check_file",
    bash_command="test -s /backups/data_{{ ds }}.tgz || exit 1",
)

Important details:

  • The command is rendered with Jinja first ({{ ds }} is the run's logical date as YYYY-MM-DD; see Templating).
  • Use set -e-style safety for multi-line scripts: bash_command="set -euo pipefail; step1; step2". Otherwise only the exit code of the last command counts.
  • Pass secrets with env= or a Connection, not by writing them into the command string where they show up in logs.
  • A bash command ending in .sh (with a trailing space) is treated as a template file path; add the space deliberately or avoid the pattern accidentally.

PythonOperator

Calls a Python function in the worker process.

Python
from airflow.providers.standard.operators.python import PythonOperator

def greet(name, punctuation="!"):
    print(f"Hello, {name}{punctuation}")
    return "greeted"          # stored as an XCom

hello = PythonOperator(
    task_id="hello",
    python_callable=greet,
    op_args=["Ada"],                  # positional
    op_kwargs={"punctuation": "?"},   # keyword
)
  • op_args / op_kwargs are templated, so op_kwargs={"day": "{{ ds }}"} works.
  • The function's return value becomes an XCom named return_value.
  • The function may take **context or named context keys (ds, ti, params...), and Airflow passes only what you ask for.
  • Raise an exception to fail the task. Raise AirflowSkipException to skip it.

In many modern DAGs the @task decorator replaces PythonOperator; both create the same kind of task.

A map of other operators

Airflow ships a core (the standard provider) and hundreds of provider packages:

Need Where it lives
Shell, Python, empty, trigger another DAG apache-airflow-providers-standard
Run SQL on Postgres, MySQL, SQLite ...-postgres, ...-mysql, ...-sqlite (via common-sql)
Cloud storage and services ...-amazon, ...-google, ...-microsoft-azure
Spark, Databricks, Snowflake, dbt ...-apache-spark, ...-databricks, ...-snowflake, ...-dbt-cloud
Containers ...-docker, ...-cncf-kubernetes
HTTP calls, email, Slack ...-http, ...-smtp, ...-slack

Install a provider with pip using the same constraint file as Airflow, then import its operators. Providers are covered in detail in chapter 7.

Choosing an operator

  • Shell tools and scripts: BashOperator (or @task.bash).
  • Your own Python logic: @task/PythonOperator.
  • A service with a provider operator: use it; it handles auth, retries and logging better than a hand-rolled script.
  • Heavy dependencies that conflict with Airflow's: run in a container or a virtualenv task.

Stand-in: a BashOperator-like task

This stand-in runs a command with Python's subprocess-free sandbox alternative: since a browser cannot start shells, it simulates the exit-code rule of BashOperator (0 = success, 99 = skipped, other = failed).

Python Stand-in: exit code to task state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: a PythonOperator-like task

This stand-in has a PythonTask class with op_args, op_kwargs, a return-value "XCom" and the skip exception.

Python Stand-in: python_callable with op_args

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Using PythonOperator for something a provider operator already does well.
  • Importing heavy packages at the top of the DAG file when only the callable needs them.
  • Forgetting that BashOperator only fails on a non-zero exit code of the whole script.
  • Storing large results as the return value (they go to XCom).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the relationship between an operator and a task?

    Operators are reusable templates; creating one in a DAG gives you a task.
  2. 2. A BashOperator command exits with code 1. What is the task state?

    Non-zero exit codes mean failure, except the special skip code 99 by default.
  3. 3. How do you pass keyword arguments to the Python function in PythonOperator?

    op_args gives positional arguments and op_kwargs gives keyword arguments.
  4. 4. What is EmptyOperator typically used for?

    It does no work and helps shape the graph, for example a start, end or join node.

Summary

  • Operators are task templates with an execute() method; tasks are configured operators inside a DAG.
  • EmptyOperator, BashOperator and PythonOperator cover markers, shell commands and Python functions.
  • All operators share arguments for retries, timeouts, pools and trigger rules.
  • Provider packages add hundreds of ready-made operators for services and databases.

Next: Custom operators and hooks