Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท DAGs and Operators

Custom Airflow Operators and Hooks: Reusable Task Code

Learn when and how to write a custom Airflow operator by extending BaseOperator, how hooks wrap connections to external systems, template_fields, and testing custom code.

When you copy the same Python function into ten DAGs, it is time to turn it into a reusable building block. Airflow has two layers for this. A hook is a Python class that knows how to talk to an external system (it handles the connection, authentication and API calls). An operator is a task template that uses hooks to do a unit of work. This lesson shows how to write both and how they fit together.

Custom classes inherit from Airflow base classes, so the real code is read-only. The stand-ins re-create the pattern (a base class with execute, a hook that owns the connection details) so you can run and modify it.

Do you need a custom operator?

Try these first, in this order:

  1. A provider operator that already exists.
  2. A @task function in a shared module (from include.utils import clean_rows).
  3. A custom operator when many DAGs need the same step with different parameters, or you want templated fields and a clear UI box.

A function is enough for most cases. A custom operator pays off for shared, parameterised, widely reused steps.

A custom operator

Python
from airflow.sdk import BaseOperator

class ThresholdCheckOperator(BaseOperator):
    """Fail the task when a value is above a threshold."""

    template_fields = ("table", "threshold")   # rendered with Jinja
    ui_color = "#ffe8cc"

    def __init__(self, *, table: str, threshold: int, **kwargs):
        super().__init__(**kwargs)
        self.table = table
        self.threshold = threshold

    def execute(self, context):
        count = self._count_rows()
        self.log.info("Table %s has %s rows", self.table, count)
        if count > int(self.threshold):
            raise ValueError(f"{self.table} has {count} rows, limit {self.threshold}")
        return count        # becomes the XCom return_value

    def _count_rows(self):
        return 42           # real code would query a database through a hook

Used in a DAG like any other operator:

Python
check = ThresholdCheckOperator(
    task_id="check_orders",
    table="orders_{{ ds_nodash }}",
    threshold=1000,
)

Key points:

  • Inherit from BaseOperator (airflow.sdk in Airflow 3; airflow.models.baseoperator in Airflow 2).
  • Accept keyword-only arguments and always call super().__init__(**kwargs) so task_id, retries and so on keep working.
  • Put the work in execute(self, context). The context dictionary holds the same things you can template (ds, ti, params, dag_run...).
  • template_fields lists attributes that Airflow renders with Jinja before execute runs.
  • Do not do expensive work in __init__: it runs every time the DAG file is parsed.
  • Use self.log rather than print; records show up in the task log with levels.
  • Raise an exception to fail; raise AirflowSkipException to skip.

Hooks: the connection layer

A hook hides the details of connecting to a system. It reads credentials from an Airflow Connection (by conn_id), builds a client, and exposes methods like get_records or upload_file. Operators call hooks, and so can your @task functions.

Python
from airflow.sdk import BaseHook
import requests

class ExampleApiHook(BaseHook):
    conn_name_attr = "example_conn_id"
    default_conn_name = "example_default"
    conn_type = "http"
    hook_name = "Example API"

    def __init__(self, example_conn_id: str = default_conn_name):
        super().__init__()
        self.example_conn_id = example_conn_id

    def get_conn(self) -> requests.Session:
        conn = self.get_connection(self.example_conn_id)
        session = requests.Session()
        session.headers["Authorization"] = f"Bearer {conn.password}"
        self.base_url = f"https://{conn.host}"
        return session

    def get_report(self, report_id: str) -> dict:
        session = self.get_conn()
        response = session.get(f"{self.base_url}/reports/{report_id}", timeout=30)
        response.raise_for_status()
        return response.json()

Why separate hook and operator?

  • The hook is reusable from anywhere and testable on its own.
  • The operator adds task behaviour: templating, logging, XCom, retries.
  • Credentials live in a Connection (in the UI, a secrets backend or an environment variable), never in code. See Connections.

Where to put the code

TEXT
dags/
plugins/            # on the Python path in older setups
include/
  my_company/
    hooks.py
    operators.py

Anything importable from the Airflow processes' PYTHONPATH works. For larger teams, package shared code as a normal Python package and install it into the Airflow image with pip. Changes to imported modules require the processes to reload them; the DAG processor picks up edits to DAG files faster than edits to deeper modules.

Turn big logic into a package, not into the DAG file

Keep the DAG file small: it should read like a table of contents (tasks and dependencies). Put SQL, API calls and data logic in modules you can unit test without Airflow.

Stand-in: operator base class with templating

This stand-in shows the shape of BaseOperator: keyword-only arguments, template_fields, and a simple {{ ds }} renderer applied before execute.

Python Stand-in: custom operator with template_fields

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: a hook that owns the connection

This hook reads a "connection" from a dictionary (standing in for the Airflow metadata store), and a separate function uses it. No network is used.

Python Stand-in: hook and connection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Testing custom operators

Unit test the logic without a scheduler. Call execute directly with a fake context, and patch hooks:

Python
def test_threshold_fails(monkeypatch):
    op = ThresholdCheckOperator(task_id="t", table="orders", threshold=10)
    monkeypatch.setattr(op, "_count_rows", lambda: 11)
    with pytest.raises(ValueError):
        op.execute(context={})

More on testing in chapter 8.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must a custom operator's init always do with **kwargs?

    BaseOperator handles task_id, retries and other shared arguments, so they must reach it.
  2. 2. What is the role of a hook?

    Hooks handle credentials and clients, while operators describe units of work.
  3. 3. What does template_fields do?

    Attributes named in template_fields can contain Jinja expressions such as {{ ds }}.
  4. 4. Where should expensive work (API calls) happen in a custom operator?

    init and module code run at every DAG parse; execute runs only when the task runs.

Summary

  • Prefer provider operators or shared @task functions; write a custom operator for widely reused parameterised steps.
  • Subclass BaseOperator, call super().__init__(**kwargs), implement execute, declare template_fields.
  • Hooks wrap connections to external systems and read credentials from Airflow Connections.
  • Keep DAG files thin, put logic in importable modules, and unit test execute directly.

Next: Templating with Jinja, params and macros