Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Testing, Monitoring and Deployment

Testing Airflow DAGs: Integrity Checks, Unit Tests and dag.test()

Learn how to test Airflow DAGs: import and integrity tests with DagBag, unit tests for task logic, dag.test() for a full local run, mocking connections and CI tips.

A DAG that fails to import does not fail loudly: it quietly vanishes from the UI, or shows a red banner while the pipeline stops running. A task with a wrong date calculation runs happily and loads wrong data. Tests catch these problems before deployment. In practice you need three layers: does the DAG load, is the logic right, and does the whole thing run end to end?

Airflow cannot run on this site. The Airflow and pytest snippets are read-only. The runnable examples are stand-ins: a miniature DagBag with cycle detection and plain test functions.

Layer 1: DAG integrity tests

An integrity test imports every DAG file and asserts there are no import errors and that your team's rules hold (tags, owners, retries, no cycles). With DagBag it takes a few lines:

Python
# tests/test_dag_integrity.py
import pytest
from airflow.models import DagBag

@pytest.fixture(scope="session")
def dagbag():
    return DagBag(dag_folder="dags/", include_examples=False)

def test_no_import_errors(dagbag):
    assert dagbag.import_errors == {}, dagbag.import_errors

def test_every_dag_has_owner_and_tags(dagbag):
    for dag_id, dag in dagbag.dags.items():
        assert dag.tags, f"{dag_id} has no tags"
        assert dag.default_args.get("owner"), f"{dag_id} has no owner"

def test_expected_dags_exist(dagbag):
    assert "sync_acme" in dagbag.dags

These tests are fast (they only import files) and are the first line in CI. Airflow itself checks for cycles during import, so a circular dependency (a >> b >> a) shows up as an import error.

A mini DagBag stand-in

This stand-in (not Airflow) holds DAGs as dictionaries and implements two of the checks: cycle detection and rule checking. It is a small version of what the real tests assert.

Python Stand-in: cycle detection and DAG rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Layer 2: unit tests for task logic

The best trick is to keep business logic in plain functions outside the DAG file, then test them like any Python code with no Airflow involved. Task functions can call them.

Python
# include/transforms.py
def add_tax(prices: dict[str, float], rate: float) -> dict[str, float]:
    if rate < 0:
        raise ValueError("rate must be >= 0")
    return {k: round(v * (1 + rate), 2) for k, v in prices.items()}
Python
# tests/test_transforms.py
import pytest
from include.transforms import add_tax

def test_add_tax():
    assert add_tax({"a": 10.0}, 0.2) == {"a": 12.0}

def test_negative_rate_rejected():
    with pytest.raises(ValueError):
        add_tax({"a": 1.0}, -0.1)

A @task function can be called directly in a test through its .function attribute (my_task.function(...)), which skips Airflow. For operators, instantiate them and call execute({}) with a fake hook or unittest.mock.patch:

Python
from unittest import mock

@mock.patch("company_airflow.operators.CrmHook")
def test_export(hook_cls, tmp_path):
    hook_cls.return_value.get_customers.return_value = [{"id": 1, "name": "Ada"}]
    op = CrmToFileOperator(task_id="t", since="2025-01-01",
                           output_path=str(tmp_path / "out.csv"))
    assert op.execute({}) == str(tmp_path / "out.csv")
Python Stand-in: tests with asserts and expected failures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Layer 3: run the whole DAG with dag.test()

dag.test() runs every task of a DAG in a single process, without a scheduler or workers, so you can debug it with print or your IDE's debugger:

Python
if __name__ == "__main__":
    dag.test()                     # python dags/my_dag.py

You can pass logical_date, run_conf (Params) and use_executor options. The CLI offers airflow dags test my_dag 2025-03-01 and airflow tasks test my_dag my_task 2025-03-01, which runs one task without recording state in the database.

Use test connections through environment variables (AIRFLOW_CONN_... pointing to a local test database or sqlite://) and test variables via AIRFLOW_VAR_..., so the run never touches production. See connections.

What to test and what not to

Test Why
Import errors, cycles, duplicate ids Catches broken DAGs before deploy
Naming, tags, owner, retries, schedule rules Enforces team conventions
Transform functions Where most bugs live
Operators with a fake hook Verifies arguments, templating, return values
A smoke run of dag.test() on small data Verifies wiring
Not: Airflow itself The scheduler is already tested

In CI

Run the integrity test and unit tests on every pull request, with the same Airflow and provider versions as production (see deploying Airflow). Keep tests fast: import time matters, because a slow DAG file also slows the real scheduler.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main purpose of a DagBag integrity test?

    Integrity tests import the DAG folder and assert import_errors is empty plus any conventions.
  2. 2. Where should you put business logic to make it easy to test?

    Plain functions can be tested with ordinary pytest tests without needing Airflow.
  3. 3. What does dag.test() do?

    It runs the DAG serially in-process, letting you use prints and debuggers.
  4. 4. How should a unit test avoid calling a real external API from an operator?

    A fake or patched hook returns fixed data so the test is fast and offline.

Summary

  • Test in three layers: integrity (imports, cycles, conventions), unit (logic and operators with fake hooks), end to end (dag.test()).
  • Keep logic in plain functions so tests need no Airflow.
  • Use test connections and variables from environment variables.
  • Run the same checks in CI with versions matching production.

Next: Logging and Monitoring