Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Providers and Executors

Sharing Code Across Airflow DAGs: Packages, Plugins, XCom Backends

Organise shared Airflow code: the include folder, internal Python packages, plugins versus packages, version drift between components and custom XCom backends. Runnable stand-ins.

With ten DAGs, copy-paste is tolerable. With a hundred, a bug fixed in one place stays alive in ninety-nine others. Shared code (helper functions, hooks, operators, constants) needs a home that every Airflow component can import, with versions you control. This lesson covers the options, from a simple folder to an internal package, and then a classic advanced extension point: the custom XCom backend.

Airflow cannot run on this site. Airflow code is read-only; the runnable examples are stand-ins in plain Python. Writing the classes themselves (hooks and operators) is covered in custom operators and hooks.

Option 1: an include folder

Airflow adds the dags/ folder (and plugins/ and config/) to sys.path. A common layout:

TEXT
dags/
  orders_etl.py
  include/
    __init__.py
    transforms.py
    sql/orders.sql
Python
from include.transforms import add_tax

This is easy and works with git-sync. The downsides: no version number, every DAG sees the current code instantly (a mistake breaks all DAGs at once), and unit tests must set up the path.

Option 2: an internal Python package

Put the shared code in its own repository with a pyproject.toml, tests and a version, build a wheel, and install it in your Airflow image next to the providers:

TOML
[project]
name = "company-airflow"
version = "1.4.0"
dependencies = ["apache-airflow>=3.0"]
Python
from company_airflow.hooks.crm import CrmHook
from company_airflow.operators.export import CrmToFileOperator

Benefits: semantic versions, normal CI, rollback by pinning the previous version, the same code in all components, and tests without Airflow's folder tricks. Teams adopt it as soon as shared code has more than one owner.

Option 3: plugins

The plugins/ folder lets you add Airflow extensions such as macros, listeners, timetables and UI extensions. Operators and hooks no longer need a plugin to be importable (a plain module works), and plugins are loaded at startup by several components, so keep them small. Prefer a package unless you specifically need a plugin hook.

Keep every component on the same version

The most confusing production bug is version drift: the scheduler imports company-airflow 1.4.0 but the workers run 1.3.2. The DAG parses fine, then the task fails with an AttributeError on a worker. Prevent it with one image built once and used by every component, plus a startup check.

Python Stand-in: detect version drift between components

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Designing the shared library well

  • Public API small and documented. Callers import from company_airflow.hooks, not from private files.
  • No work at import time. Importing the package must not open connections or read Variables, because the scheduler imports it on every parse (dynamic DAGs).
  • Backwards compatibility. Deprecate with warnings for one release before removing an argument.
  • Tests with fake hooks (testing).
  • A changelog so DAG authors know what a version bump does.

Advanced: a custom XCom backend

XComs store small values in the metadata database. A custom XCom backend changes where values are stored: it overrides serialize_value to write big values to object storage and keep only a reference in the database, and deserialize_value to fetch them back. Tasks keep using normal returns and arguments. The class is set with [core] xcom_backend.

Python
from airflow.sdk.bases.xcom import BaseXCom

class S3XComBackend(BaseXCom):
    PREFIX = "xcom_s3://"
    THRESHOLD = 32 * 1024

    @staticmethod
    def serialize_value(value, **kwargs):
        raw = json.dumps(value).encode()
        if len(raw) < S3XComBackend.THRESHOLD:
            return BaseXCom.serialize_value(value)            # small: normal path
        key = f"xcom/{kwargs['dag_id']}/{kwargs['run_id']}/{kwargs['task_id']}.json"
        upload_to_s3(key, raw)
        return BaseXCom.serialize_value(S3XComBackend.PREFIX + key)

    @staticmethod
    def deserialize_value(result):
        value = BaseXCom.deserialize_value(result)
        if isinstance(value, str) and value.startswith(S3XComBackend.PREFIX):
            return json.loads(download_from_s3(value[len(S3XComBackend.PREFIX):]))
        return value

(Method signatures differ between Airflow versions; check the documentation of yours.) Pros: large values become possible and the database stays small. Cons: you must clean old objects, the data location is hidden from DAG readers, and a storage outage now breaks XCom reads. Many teams prefer explicit paths in plain XComs for visibility.

Python Stand-in: offload big values, keep a reference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing

Need Choice
A few helpers, one team include/ folder
Several teams, versions, rollbacks internal package in the image
Macros, listeners, UI extensions plugin
Large task outputs without changing DAG code custom XCom backend (use sparingly)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a major advantage of an internal package over an include folder?

    A versioned package gives controlled upgrades and rollbacks, unlike a folder that changes under every DAG at once.
  2. 2. What is version drift?

    When scheduler and workers differ, DAGs may parse fine and then fail at run time.
  3. 3. What does a custom XCom backend store in the metadata database for a big value?

    The value goes to object storage; the database keeps only a pointer that is resolved on read.
  4. 4. Why must shared library imports avoid doing work at import time?

    Work done at import time runs on every parse, slowing the scheduler and risking failures.

Summary

  • Start with an include/ folder; move to a versioned internal package when sharing grows.
  • Ship one image so every component has identical shared code and providers.
  • Keep imports free of side effects and the public API small.
  • A custom XCom backend offloads big values but hides data locations; use it deliberately.

Next: Testing Airflow DAGs