Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 37 ยท Getting Started

Your First Airflow DAG: Write, Test and Run

Write your first Apache Airflow DAG with BashOperator and PythonOperator, set dependencies with >>, test it from the command line and read the results in the UI.

In this lesson you write a complete DAG, put it where Airflow can find it, test it without waiting for the scheduler, and learn what every line does. The DAG fetches a "greeting", transforms it and prints a summary: tiny, but it has the same anatomy as a real pipeline.

The DAG files below are real Airflow 3 code and are read-only here. Below them you can run a stand-in that imitates how a DAG object collects tasks and runs them in order, so you can see the mechanics in Python you can edit.

The DAG file

Save this as ~/airflow/dags/hello_pipeline.py:

Python
from datetime import datetime

from airflow.sdk import DAG
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.operators.python import PythonOperator


def build_summary():
    words = ["airflow", "makes", "workflows", "visible"]
    print(" ".join(w.capitalize() for w in words))


with DAG(
    dag_id="hello_pipeline",
    description="My first DAG",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,
    tags=["tutorial"],
) as dag:
    fetch = BashOperator(
        task_id="fetch",
        bash_command="echo 'fetching data...'",
    )

    summarize = PythonOperator(
        task_id="summarize",
        python_callable=build_summary,
    )

    done = BashOperator(
        task_id="done",
        bash_command="echo 'all finished'",
    )

    fetch >> summarize >> done

What each part means

Imports. DAG comes from airflow.sdk in Airflow 3. In Airflow 2 you imported it from airflow and the operators from airflow.operators.bash; the old paths still work for a while with deprecation warnings. The standard operators (BashOperator, PythonOperator, EmptyOperator...) live in the apache-airflow-providers-standard package.

dag_id. The unique name of the DAG. It appears in the UI and on the command line. Make it lowercase with underscores.

start_date. The earliest date the DAG may have runs for. It is a point in time, not "when the DAG starts now". Use a fixed date in the past, never datetime.now(), which changes on every parse.

schedule. How often the DAG runs: a cron string, a preset like @daily, a timedelta, or None for manual only. See Scheduling.

catchup=False. Do not create runs for every missed interval between start_date and today. In Airflow 3 this is already the default, but writing it makes the intent visible.

tags. Labels for filtering in the UI.

Tasks. Each operator instance is a task. task_id must be unique inside the DAG.

Dependencies. fetch >> summarize >> done means "fetch, then summarize, then done".

Dependency syntax

Python
a >> b            # a runs before b
b << a            # same thing
a >> [b, c]       # fan out: b and c both wait for a
[b, c] >> d       # fan in: d waits for both
a.set_downstream(b)   # method form

The >> form is by far the most common.

Where Airflow looks for DAG files

The scheduler's DAG processor scans the dags/ folder on a timer and imports every .py file that contains the words dag and airflow (a cheap safe-mode check). It runs the file, collects every DAG object it finds, and stores the serialized structure in the metadata database. The UI reads that copy.

Two consequences:

  1. Your file is executed repeatedly just to read its structure. Keep top-level code fast: no database calls, no API requests, no heavy imports.
  2. A syntax error or failed import does not break Airflow. The DAG simply does not appear, and the UI shows an import error with the traceback.

Test it without the scheduler

Terminal
# check that the file imports
python ~/airflow/dags/hello_pipeline.py

# run the whole DAG once, in one process, without recording scheduler state
airflow dags test hello_pipeline

# run a single task
airflow tasks test hello_pipeline fetch 2025-01-01

airflow dags test is the fastest feedback loop while developing. It runs every task of the DAG in order in the current process and prints the logs to your terminal.

Run it for real

  1. Open the UI at http://localhost:8080.
  2. Find hello_pipeline. New DAGs are paused by default; flip the toggle to unpause.
  3. Press Trigger to start a manual run.
  4. Open the run, click a task square and choose Logs to see what the task printed.

Stand-in: a tiny DAG class

This stand-in has a Dag class that collects tasks and dependencies the way the with DAG(...) block does, then runs them in dependency order. The >> operator is implemented with __rshift__, which is exactly how Airflow offers the same syntax.

Python Stand-in: a mini DAG with the arrow operator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Change the line fetch >> summarize >> done to done >> fetch and run again to see the order change.

Stand-in: detecting duplicate task ids

Airflow raises an error if two tasks in a DAG share a task_id. This stand-in shows the same guard:

Python Stand-in: task_id must be unique

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common beginner mistakes

  • Using datetime.now() as start_date.
  • Putting API calls or database queries at the top level of the DAG file.
  • Forgetting to unpause the DAG and wondering why nothing runs.
  • Reusing a task_id inside one DAG.
  • Naming your file after a library (airflow.py, datetime.py), which shadows the real import.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does fetch >> summarize >> done mean?

    The >> operator draws a dependency: the left task runs before the right one.
  2. 2. Why should you avoid slow code (API calls, queries) at the top level of a DAG file?

    Top-level code runs on every parse, so slow code makes the scheduler slow and can hide DAGs.
  3. 3. What should start_date be?

    A dynamic start_date changes on every parse and leads to confusing schedules. Use a fixed date.
  4. 4. Which command runs a whole DAG once in your terminal without the scheduler?

    airflow dags test runs all tasks in one process and prints their logs, ideal while developing.

Summary

  • A DAG file defines a DAG object, tasks inside it and dependencies with >>.
  • dag_id, start_date, schedule and catchup are the key arguments.
  • The DAG processor imports your file repeatedly, so keep top-level code light.
  • Use airflow dags test and airflow tasks test for fast feedback; unpause the DAG to schedule it.

Next: The Airflow UI and command line