Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 37 ยท Getting Started

What Is Apache Airflow? Workflows as Code

Learn what Apache Airflow is, what problems workflow orchestration solves, when to use it (and when not), and how DAGs, tasks and schedules fit together.

Apache Airflow is an open-source platform for authoring, scheduling and monitoring workflows. A workflow here is any set of jobs that must run in a certain order: download a file, clean it, load it into a database, build a report, send an email. You describe the workflow in Python code, and Airflow takes care of running the steps on time, in order, retrying the ones that fail and showing you what happened.

Before you write any code it helps to see why such a tool exists. This lesson explains the problem Airflow solves, the vocabulary you will meet in every later lesson, and what Airflow is deliberately not good at.

About the examples. Airflow needs a metadata database, a scheduler and a web server, so it cannot run inside this website. Real Airflow code is shown in read-only blocks. Next to it you get small runnable Python programs that are clearly labelled stand-ins: they model the idea (ordering tasks, computing schedules) in plain Python so you can experiment. They are not Airflow itself.

The problem: cron is not enough

Imagine a nightly job that works like this: at 02:00 export orders, at 02:30 transform them, at 03:00 load them into a warehouse. With cron you write three lines and hope. Then reality shows up:

  • The export takes 45 minutes one night, and the transform starts on half a file.
  • The warehouse is down. Nobody notices until the CEO asks why the dashboard is stale.
  • You need to re-run last Tuesday after fixing a bug. With cron you edit scripts by hand.
  • A fourth job must wait for two upstream jobs.

An orchestrator fixes these by treating the workflow as data: tasks, dependencies, schedules, retries, logs and history are all first-class objects.

Core vocabulary

Term Meaning
DAG Directed Acyclic Graph: your workflow. Nodes are tasks, arrows are "runs before" relations, and there are no loops.
Task One unit of work inside a DAG (run a script, call an API, run SQL).
Operator A reusable template for a task (BashOperator, PythonOperator, ...).
Task instance One task in one specific run. It has a state such as success or failed.
DAG run One execution of the whole DAG for one point in time.
Scheduler The process that decides what should run now.
Executor The mechanism that actually launches the task (locally, on a Celery worker, in Kubernetes).

The word acyclic matters. If task A needs B and B needs A, nothing could ever start. Airflow refuses to load such a DAG.

flowchart LR
    E["extract"] --> T["transform"]
    T --> L["load"]
    T --> Q["quality check"]
    L --> R["report"]
    Q --> R

A small DAG: report waits for both load and quality check.

What Airflow code looks like

Here is a complete, real DAG (Airflow 3 style). Do not worry about every detail yet; later lessons take it apart.

Python
from datetime import datetime

from airflow.sdk import DAG
from airflow.providers.standard.operators.bash import BashOperator

with DAG(
    dag_id="hello_airflow",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    say_hello = BashOperator(task_id="say_hello", bash_command="echo 'Hello'")
    say_bye = BashOperator(task_id="say_bye", bash_command="echo 'Bye'")

    say_hello >> say_bye

Read it aloud: "a DAG called hello_airflow, running daily; say_hello runs, then say_bye." The >> operator draws the arrow.

Because the file is ordinary Python, you can generate tasks in loops, share helper functions, use version control and review changes in pull requests. That idea, workflows as code, is the biggest reason people choose Airflow.

A stand-in: ordering tasks by dependency

The heart of any orchestrator is "given tasks and their upstream dependencies, in which order can they run?" This stand-in answers that with a topological sort. It is not Airflow's code, only the underlying idea.

Python Stand-in: which task can run next?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that load and quality_check land in the same wave. A real scheduler would start both at once if it had free worker slots. This is how Airflow achieves parallelism without you writing threads.

Now see what happens when the graph has a cycle:

Python Stand-in: cycles are rejected

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Airflow does the same kind of check when it parses your file and reports a DagCycleException.

What Airflow is good at

  • Batch workflows that run on a schedule or when an event happens.
  • Dependencies across systems: databases, cloud storage, APIs, Spark, dbt, notebooks.
  • Operational visibility: logs per task, run history, a graph view, alerts.
  • Re-runs and backfills for past periods.
  • A huge ecosystem of provider packages (AWS, Google Cloud, Azure, Snowflake, Postgres, ...).

What Airflow is not

  • Not a streaming engine. For continuous event streams use Kafka, Flink or Spark Streaming. Airflow starts and watches jobs; it does not process records one by one.
  • Not a data-processing engine. Airflow should trigger a Spark job or a SQL statement rather than crunch gigabytes inside the worker process.
  • Not for sub-second latency. Scheduling overhead is measured in seconds.
  • Not a place to pass big data between tasks. Tasks exchange small values (see XComs); large data goes to storage.

A good rule: Airflow is the conductor, not the orchestra.

A short history

Airflow started at Airbnb in 2014, joined the Apache Incubator in 2016 and became a top-level Apache project in 2019. Version 2 (2020) added the TaskFlow API and a faster scheduler. Version 3 (2025) introduced a new API server, a Task SDK (airflow.sdk) and event-driven scheduling features. This course uses the Airflow 3 style and points out differences from Airflow 2 where they affect your code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the A in DAG stand for, and why does it matter?

    Acyclic means a task can never depend on itself through a chain. Without that, no valid run order would exist.
  2. 2. Which job is the best fit for Airflow?

    Airflow orchestrates scheduled batch workflows. It is not meant for low-latency event processing.
  3. 3. What is a task instance?

    A DAG run contains one task instance per task. Each instance has its own state, logs and try number.
  4. 4. Why is it a problem to do heavy data crunching inside an Airflow task?

    Airflow is the conductor. Push heavy work to systems built for it and keep tasks light.

Summary

  • Airflow lets you define, schedule and monitor workflows as Python code.
  • A workflow is a DAG of tasks; a DAG run is one execution, a task instance is one task in one run.
  • It beats cron when you need dependencies, retries, history, alerts and re-runs.
  • It orchestrates batch work; it is not a streaming or data-processing engine.
  • On this site the Airflow code is read-only; the Python programs are labelled stand-ins.

Next: Installing Airflow: pip, standalone and Docker