Apache Airflow Tutorial
Learn Apache Airflow step by step: 37 lessons in 9 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Apache Airflow? Workflows as Code Learn what Apache Airflow is, what problems workflow orchestration solves, when to use it (and when not), and how DAGs, tasks and schedules fit together.
- Installing Apache Airflow: pip, Standalone and Docker Install Apache Airflow step by step: virtual environment, constraint files, airflow standalone, Docker Compose, AIRFLOW_HOME, and the first checks to run.
- Your First Airflow DAG: Write, Test and Run Write your first Apache Airflow DAG with BashOperator and PythonOperator, set dependencies with >>, test it from the command line and read the results in the UI.
- The Airflow UI and Command Line: A Guided Tour Tour the Apache Airflow web UI (DAG list, grid, graph, logs, triggers) and the most useful CLI commands for listing, testing, triggering and pausing DAGs.
Concepts and Architecture
- Airflow Architecture: Scheduler, Executor, Workers and Database Understand how Apache Airflow works inside: scheduler, DAG processor, API server, triggerer, executor, workers and the metadata database, and how a task travels between them.
- DAG Runs and Task Instances in Airflow Learn how Airflow represents one execution of a DAG, how run ids and run types work, and how tasks become task instances with try numbers, pools and priorities.
- Airflow Task Lifecycle: States, Retries and Timeouts Follow a task through every Airflow state from scheduled to success, and learn how retries, retry_delay, exponential backoff, timeouts and callbacks shape failures.
- What Changed in Airflow 3: Task SDK, API Server and More Learn the main differences between Airflow 2 and Airflow 3: the Task SDK, API server, DAG versioning, assets, new imports and removed features, with migration tips.
DAGs and Operators
- Ways to Define an Airflow DAG: Context Manager, Decorator, Object Compare the three ways to declare a DAG in Apache Airflow: the with statement, the @dag decorator and the constructor, plus DAG arguments, naming and file layout rules.
- The Airflow TaskFlow API: @task and Passing Data Write Airflow DAGs with plain Python functions using the TaskFlow API: the @task decorator, return values as XComs, multiple outputs, dependencies and mixing in operators.
- Airflow Operators: BashOperator, PythonOperator and EmptyOperator Learn what Airflow operators are and use the core ones: BashOperator, PythonOperator, EmptyOperator and others, with arguments, return codes, op_kwargs and a map of provider operators.
- Custom Airflow Operators and Hooks: Reusable Task Code Learn when and how to write a custom Airflow operator by extending BaseOperator, how hooks wrap connections to external systems, template_fields, and testing custom code.
- Airflow Templating with Jinja: ds, Macros and Rendered Fields Use Jinja templates in Apache Airflow: {{ ds }}, logical_date, data_interval, params, macros, custom filters, template files and how to debug rendered values.
Scheduling and Dependencies
- Airflow Schedules: Cron Expressions, Presets and Timetables Learn how to schedule Airflow DAGs with cron expressions, presets such as @daily, timedelta intervals, None for manual runs and custom timetables, with time zone tips.
- Airflow Data Intervals and the Logical Date Explained Understand Airflow data intervals, logical_date, data_interval_start and end, and why a daily run covers yesterday's data. Includes interval and trigger timetables and examples.
- Airflow Catchup, Backfills and Re-running Past Data Learn what catchup does in Apache Airflow, how backfills re-process historical dates, how to clear and re-run tasks, and how to keep reruns safe with idempotent tasks.
- Airflow Task Dependencies: >>, chain, Fan-out and Fan-in Wire Airflow tasks together with >>, <<, lists, chain() and cross_downstream(), build fan-out and fan-in patterns, avoid cycles, and use depends_on_past and edge labels.
- Airflow Task Groups: Organizing Large DAGs Visually Organize big Apache Airflow DAGs with TaskGroup and @task_group: prefixes, nesting, default arguments, group-level dependencies, and why SubDAGs are gone.
XComs, Variables and Connections
- XComs: Passing Small Messages Between Airflow Tasks Learn how Airflow XComs let tasks share small values: push and pull, return_value, keys, templates, size limits and when not to use XCom. Runnable stand-ins included.
- The TaskFlow API: Passing Data Between Tasks Cleanly Use Airflow's TaskFlow API (@task) to pass values between tasks without manual XCom calls: multiple outputs, mixing with classic operators and common mistakes.
- Airflow Variables and Params: Configuring DAGs Without Editing Code Learn Airflow Variables, environment-variable overrides, Jinja access, JSON values and DAG Params for run-time input, plus the parse-time cost trap. Runnable stand-ins.
- Airflow Connections and Hooks: Talking to External Systems Learn how Airflow Connections store host, login and secrets, how URIs and environment variables define them, and how Hooks use them to reach databases and APIs.
Sensors, Branching and Dynamic DAGs
- Airflow Sensors: Waiting for Files, Time and Other Events Understand Airflow sensors: poke and reschedule modes, poke_interval, timeout, soft_fail, deferrable sensors and common built-in sensors, with a runnable poke-loop stand-in.
- Branching in Airflow DAGs: Choosing a Path at Run Time Learn how Airflow branching works: @task.branch, BranchPythonOperator, skipped states, joining branches with trigger rules, ShortCircuit and common branching bugs.
- Airflow Trigger Rules: When Does a Task Run? Learn every common Airflow trigger rule (all_success, all_done, one_failed, none_failed and more), how upstream states decide, cleanup patterns and a runnable rule evaluator.
- Dynamic Task Mapping in Airflow: One Task per Item Learn dynamic task mapping in Airflow: expand, partial, mapped results, reduce steps, expand_kwargs, limits and when to prefer it over loops. Runnable map-reduce stand-in.
- Dynamic DAG Generation in Airflow: DAG Factories and Config Files Generate many similar Airflow DAGs from a config with a factory function: the globals() pattern, parse-time rules, config-driven pipelines and pitfalls. Runnable stand-in.
Providers and Executors
- Airflow Providers: Packages for AWS, Google, Databases and More Learn how Airflow provider packages add operators, hooks, sensors and connection types for services like AWS, GCP, Postgres and Slack, how to install and pin them and find the right one.
- Airflow Executors: Local, Celery, Kubernetes and Parallelism Understand Airflow executors: LocalExecutor, CeleryExecutor, KubernetesExecutor, multiple executors, parallelism, pools and slots, with a runnable scheduling simulation.
- Sharing Code Across Airflow DAGs: Packages, Plugins, XCom Backends Organise shared Airflow code: the include folder, internal Python packages, plugins versus packages, version drift between components and custom XCom backends. Runnable stand-ins.
Testing, Monitoring and Deployment
- Testing Airflow DAGs: Integrity Checks, Unit Tests and dag.test() Learn how to test Airflow DAGs: import and integrity tests with DagBag, unit tests for task logic, dag.test() for a full local run, mocking connections and CI tips.
- Airflow Logging and Monitoring: Logs, Metrics and Health Checks Learn where Airflow task logs live, remote logging, structured log messages, StatsD and OpenTelemetry metrics, health endpoints and what to watch in production.
- Airflow Alerting: Retries, Callbacks, Notifiers and Deadlines Make Airflow tell you when something breaks: retries with backoff, on_failure_callback, notifiers (Slack, email), timeouts, SLA replacements and alert design that avoids noise.
- Deploying Airflow: Docker, Helm, CI/CD and Configuration Learn how to deploy Airflow to production: components, Docker images, Helm on Kubernetes, DAG delivery with Git sync, AIRFLOW__ configuration, secrets and a CI/CD checklist.
Projects and Practice
- Project: A Daily ETL Pipeline in Airflow Build a complete daily ETL DAG: extract, validate, transform, load into a database idempotently, with retries and a quality gate. Runnable pipeline stand-in with SQLite.
- Project: Idempotent Backfills and Reruns in Airflow Learn data intervals, catchup and backfills by building an idempotent partitioned pipeline: compute run dates, rerun safely, avoid duplicates and handle late data.
- Airflow Practice: Exercises, Debugging Scenarios and Review Quiz Test your Apache Airflow knowledge with debugging scenarios, DAG design exercises, runnable Python stand-in challenges and a mixed review quiz covering XComs, sensors, branching and ops.