Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 37 ยท Projects and Practice

Airflow Practice: Exercises, Debugging Scenarios and Review Quiz

Test your Apache Airflow knowledge with debugging scenarios, DAG design exercises, runnable Python stand-in challenges and a mixed review quiz covering XComs, sensors, branching and ops.

You have seen the pieces: XComs, variables, connections, sensors, branching, mapping, executors, testing, monitoring and deployment. This lesson puts them to use with scenarios in the style of real work and interviews. Try each exercise before opening the discussion under it.

Airflow cannot run on this site. Airflow code is read-only; the runnable exercises are stand-ins in plain Python that train the same reasoning.

Scenario 1: The join that never runs

A DAG has choose (a branch task), two paths fast and slow, and a publish task downstream of both. On every run publish is skipped. Why, and what is the fix?

Discussion: publish uses the default all_success. One of its parents is always skipped, so the rule is never met. Set trigger_rule="none_failed_min_one_success" on publish (branching, trigger rules).

Scenario 2: The DAG that makes the scheduler slow

A DAG file starts with CONFIG = requests.get("https://config.internal/pipelines").json() and builds tasks from it. The UI sometimes loses the DAG and the scheduler lags. What is wrong?

Discussion: module-level network code runs on every parse. Fetch the config in a separate job and ship it as a file, or read it inside a task (dynamic DAGs).

Scenario 3: Doubled revenue

After a worker crashed and the task was retried, a report shows twice the revenue for one day. Which property was missing, and how do you repair the load task?

Discussion: idempotency. Replace the partition (delete by interval, then insert), or upsert on a natural key (idempotent backfills).

Scenario 4: Eighty idle slots

Fifty sensors wait for files in the default mode and the cluster cannot start real work. What do you change?

Discussion: use mode="reschedule" or deferrable sensors, set a pool, and set realistic timeouts (sensors, executors).

Exercise A: order the tasks

Given upstream relationships, produce a valid run order (a topological sort). This is exactly what a scheduler does to decide what can run.

Python Exercise A: topological order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how tasks in the same "ready" batch (extract_a, extract_b) could run in parallel. How many slots would make this run fastest?

Exercise B: write a retry policy

Implement a function that returns how many seconds to wait before attempt n with exponential backoff capped at a maximum, then use it for six attempts. Then decide: which errors should not be retried at all?

Python Exercise B: backoff schedule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise C: spot the bugs

Read this DAG and list at least five problems before reading the answers.

Python
import requests
from airflow.sdk import dag, task, Variable
from datetime import datetime

API_KEY = Variable.get("api_key")                      # (1)
DATA = requests.get("https://example.com/list").json() # (2)

@dag(schedule="@daily", start_date=datetime.now())     # (3)
def bad():
    @task
    def fetch():
        today = datetime.now().date()                  # (4)
        return requests.get(URL, params={"d": today}).json()   # (5) big XCom?

    @task
    def load(rows):
        db.execute("INSERT INTO t VALUES (?)", rows)   # (6)
        print("key", API_KEY)                          # (7)

    load(fetch())

Answers: (1) and (2) run at parse time (variables); (3) start_date must be fixed, not now(), or runs are never scheduled correctly; (4) use the logical date, not the clock; (5) a full API response in an XCom may be huge, so store it and return a path (XComs); (6) a blind insert is not idempotent; (7) printing a secret to the log. Also URL and db are undefined.

Exercise D: capacity planning

You have a pool of 4 slots and 10 independent tasks that take 3, 3, 3, 3, 3, 3, 3, 3, 3 and 12 minutes. When does the run finish, and how would you improve it?

Python Exercise D: pool simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Starting the longest task first (priority weights) gives a shorter run; extra slots stop helping once the 12-minute task is the bottleneck. Splitting it with mapping is the real improvement.

Mixed review quiz

Quick check ยท 6 questions

of 6 answered You scored / 6 ยท perfect!

  1. 1. A task needs a database password. Where should it be stored?

    Connections (with a secrets backend) keep credentials out of code, logs and Git.
  2. 2. A sensor waits up to 12 hours for a file. What is the best mode?

    Long waits should free the worker slot between checks.
  3. 3. You need one task instance per file, with the number of files known only at run time. Use:

    Dynamic task mapping creates instances at run time from upstream output.
  4. 4. Which executor runs each task in its own pod?

    KubernetesExecutor launches a pod per task instance.
  5. 5. A cleanup task must run even if earlier tasks failed. Which trigger rule?

    all_done runs after every parent has finished, whatever the result.
  6. 6. Why enable remote logging?

    Logs stored only on local disk are lost when containers are replaced.

Where to go next

Build something real: pick a data source you care about, write the DAG, put it in Git, add the integrity test and run it locally with airflow standalone or Docker Compose. Then add a sensor, a retry policy and a failure notifier. When you can explain why each part is there, you understand Airflow. Return to the Apache Airflow track page any time to revisit a chapter.

Summary

  • Most Airflow bugs come from a few causes: parse-time work, non-idempotent writes, wrong trigger rules, blocked slots and misuse of XComs.
  • Practise by reasoning about scheduling: dependencies, slots, intervals and states.
  • Test, monitor and alert on every pipeline you ship.

Next: Back to the Apache Airflow track