Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 36 ยท Notebooks and Jobs

Multi-Task Workflows and Dependencies

Build Databricks workflows with several tasks: depends_on, run-if conditions, if/else and for-each tasks, task values, and a runnable dependency-graph stand-in.

Real pipelines are rarely one notebook. You extract data, validate it, transform it into several tables, then refresh a dashboard. A Databricks job can hold many tasks and a directed acyclic graph (DAG) of dependencies between them. Databricks starts every task whose upstream tasks are done, so independent branches run in parallel.

Databricks cannot run on this site. Job definitions are read-only; the runnable blocks are plain Python stand-ins that execute a dependency graph the way the scheduler reasons about it.

Dependencies

Each task has a task_key and, optionally, a depends_on list:

YAML
tasks:
  - task_key: extract_orders
    notebook_task: { notebook_path: ./extract_orders }
  - task_key: extract_customers
    notebook_task: { notebook_path: ./extract_customers }
  - task_key: build_silver
    depends_on:
      - task_key: extract_orders
      - task_key: extract_customers
    notebook_task: { notebook_path: ./build_silver }
  - task_key: build_gold_daily
    depends_on: [ { task_key: build_silver } ]
    notebook_task: { notebook_path: ./gold_daily }
  - task_key: build_gold_customers
    depends_on: [ { task_key: build_silver } ]
    notebook_task: { notebook_path: ./gold_customers }

The two extracts start together. build_silver waits for both. The two gold tasks wait for silver and then run in parallel.

flowchart LR
    A["extract_orders"] --> C["build_silver"]
    B["extract_customers"] --> C
    C --> D["build_gold_daily"]
    C --> E["build_gold_customers"]

A job graph: two parallel sources, one join point, two parallel outputs.

Run-if conditions

By default a task runs only if all its dependencies succeeded. You can change that with a run_if condition:

Run if The task runs when
All succeeded (default) every upstream task succeeded
At least one succeeded any upstream succeeded
None failed upstream tasks succeeded or were skipped
All done upstream tasks finished, whatever the result
At least one failed a cleanup or alert branch
All failed every upstream failed

"All done" is perfect for a final "release the lock" or "send summary" task; "at least one failed" suits an alerting task.

Passing values between tasks

Tasks can publish small values (not data!) to later tasks:

Python
# in task extract_orders
dbutils.jobs.taskValues.set(key="row_count", value=1250)

# in a later task
rows = dbutils.jobs.taskValues.get(taskKey="extract_orders", key="row_count", default=0, debugValue=0)

In job settings you can also reference them as {{tasks.extract_orders.values.row_count}}. Use them for counts, file names or flags. Large data should go through tables.

Control-flow tasks

  • If/else condition task: compares two values (for example {{tasks.extract_orders.values.row_count}} > 0) and routes to a true or false branch.
  • For each task: runs a nested task once per item in an input list, with a concurrency limit. Good for "process each country" without writing 30 tasks.
  • Run job task: calls another job, so teams can build small reusable jobs.
YAML
- task_key: process_each_region
  for_each_task:
    inputs: '["EU","US","APAC"]'
    concurrency: 2
    task:
      task_key: process_region
      notebook_task:
        notebook_path: ./process_region
        base_parameters: { region: "{{input}}" }

Running a graph yourself

The stand-in below sorts tasks into waves: each wave contains tasks whose dependencies are done, which is exactly the parallelism a job gets. It also detects cycles, which a real job rejects.

Python Stand-in: execution waves of a DAG

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Failure handling with run-if

This stand-in runs the same graph when one task fails, and shows which tasks are skipped and which still run because of their run-if rule.

Python Stand-in: run-if rules after a failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Design tips

  • Keep tasks small and idempotent: re-running with the same date must give the same result.
  • One notebook per task, shared code in modules, parameters through widgets (see Widgets and Parameters).
  • Put data-quality checks as their own task between stages, so a bad load stops the graph early.
  • Use repair run after a fix; successful tasks are not repeated.
  • Prefer a declarative pipeline task for table-to-table transformations; keep the job for orchestration.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two tasks have no dependency on each other. What happens?

    Databricks runs every task whose dependencies are satisfied.
  2. 2. Which run-if rule suits a final cleanup that must happen even after failures?

    All done fires when upstream tasks finished, no matter how.
  3. 3. What are task values meant for?

    Large data belongs in tables; task values are tiny messages.
  4. 4. A for-each task is best for:

    It fans out one nested task per input with a concurrency limit.

Summary

  • depends_on builds a DAG; independent tasks run in parallel.
  • run_if rules, if/else and for-each tasks add control flow.
  • Task values pass small messages; tables carry data.
  • Make tasks idempotent, add quality gates and use repair runs.

Next: The Unity Catalog Namespace