Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 36 ยท Notebooks and Jobs

Jobs and Schedules

Run Databricks code on a schedule or on events: create a job, choose task types, set cron or file-arrival triggers, pass parameters, and read run results.

Notebooks are great for exploring. Production work needs something that runs without you: every night, when a file arrives, or after another system finishes. In Databricks that something is a job (shown in the UI under Jobs & Pipelines, historically called Workflows).

A job is a named definition: what to run (tasks), on which compute, when (trigger), with which parameters, and what to do on failure. Each execution of that definition is a run.

Databricks cannot run here, so job definitions below are read-only. The runnable blocks are stand-ins in plain Python: a cron-field matcher and a tiny run recorder.

Task types

A task is one unit of work. The common types:

Task type Runs
Notebook A notebook from the workspace or a Git folder
Python script / wheel A .py file or an installed package entry point
SQL A query, alert or dashboard refresh on a SQL warehouse
Pipeline A declarative pipeline update (see Declarative Pipelines)
dbt A dbt project command
JAR / Spark submit Compiled Java or Scala code
Run job / If-else / For each Control flow between jobs and tasks

Defining a job

Through the UI you fill a form. In code the same thing is a small JSON (or YAML in Asset Bundles) document:

JSON
{
  "name": "nightly-orders",
  "tasks": [
    {
      "task_key": "load_orders",
      "notebook_task": {
        "notebook_path": "/Workspace/Repos/shop/etl/load_orders",
        "base_parameters": { "run_date": "{{job.start_time.iso_date}}" }
      },
      "job_cluster_key": "etl",
      "max_retries": 2,
      "timeout_seconds": 3600
    }
  ],
  "job_clusters": [
    { "job_cluster_key": "etl",
      "new_cluster": { "spark_version": "17.3.x-scala2.13", "node_type_id": "i3.xlarge", "num_workers": 2 } }
  ],
  "schedule": {
    "quartz_cron_expression": "0 30 2 * * ?",
    "timezone_id": "UTC",
    "pause_status": "UNPAUSED"
  },
  "email_notifications": { "on_failure": ["[email protected]"] }
}

The version number in spark_version is an example; use the runtime versions shown in your workspace.

Compute for jobs

  • Serverless compute starts in seconds and needs no sizing; Databricks manages it.
  • Job clusters are created for the run and terminated afterwards. They are cheaper than all-purpose clusters because they exist only for the job and are billed at a lower job rate.
  • All-purpose clusters are meant for interactive work. Running production jobs on them is usually more expensive and lets other users interfere.

Triggers

A job can start in five ways:

  1. Scheduled with a Quartz cron expression (second, minute, hour, day of month, month, day of week).
  2. Periodic, every N minutes/hours/days.
  3. File arrival, when new files land in a Unity Catalog volume or external location.
  4. Table update, when a Delta table changes.
  5. Continuous, restarting immediately after each run finishes (useful for streaming).

You can always start a job manually or through the REST API (POST /api/2.1/jobs/run-now) with parameters.

Quartz cron fields differ from Linux cron: they have a seconds field first, and either day-of-month or day-of-week must be ?.

Python Stand-in: reading a cron expression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Parameters and dynamic values

Job-level parameters are available to every task. Inside a task you can reference:

  • {{job.parameters.run_date}} for your own parameters,
  • {{job.start_time.iso_date}} and {{job.run_id}} for run facts,
  • {{tasks.extract.values.row_count}} for values one task publishes to later tasks with dbutils.jobs.taskValues.set(key="row_count", value=120).

The notebook side is the widget code from Widgets and Parameters.

Runs, retries and notifications

Every run has a state (Pending, Running, Succeeded, Failed, Skipped, Canceled) and a result per task, with logs and the notebook output saved. Useful settings:

  • Retries with an optional delay between attempts, for transient failures.
  • Timeout, so a stuck task does not run for days.
  • Max concurrent runs (default 1), so a slow night does not overlap the next one.
  • Notifications by email, Slack or webhook on start, success, failure or slow duration.
  • Repair run: re-run only the failed tasks without repeating successful ones.
Python Stand-in: a run with retries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Scheduling a job on an all-purpose cluster that other people restart.
  • Forgetting the time zone: UTC is safest; daylight saving can skip or repeat local times.
  • Using the default concurrency of 1 but a schedule shorter than the run time, so runs are skipped.
  • Hard-coding dates in notebooks instead of using parameters.
  • Sending failure emails to one person instead of a group.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the Quartz expression 0 30 2 * * ? mean?

    Fields are second, minute, hour, day of month, month, day of week.
  2. 2. Why prefer a job cluster or serverless over an all-purpose cluster for production?

    Isolation and cost: the compute exists only for the run.
  3. 3. What does 'repair run' do?

    It saves time and cost by keeping the successful tasks' results.
  4. 4. Which trigger starts a job when new files land in a volume?

    File arrival triggers watch a volume or external location.

Summary

  • A job defines tasks, compute, trigger, parameters and alerts; each execution is a run.
  • Use serverless or job clusters for production, cron/periodic/file-arrival/continuous for triggers.
  • Retries, timeouts, concurrency limits and repair runs make jobs robust.
  • Pass dates and settings as parameters, not hard-coded values.

Next: Multi-Task Workflows