Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 36 ยท Platform and Workspace

Compute and Clusters

Learn the kinds of Databricks compute, how a cluster is made of a driver and workers, autoscaling, auto-termination, runtimes, access modes and cost control.

Compute is the machinery that runs your code. In Databricks you rarely own a server; you create a cluster (a group of virtual machines) or use serverless compute, and Databricks starts, configures and stops it for you. You pay for the time it runs, so knowing the types and the cost levers matters as much as knowing how to start one.

Anatomy of a cluster

A cluster has one driver and zero or more workers. Your notebook code runs on the driver; when you use Spark, the driver splits the job into tasks and sends them to the workers. A single-node cluster has only a driver, which is fine for small data and pure Python.

flowchart LR
    N["Notebook"] --> D["Driver<br/>plans the job"]
    D --> W1["Worker 1<br/>executors"]
    D --> W2["Worker 2<br/>executors"]
    D --> W3["Worker 3<br/>executors"]

The driver coordinates; workers do the parallel work. More workers means more parallel tasks, not a faster single task.

Types of compute

Type Best for Notes
All-purpose cluster Interactive notebooks Shared by people, stays up until stopped or auto-terminated
Job compute Scheduled production runs Created per run, ends when the job ends, cheaper
SQL warehouse SQL queries and dashboards Tuned for BI, sizes like T-shirts (see SQL warehouses and serverless)
Serverless compute Notebooks and jobs without cluster management Starts in seconds; Databricks manages sizing
Pools Faster starts with idle ready instances Classic compute only

Free Edition provides serverless only, so you will not create clusters there. The concepts below still matter in paid workspaces and in interviews.

Important cluster settings

  • Databricks Runtime (DBR) is the software image: Spark, Delta Lake, Python and many libraries. Pick a recent LTS (long-term support) version for stability.
  • Node type is the VM size (memory-optimised, compute-optimised, GPU).
  • Workers: a fixed number, or autoscaling between a minimum and a maximum.
  • Auto-termination stops an idle cluster after a number of minutes. Set it. An idle forgotten cluster is the classic surprise bill.
  • Access mode controls sharing and security: single user (one person or service principal) or standard shared compute, both compatible with Unity Catalog.
  • Spark config and environment variables tune behaviour.
  • Libraries installed on the cluster are available to every notebook attached.

A cluster definition is often written as JSON for automation (read-only example):

JSON
{
  "cluster_name": "etl-nightly",
  "spark_version": "15.4.x-scala2.12",
  "node_type_id": "i3.xlarge",
  "autoscale": { "min_workers": 2, "max_workers": 8 },
  "autotermination_minutes": 30
}

Versions and node names differ by cloud and change over time, so treat these values as illustrations.

Autoscaling

With autoscaling, Databricks adds workers when many tasks are waiting and removes them when idle. It helps with uneven workloads but cannot fix a job that has one giant task, because one task runs on one core. Autoscaling also does not help if the bottleneck is the driver, such as collecting a huge result.

Choosing sensibly

  1. Develop interactively on a small shared all-purpose cluster or serverless.
  2. Schedule production work on job compute or serverless, never on the shared dev cluster.
  3. Start with fewer, bigger or fewer, smaller nodes depending on whether the job is memory-bound or CPU-bound, then measure.
  4. Use spot (discounted, interruptible) instances for fault-tolerant batch work.
  5. Always set auto-termination.

Stand-in: an autoscaling decision

The stand-in below mimics the idea of autoscaling: given the number of waiting tasks, decide how many workers to run within a minimum and maximum. Real autoscaling uses more signals; this is only the principle.

Python Stand-in: autoscale between min and max workers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Costs are measured in DBUs (Databricks Units) per hour plus the cloud VM cost. The stand-in below estimates the cost of the same job on an always-on cluster versus one that auto-terminates. The prices are made-up numbers, only to show the shape of the calculation.

Python Stand-in: why auto-termination saves money

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Libraries and the environment

Install notebook-scoped Python libraries with %pip install in the first cell; they apply to your session only and do not disturb colleagues. Cluster-scoped libraries (set in the cluster config) apply to everyone attached. For repeatable jobs, pin versions in a requirements file so tomorrow's run matches today's.

Python
%pip install requests==2.32.3

After %pip you may need to restart the Python process so imports see the new package; Databricks offers dbutils.library.restartPython() for this.

Common mistakes

  • Running scheduled jobs on an all-purpose cluster (more expensive and fragile).
  • Leaving auto-termination off.
  • Adding workers to fix a slow job that is actually limited by one skewed partition (see Spark architecture).
  • Calling .collect() on big data and crashing the driver.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does your notebook's Python code run on a classic cluster?

    The driver runs your notebook code. Spark distributes DataFrame work to workers.
  2. 2. Which compute type is best for a nightly production job?

    Job compute is created for the run and ends afterwards, which is cheaper and isolates the job from interactive users.
  3. 3. What does auto-termination do?

    It prevents idle clusters from running and billing indefinitely.
  4. 4. Autoscaling cannot help when...

    A single task runs on one core, so extra workers do not speed it up.

Summary

  • A cluster is one driver plus workers; serverless hides the cluster management.
  • Use all-purpose compute for exploration and job compute or serverless for production.
  • Set a runtime, node types, autoscaling and always auto-termination.
  • Control cost by right-sizing and by stopping what you are not using.

Next: SQL Warehouses and Serverless