Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 36 ยท Spark and DataFrames

Spark Architecture: Driver, Executors, Partitions

Understand how Apache Spark runs on Databricks: driver and executors, partitions, jobs, stages and tasks, the shuffle, and why skew and small files hurt performance.

When you write df.groupBy("city").count() on a billion rows, something has to split that work across machines and bring the answers back. That something is Apache Spark. You can use Databricks for a long time without knowing the internals, but the day a job is slow or runs out of memory, this model is what lets you reason about the problem.

This lesson builds the mental model: who does what, how data is divided, and where the expensive step called the shuffle comes from.

The cast

  • Driver: one process that runs your program, builds a plan and schedules work.
  • Executors: processes on worker machines that run tasks and hold cached data.
  • Cluster manager: allocates machines (Databricks handles this).
  • Cores: each executor has several; one core runs one task at a time.
flowchart TD
    U["Your code"] --> D["Driver<br/>plan: jobs, stages, tasks"]
    D --> E1["Executor 1<br/>core core core core"]
    D --> E2["Executor 2<br/>core core core core"]
    E1 --> S[("Storage<br/>Delta / Parquet")]
    E2 --> S

The driver plans; executors read slices of the data from storage and process them in parallel.

Partitions: how data is divided

Spark does not move rows one by one. It divides a DataFrame into partitions, each a chunk of rows processed by one task. If your table has 200 partitions and the cluster has 40 cores, about 5 rounds of tasks run. Too few partitions leave cores idle. Too many tiny partitions waste time on scheduling overhead. A common target is partitions of roughly 100 to 200 MB.

Python
df = spark.read.table("main.sales.orders")      # read-only Databricks code
print(df.rdd.getNumPartitions())                # e.g. 64 (not available on serverless)
df2 = df.repartition(200)                       # full shuffle into 200 partitions
df3 = df.coalesce(10)                           # merge down without a full shuffle

On serverless compute RDD APIs are not available, so you rely on Spark's automatic tuning (Adaptive Query Execution) more than manual partition counts.

Jobs, stages and tasks

Running an action (like count() or write) triggers a job. Spark splits a job into stages at points where data must be rearranged across machines, then splits each stage into tasks, one per partition.

  • Narrow transformations (filter, select, withColumn) work inside one partition. They are pipelined together into one stage.
  • Wide transformations (groupBy, join, distinct, orderBy) need rows with the same key to meet on the same partition. That requires a shuffle: executors write intermediate data, send it over the network and read it back.

The shuffle is the most expensive step in most Spark jobs, so good Spark code tries to reduce what is shuffled: filter early, select only needed columns, and use broadcast joins when one side is small (see Joins, aggregations and window functions).

Reading the plan

You can ask Spark how it will run a query (read-only Databricks code):

Python
orders.groupBy("city").count().explain()

The printed physical plan lists steps such as Scan, HashAggregate and Exchange (the shuffle). Finding the Exchange nodes tells you where data moves. The Spark UI shows the same information as a graph of stages, with times per task, which is where you spot a slow straggler.

Common performance villains

  • Data skew: one key (say, a null or a popular customer) holds most of the rows, so one task works much longer than the rest. Adding workers does not help.
  • Small files: millions of tiny files make reading slow because of per-file overhead. See Delta OPTIMIZE.
  • Collecting to the driver: .collect() or .toPandas() on a large result pulls everything into one machine's memory.
  • Python UDFs: they move rows between the JVM and Python, which is slower than built-in functions.
  • Spilling: when memory is short Spark writes to disk, and time rises.

Spark 3 added Adaptive Query Execution (AQE), which re-optimises while the query runs: it can coalesce small shuffle partitions, switch a join strategy and split skewed partitions. It is on by default in Databricks.

Stand-in: partitions and a shuffle

The stand-in simulates the two-step shape of a distributed groupBy: each partition aggregates locally (a "map-side" step), then the partial results are routed by key so equal keys meet (the shuffle) and are combined. This is plain Python, not Spark, but it follows the same logic.

Python Stand-in: local aggregate, shuffle by key, combine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The next stand-in shows skew. We give each task the same speed (one row per time unit) and compare an even split with a skewed one. The job time is set by the slowest task.

Python Stand-in: skew makes one task the bottleneck

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A practical checklist when a job is slow

  1. Open the Spark UI and find the slowest stage.
  2. Look for one task far slower than the others (skew) or huge shuffle read/write sizes.
  3. Check whether you read more columns or rows than needed.
  4. Check the file layout of the source table (many small files?).
  5. Only then consider a bigger cluster.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What triggers a Spark job?

    Transformations are lazy. An action forces Spark to build and run a job.
  2. 2. Which of these needs a shuffle?

    groupBy must bring rows with the same key together, which moves data between executors.
  3. 3. A single task takes ten times longer than the others. What is the likely cause?

    Skew puts a large share of rows into one partition, so one task does most of the work.
  4. 4. Roughly how many tasks does a stage have?

    Each task processes one partition of data.

Summary

  • The driver plans work; executors run tasks on partitions in parallel.
  • Actions create jobs; shuffles split jobs into stages.
  • Wide transformations shuffle data, which is usually the main cost.
  • Skew, small files and driver collection are classic causes of slow jobs; AQE helps with some.

Next: DataFrame Basics