Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 48 ยท Under the Hood

Shuffles and Stages

Understand what triggers a Spark shuffle, how map and reduce sides exchange data, how stages are cut, what spill means, and how to read it in the Spark UI.

A shuffle is Spark's way of redistributing data across partitions so that records that belong together end up on the same machine. It is the most expensive thing Spark does: data is written to local disk, sent over the network, read back, and often sorted along the way. When a Spark job is slow, a shuffle is the usual suspect, and the number of shuffles decides how many stages your job has.

This lesson explains which operations cause a shuffle, how the map side and the reduce side cooperate, what spill and shuffle files are, and how to find all of this in the Spark UI. You should know Spark Architecture and the difference between narrow and wide transformations from Lazy Evaluation, Transformations and Actions.

What triggers a shuffle

A narrow transformation, such as map, filter or select, computes each output partition from exactly one input partition. No data moves, and Spark can pipeline several narrow steps together. A wide transformation needs data from many input partitions to build one output partition. Typical triggers:

  • Aggregations by key: groupBy().agg(), reduceByKey, groupByKey, distinct, countByKey.
  • Joins where Spark cannot broadcast a small side (see Joins).
  • Sorting: orderBy, sortBy, which needs range partitioning.
  • Repartitioning: repartition, repartitionByRange (while coalesce normally avoids a shuffle; see Partitions and Partitioning).
  • Window functions with partitionBy, which must gather each group together.

Stages and the DAG

Spark cuts a job into stages at every shuffle boundary. Inside a stage, narrow operations are pipelined: each task takes one partition through the whole chain without writing intermediate results. A new stage begins wherever the plan needs the output of a shuffle.

flowchart LR
    R["Read file"] --> M["map: word to (word, 1)"]
    M --> F["filter: drop short words"]
    F --> X["Shuffle (exchange)"]
    X --> S["reduceByKey: sum counts"]
    S --> W["Write result"]
    subgraph st1["Stage 1: shuffle map stage"]
        R
        M
        F
    end
    subgraph st2["Stage 2: result stage"]
        S
        W
    end

The shuffle splits the word count into two stages; tasks of stage 2 start only after stage 1 has written its shuffle output.

Each stage runs one task per partition. The first stage is a shuffle map stage: its tasks end by writing shuffle output. The last stage is a result stage and sends results to the driver or to storage. All tasks of a stage can run in parallel, but a stage cannot start until the stages it depends on are complete, and that barrier is part of why shuffles hurt.

The map side and the reduce side

Take a word count. In stage 1 (the map side) each task processes its input partition, applies a map-side combine if one is available, and then writes its records into shuffle files, grouped by the destination partition. In stage 2 (the reduce side) each task is responsible for one output partition; it fetches the blocks meant for it from every map task, over the network, merges them, and runs the reduction.

Today Spark uses a sort-based shuffle. Each map task writes a single data file, sorted by destination partition, plus a small index file that says where each partition's bytes begin and end. This keeps the number of files proportional to the number of map tasks, not to map tasks times reduce tasks, as an older hash-based design did. Shuffle files live on the executors' local disks, and a separate shuffle service or the executors themselves serve them to the reduce tasks.

Two settings shape the reduce side. spark.sql.shuffle.partitions (default 200) sets the number of reduce tasks for DataFrame shuffles, and Adaptive Query Execution can merge small ones after it sees real sizes.

Spill

During a shuffle, tasks hold records in memory to sort and combine them. If a task's execution memory fills up, Spark spills its in-memory data to disk, sorted, and later merges the spilled pieces. Spill keeps the job alive instead of failing with an out-of-memory error, but it costs disk I/O and CPU. In the Spark UI you see two numbers per task: Spill (Memory), the size of the data before serialization, and Spill (Disk), the compressed size written. Large spill usually means too few shuffle partitions (each task has too much data) or too little memory per core. Raising the partition count reduces the data per task.

A shuffle stand-in

The real exchange of bytes between machines is not something we can run here. The stand-in below simulates the logical steps with plain Python: three map tasks each split their records by destination partition and write "shuffle blocks", and three reduce tasks fetch their blocks and finish the work. It uses a stable hash (CRC32), and it counts how many records cross between "tasks". It models the flow only, not the sort-based file layout or the network.

Python A simulated hash shuffle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All counts for one word end up in one reducer, which is the whole point of the shuffle. Because of the map-side combine, fewer records were fetched than were read.

Finding stage boundaries

The stand-in below does what Spark's scheduler does conceptually: it walks a chain of transformations and starts a new stage after each wide one. It is a simplified model, because a real DAG can branch and join.

Python Cutting a pipeline into stages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two wide operations produced three stages. In real Spark, a join of two datasets reads both sides in separate map stages, and a join between a big table and a broadcast small table needs no shuffle at all, so the real plan would have a different shape. Use the stand-in to build intuition, and use explain() for the truth.

Reading the Spark UI

The Spark UI (port 4040 on the driver while an application runs) shows shuffles directly:

  • Jobs tab: one job per action, with a DAG visualization. Boxes are stages, and the line between them is a shuffle.
  • Stages tab: for each stage you see Shuffle Write (bytes the map side produced) and Shuffle Read (bytes the reduce side fetched), task counts, and durations. A stage with huge shuffle read is the expensive one.
  • Task metrics: the median, max and percentile durations of tasks. If the maximum is far larger than the median, one partition holds most of the data (skew). Spill (Memory) and Spill (Disk) columns appear when tasks spilled.
  • SQL tab: the physical plan with Exchange nodes (the shuffles) and row counts and times per operator.

For more on diagnosing these, see Testing, Debugging and the Spark UI.

Reducing shuffle cost

  • Filter and project before the shuffle so fewer bytes move.
  • Prefer map-side combines: reduceByKey or DataFrame aggregations over groupByKey.
  • Broadcast small tables in joins instead of shuffling both sides.
  • Avoid unnecessary distinct, orderBy and repartition calls. A global sort is one of the heaviest shuffles.
  • Tune the partition count so tasks are neither tiny nor memory-starved.
  • Reuse partitioned or bucketed data so Spark can skip repeated shuffles.
  • Cache wisely: results of a costly shuffle that are used twice should be stored (see Caching and Persistence). Spark also reuses shuffle files from a completed stage within the same application, which shows as "skipped" stages in the UI.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these operations normally triggers a shuffle?

    Aggregating by key needs all records for a key in one partition, so data is redistributed. filter, map and select are narrow.
  2. 2. Where does Spark cut a job into stages?

    Narrow operations are pipelined into one stage; a wide operation ends a stage and its output is shuffled to the next one.
  3. 3. What does the reduce side of a shuffle do?

    Each reduce task owns one output partition and pulls the matching blocks from all map tasks over the network.
  4. 4. What does a large Spill (Disk) value in a stage usually suggest?

    When memory fills up during a shuffle, Spark spills sorted data to disk. More partitions or more memory per core usually helps.

Summary

  • A shuffle redistributes data so related records meet; it writes to local disk, uses the network and creates a barrier between stages.
  • Aggregations, most joins, sorts, window functions and repartition trigger shuffles; narrow transformations do not.
  • Map tasks write sorted shuffle files with an index; reduce tasks fetch their blocks from all map tasks.
  • Spill is the safety valve when execution memory runs out; it costs disk I/O.
  • The Spark UI shows Shuffle Read/Write, spill, task skew and Exchange nodes; reduce shuffles by filtering early, combining locally and broadcasting small tables.

Next: avoid recomputing expensive results with Caching and Persistence.