Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 48 ยท Under the Hood

Partitions and Partitioning

Learn how Spark splits data into partitions, how to choose a partition count, repartition versus coalesce, hash and range partitioners, and bucketing.

A partition is a slice of a distributed dataset that lives on one executor and is processed by one task. Spark's parallelism comes from partitions: 8 partitions can keep 8 cores busy, and one giant partition keeps a single core busy while the rest of the cluster idles. Partitioning is therefore one of the first things to check when a job is slow, uses too much memory, or writes thousands of tiny files.

This lesson explains how many partitions you get, how to change them with repartition and coalesce, how keys are routed to partitions, and how storage-level partitioning (partitionBy and bucketing) differs from in-memory partitioning. It assumes you know RDD Basics and DataFrames: Creating and Inspecting.

Where partition counts come from

You rarely pick the count directly at first; Spark derives it.

  • Reading files. Spark splits input into chunks of up to spark.sql.files.maxPartitionBytes (128 MB by default), so a 10 GB Parquet dataset becomes roughly 80 or more partitions. Many tiny files each need at least a partial partition, which is the small-files problem.
  • After a shuffle. Joins and aggregations on DataFrames produce spark.sql.shuffle.partitions partitions, 200 by default. That is often too many for small data and too few for huge data. With Adaptive Query Execution (AQE) enabled, which is the default in current Spark versions, Spark can merge small post-shuffle partitions automatically.
  • parallelize. The RDD API uses spark.default.parallelism (usually the total number of cores) unless you pass a number.
Python
df = spark.read.parquet("events/")
df.rdd.getNumPartitions()          # for example 84

spark.conf.get("spark.sql.shuffle.partitions")   # '200'

A common rule of thumb is 2 to 4 tasks per available core, and partitions of roughly 100 to 200 MB. Far fewer partitions leaves cores idle and risks out-of-memory errors; far more adds scheduling overhead for each tiny task.

repartition and coalesce

Two methods change the partition count, and the difference is whether a shuffle happens.

Python
more = df.repartition(200)               # full shuffle, data spread evenly
by_key = df.repartition(50, "user_id")   # hash partition by column(s)
ranged = df.repartitionByRange(50, "ts") # range partition by column

fewer = df.coalesce(10)                  # merge existing partitions, no full shuffle

repartition(n) can increase or decrease the number of partitions. It performs a full shuffle, so every record may move across the network, but the result is evenly sized. With column arguments, rows with equal key values land in the same partition, which helps before a join or a per-key write.

coalesce(n) can only decrease the count. It glues neighbouring partitions together without a full shuffle, which makes it cheap, but the sizes may end up uneven because it does not rebalance. Use it to reduce the number of output files after a heavy filter. If you ask a DataFrame to coalesce to more partitions than it has, nothing changes.

flowchart LR
    subgraph before["Before: 4 partitions"]
        P1["P1"]
        P2["P2"]
        P3["P3"]
        P4["P4"]
    end
    P1 --> C1["coalesce(2): P1 + P2"]
    P2 --> C1
    P3 --> C2["coalesce(2): P3 + P4"]
    P4 --> C2

coalesce merges whole partitions in place, so no record has to travel to an arbitrary partition.

One trap: an aggressive coalesce(1) before a big computation also reduces the parallelism of the whole stage before it, because Spark can fuse the coalesce into the earlier work. If you need one output file but want parallel work first, use repartition(1), which puts a shuffle boundary in between.

Hash and range partitioners

How does Spark decide which partition a record belongs to? A partitioner maps a key to a partition number.

  • Hash partitioning computes hash(key) mod numPartitions. Equal keys always meet in the same partition, and distribution is even unless one key is very frequent (skew). Shuffles for groupBy, reduceByKey and joins use it.
  • Range partitioning samples the data to find boundaries, then puts keys in sorted ranges: partition 0 gets the smallest keys, partition 1 the next, and so on. It is what orderBy and repartitionByRange use, because concatenating the sorted partitions gives a globally sorted result.

For RDDs, a pair RDD can carry a known partitioner. Operations like mapValues preserve it, while map drops it, which is why Key-Value (Pair) RDDs recommended mapValues. If both sides of a join already share the same partitioner, Spark can skip a shuffle.

A stand-in for partitioners

The stand-in below spreads 12 records across 3 partitions with four strategies and prints the sizes. It uses a small stable hash, because Python's built-in hash for strings changes from run to run. Real Spark uses a different hash function (Murmur3 for DataFrames), so the exact placement differs, but the behavior is the same: hash spreads by key, range keeps order, round-robin is even regardless of keys, and coalesce just merges.

Python Four ways to distribute records

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that hash partitioning puts all u1 records together but its sizes are uneven (2, 4 and 6), because u1 is frequent. Round-robin gives equal sizes (4, 4, 4) but spreads each key everywhere, so it is no help for per-key work. Range partitioning keeps keys in sorted groups, yet its sizes are uneven too (7, 3, 2), because u1 is frequent here as well. coalesce(3) just merges neighbouring partitions, so its result depends on which ones were next to each other. This is the trade-off behind every partitioning choice.

Partitioned tables on disk

Do not confuse in-memory partitions with storage partitioning. partitionBy on the writer creates one directory per distinct value, so queries that filter on that column can skip whole folders (partition pruning):

Python
(df.write
   .partitionBy("country", "year")
   .mode("overwrite")
   .parquet("warehouse/events"))
# warehouse/events/country=DE/year=2025/part-00000-....parquet
# warehouse/events/country=US/year=2025/part-00000-....parquet

Pick low-cardinality columns that queries filter on. Partitioning by something like user_id creates millions of tiny directories and files, which is a performance disaster. Because each task writes a file per partition value it holds, also repartition by the same column before writing to avoid a flood of small files: df.repartition("country", "year").write.partitionBy(...). More on writing in Reading and Writing Data.

Bucketing is a different layout: rows are hashed on a column into a fixed number of buckets that are stored as separate files, and the layout is recorded in the table metadata.

Python
(df.write
   .bucketBy(16, "user_id")
   .sortBy("user_id")
   .mode("overwrite")
   .saveAsTable("events_bucketed"))

If two tables are bucketed on the join key with the same bucket count, Spark can join them without a shuffle, since matching keys already sit in matching buckets. Bucketing only works with saveAsTable (a catalog table), and both tables must be written consistently, so it pays off mainly for large tables that are joined repeatedly.

Choosing a strategy

  • Too few partitions: idle cores, big tasks, memory errors. Use repartition upward.
  • Too many tiny partitions: scheduling overhead and small files. Use coalesce, or let AQE merge post-shuffle partitions.
  • Before a join or per-key write: repartition("key").
  • Filtering on a column most of the time: partition the table on disk by it, if it has few values.
  • Frequent joins of two big tables on one key: consider bucketing.
  • Skewed keys: see Performance Tuning.

Next to each of these lies a shuffle, which is the expensive part. We take a closer look in Shuffles and Stages.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the default value of spark.sql.shuffle.partitions?

    DataFrame shuffles produce 200 partitions by default, which is often too many for small data. AQE can merge small ones.
  2. 2. How does coalesce(n) differ from repartition(n)?

    coalesce merges neighbouring partitions without redistributing records, so sizes can be uneven. repartition does a full shuffle and balances the data.
  3. 3. Which partitioner puts equal keys together and is used by groupBy and joins?

    Hash partitioning sends a key to hash(key) mod n, so equal keys always meet. Range partitioning is for sorted output.
  4. 4. You partition a table on disk by user_id, which has millions of distinct values. What is likely to happen?

    partitionBy creates a directory per value, so high-cardinality columns produce a small-files disaster. Choose low-cardinality columns.
  5. 5. What can bucketing give you that partitionBy cannot?

    Bucketed tables hash rows on the key into a fixed number of files, so matching buckets can be joined directly.

Summary

  • A partition is a slice of data processed by one task; the count controls parallelism, task size and output file count.
  • File reads follow maxPartitionBytes, DataFrame shuffles default to 200 partitions, and AQE can merge small ones.
  • repartition shuffles to an exact, balanced count (or by key); coalesce cheaply merges partitions down.
  • Hash partitioning groups equal keys, range partitioning keeps sorted ranges, and both can suffer from skew or uneven sizes.
  • partitionBy and bucketing are on-disk layouts that enable partition pruning and shuffle-free joins; choose columns carefully.

Next: see what a shuffle really does in Shuffles and Stages.