Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 48 ยท Performance and Production

Performance Tuning

Make Spark jobs faster: find data skew and fix it with salting, use broadcast joins and AQE, size shuffle partitions, tune memory and Kryo, avoid small files.

A Spark job that works is not the same as a Spark job that works well. The difference can be 10 minutes against 3 hours, or a job that finishes against one that runs out of memory at 2 a.m. The good news is that most slowness has a small number of causes, and you can learn to recognize them in the Spark UI.

This lesson is a tour of those causes in rough order of how often they bite: too much shuffling, skewed data, badly sized partitions, wrong join strategy, memory pressure, slow serialization and small files. It builds on Shuffles and Stages, Partitions and Partitioning and Catalyst Optimizer and Query Plans. Spark itself cannot run here, so the settings are read-only code and the TryIts are stand-ins that show the ideas with plain Python.

Start with measurement

Change one thing at a time and look at evidence first. The places to look:

  • Spark UI, Stages tab: task duration summary. If the maximum is far above the median, you have skew or a slow node. Check shuffle read size and spill columns.
  • SQL tab: the physical plan with real row counts and times per operator.
  • df.explain("formatted"): which join strategy was chosen, whether filters were pushed down.

The cheapest wins usually come before any tuning knob:

  1. Filter and select early, so less data is read and shuffled. Parquet and ORC support column pruning and predicate pushdown.
  2. Prefer built-in functions over Python UDFs (User-Defined Functions and Pandas UDFs).
  3. Use DataFrames and SQL rather than raw RDDs, so Catalyst and Tungsten can optimize.
  4. Avoid actions you do not need, such as an extra count() for logging.

Data skew

Hash partitioning sends all rows with the same key to the same partition. If one key is far more common than the others (country "US", user "anonymous", NULL), one task receives a huge share of the data while the rest finish quickly and sit idle. The stage ends only when the last task does.

Signs of skew: one or a few tasks take much longer than the median, have a far larger shuffle read, spill to disk, or fail with out-of-memory errors while others use little.

Salting

Salting spreads a hot key over several partitions by adding a random number to the key. For an aggregation, you aggregate twice:

  1. Group by (key, salt): the hot key's rows are now split into N groups handled by different tasks.
  2. Group by key again to combine the N partial results.
flowchart LR
    A["Rows with hot key<br/>'US' x 1,000,000"] --> B["Add salt 0..3<br/>US_0 US_1 US_2 US_3"]
    B --> C["Stage 1: partial sums<br/>4 tasks share the work"]
    C --> D["Stage 2: sum partials<br/>by original key"]

Salting turns one huge group into several smaller ones, then combines the partial results.

For a skewed join, you add a random salt (0 to N-1) to the big, skewed side and replicate each row of the other side N times, once per salt value. The join key becomes (key, salt).

Python
from pyspark.sql import functions as F

N = 8
big_salted = big.withColumn("salt", (F.rand() * N).cast("int"))
small_salted = small.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))

joined = big_salted.join(small_salted, on=["key", "salt"]).drop("salt")

The stand-in measures the load per partition before and after salting, using a deterministic hash and round-robin salts.

Python Skew and salting: rows per partition

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Adaptive Query Execution fixes much of this automatically

Since Spark 3.0 Adaptive Query Execution (AQE) re-optimizes the plan at stage boundaries using real statistics gathered while the job runs. It is enabled by default from Spark 3.2 (spark.sql.adaptive.enabled). It does three things:

  • Coalesces small shuffle partitions after a shuffle, so you do not pay for hundreds of tiny tasks.
  • Switches join strategy at runtime: if one side turns out to be small after filtering, it becomes a broadcast join.
  • Splits skewed partitions in sort-merge joins (spark.sql.adaptive.skewJoin.enabled). A partition counts as skewed when it is larger than skewedPartitionFactor (default 5) times the median partition size and larger than a byte threshold (default 256 MB).

AQE handles skewed joins well. It does not rescue every skewed aggregation, and a hot key that is a single huge group still needs salting or a different approach.

Join strategies

How two tables are joined matters more than almost anything else:

Strategy When Spark uses it Cost
Broadcast hash join One side is small (below spark.sql.autoBroadcastJoinThreshold, default 10 MB) or you give a hint No shuffle of the big side: the small table is sent to every executor
Shuffle hash join Medium sizes, when hash join is preferred Shuffles both sides, builds a hash map per partition
Sort-merge join Default for two large tables Shuffles and sorts both sides
Broadcast nested loop / cartesian No equality condition Very expensive; avoid

Force a broadcast when you know a table is small but statistics are missing:

Python
from pyspark.sql.functions import broadcast

result = orders.join(broadcast(countries), "country_code")

Broadcasting a table that is too large makes the driver and executors run out of memory, so keep it to tens or a few hundred megabytes. Joins are explained in Joins. The next stand-in counts how many rows travel over the network for each strategy.

Python Rows moved: shuffle join versus broadcast join

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Shuffle partitions

After a shuffle (a join, groupBy, distinct, repartition), the data lands in spark.sql.shuffle.partitions partitions, 200 by default. That number does not depend on your data:

  • Too few: partitions are huge, tasks spill to disk or run out of memory, and cores sit idle.
  • Too many: thousands of tiny tasks where scheduling overhead exceeds the work.

A common target is 100 to 200 MB of shuffle data per partition and at least two to three times as many partitions as total cores. With AQE on, set the initial number a bit high and let spark.sql.adaptive.coalescePartitions.enabled merge small ones. Use repartition(n) to increase and coalesce(n) to reduce without a full shuffle.

The stand-in imitates AQE's coalescing: it merges neighboring partitions until each one approaches a target size.

Python Coalescing small shuffle partitions toward a target size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Memory and executors

Executor memory is split into regions. The unified region (spark.memory.fraction, default 0.6 of the heap minus a reserved 300 MB) is shared between execution (shuffles, joins, sorts, aggregations) and storage (cached data). Execution can evict cache when it needs room, but not the other way round. The rest is user memory for your own objects and Spark's internal metadata, and spark.executor.memoryOverhead adds off-heap space (default 10 percent of executor memory, at least 384 MB), which PySpark needs for its Python workers.

Practical guidance:

  • Out of memory (OOM) in an executor: look for skew first, then increase spark.executor.memory, memoryOverhead or the partition count. YARN or Kubernetes killing a container for exceeding memory points at overhead.
  • Garbage collection time above a few percent of task time suggests too many small objects or too little heap. Prefer DataFrames (stored in compact binary form by Tungsten) over RDDs of Java objects.
  • Executor size: very large heaps (over about 64 GB) mean long GC pauses; many tiny executors waste overhead. Something like 4 to 5 cores and 16 to 32 GB per executor is a common start.
  • Driver OOM is usually collect() of too much data or a broadcast that is too large: raise spark.driver.memory or stop pulling data to the driver.

Serialization and Kryo

Data crossing the network or written to disk in serialized form must be serialized. The default for RDDs of JVM objects is Java serialization, which is slow and bulky. Kryo is faster and produces smaller output:

Python
spark = (SparkSession.builder
         .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
         .config("spark.kryo.registrationRequired", "false")
         .getOrCreate())

Kryo mostly matters for RDD-based Scala and Java code and for cached data in serialized storage levels. DataFrames use their own binary format (Tungsten) and Kryo is rarely the bottleneck there. In PySpark the data between Python and the JVM is pickled or moved through Arrow; use Arrow-based pandas UDFs and spark.sql.execution.arrow.pyspark.enabled=true for toPandas() to cut that cost.

The small-files problem

Writing many small files makes later jobs slow. Each file costs a task, a file open and metadata lookups on the file system or object store. Typical causes: too many output partitions, a high-cardinality partitionBy, or a streaming job with a short trigger.

Fixes:

  • coalesce(n) or repartition(n) before writing, so each output file reaches roughly 128 to 512 MB.
  • repartition("date") before partitionBy("date"), so each date folder gets a few files instead of one per task.
  • Compact existing data periodically (Delta Lake OPTIMIZE, or a scheduled job that reads and rewrites a partition).
  • Cap rows per file with spark.sql.files.maxRecordsPerFile.

The opposite problem exists too: a few gigantic non-splittable files (one big .gz CSV) cannot be read in parallel at all. Prefer splittable columnar formats such as Parquet.

Real versus stand-in. The settings above are real Spark configuration. The TryIts only model the idea: a hash function and counters for skew, arithmetic for join cost, and a greedy merge for AQE's coalescing. Real AQE uses runtime shuffle statistics per partition, and real join costs depend on network, compression and executor placement.

A tuning checklist

  1. Read the Spark UI: find the slowest stage and the skew between tasks.
  2. Reduce the data: filter, select, prune partitions, use Parquet.
  3. Replace UDFs with built-in functions or pandas UDFs.
  4. Check the join strategy; broadcast small tables.
  5. Enable AQE, and size spark.sql.shuffle.partitions for your data.
  6. Cache only what is reused (Caching and Persistence).
  7. Fix skew with AQE's skew handling, salting or a pre-aggregation.
  8. Only then touch memory and executor sizes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. One task in a stage runs for 20 minutes while the others take 30 seconds. What is the most likely cause?

    A stage ends when its last task ends. A hot key sends most rows to one partition, which makes that task the straggler.
  2. 2. How does salting fix a skewed aggregation?

    Grouping by (key, salt) spreads the hot key over several tasks. A second grouping by the original key adds the partial results.
  3. 3. When is a broadcast hash join the best choice?

    Broadcasting the small table avoids shuffling the big one. It fails if the 'small' table does not fit in executor and driver memory.
  4. 4. What does Adaptive Query Execution do with many tiny shuffle partitions?

    AQE reads real shuffle statistics and merges neighboring small partitions toward a target size, cutting task overhead.
  5. 5. Which change most directly reduces the number of small output files?

    Each output task writes its own file, so reducing the number of partitions at write time makes fewer, larger files.

Summary

  • Measure first: the Spark UI shows skew, spill and the slowest stage; explain shows join strategies.
  • Reduce data early, prefer built-ins and DataFrames over UDFs and RDDs.
  • Skew shows up as a few very slow tasks; fix it with AQE skew handling, salting or pre-aggregation.
  • Broadcast small tables; the default sort-merge join shuffles and sorts both sides.
  • Size spark.sql.shuffle.partitions for your data and let AQE coalesce small partitions.
  • Know the memory regions, use Kryo for RDD-heavy code and Arrow for Python transfers, and avoid small files with repartition/coalesce and compaction.

Next: run your tuned jobs on real clusters in Deploying Spark: YARN, Kubernetes, Standalone and Cloud.