Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 48 ยท Performance and Production

Testing, Debugging and the Spark UI

Learn to unit test Spark transformations, run tests in local mode, read the Spark UI, find skew and fix common errors such as OOM and serialization failures.

Spark jobs fail in unusual ways. A transformation can be correct on ten rows and wrong on ten billion, a variable can change on your laptop but not on the cluster, and a job can "work" while one straggler task holds up everything. Good habits make these problems cheap: test the logic in small pieces, run it in local mode before the cluster, and learn to read the Spark UI, which shows exactly where time and memory go.

The real Spark UI and a real SparkSession cannot run on this site, so the Spark code below is read-only. The runnable examples are stand-ins in plain Python: a test harness that splits data into partitions, a table of task metrics that flags skew, and a small demonstration of the serialization problem. They show the ideas honestly without needing a cluster. For background on tasks and stages see Spark Architecture.

Separate logic from plumbing

The easiest code to test is a plain function. Keep the business logic (cleaning a value, parsing a line, computing a score) in ordinary functions that take and return ordinary values, and let a thin layer apply them with map, withColumn or a UDF. Then most tests need no Spark at all.

Python
# logic.py: no Spark import, trivial to test
def normalize_country(raw):
    return (raw or "").strip().upper() or "UNKNOWN"

# job.py: the thin Spark layer
from pyspark.sql import functions as F
from logic import normalize_country

def clean(df):
    normalize = F.udf(normalize_country)
    return df.withColumn("country", normalize("country"))

For DataFrame code, write functions that take a DataFrame and return a DataFrame. Each is a small, testable unit, and they chain with df.transform(f). When a test does need Spark, use local mode: the driver and executors share one JVM on your machine, started in about a second with a tiny shuffle setting so tests stay fast.

Python
# conftest.py
import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")
def spark():
    session = (SparkSession.builder
               .master("local[2]")
               .appName("tests")
               .config("spark.sql.shuffle.partitions", "2")   # default 200 is slow for tiny data
               .config("spark.ui.enabled", "false")
               .getOrCreate())
    yield session
    session.stop()

# test_clean.py
from pyspark.testing import assertDataFrameEqual     # Spark 3.5 and later

def test_clean_uppercases_country(spark):
    df = spark.createDataFrame([(" de ",), (None,)], ["country"])
    expected = spark.createDataFrame([("DE",), ("UNKNOWN",)], ["country"])
    assertDataFrameEqual(clean(df), expected)

Create one session per test run (scope="session"), because starting a JVM for every test is what makes Spark test suites slow. Keep test data tiny, set spark.sql.shuffle.partitions low, and compare DataFrames with an order-insensitive helper rather than collect() plus a list equality, since row order across partitions is not guaranteed. Older projects use libraries such as chispa for the same job.

The stand-in below tests plain-function logic with Python's unittest, the same way you would test the first file above.

Python Unit testing the logic without Spark

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Test across partitions

Some bugs only appear when data is split. A function you pass to reduce or reduceByKey must be associative (grouping does not matter) and commutative (order does not matter), because Spark combines partial results from partitions in any order. If it is not, the answer changes with the number of partitions: the test passes on your laptop with one partition and fails in production with 2,000.

So test reduce-style logic with several partitionings of the same data.

Python A test harness that splits data into partitions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A property check like this is a cheap safety net: run the function with 1, 2 and 3 partitions and assert the results are equal. For the average, the right pattern is to carry (sum, count) and divide at the end, as shown in Key-Value (Pair) RDDs.

Reading the Spark UI

Every running application serves a web UI on the driver at port 4040 (4041, 4042 and so on if the port is busy). For finished applications, turn on event logging (spark.eventLog.enabled=true, spark.eventLog.dir=...) and start the History Server, usually on port 18080, which replays the same pages later.

flowchart TD
    A["Jobs tab:<br/>one row per action"] --> B["Stages tab:<br/>pieces split at shuffles"]
    B --> C["Task table:<br/>time, input, shuffle, spill, GC"]
    A --> D["SQL / DataFrame tab:<br/>physical plan per query"]
    E["Executors tab:<br/>memory, GC, failed tasks"] --> C
    F["Storage tab:<br/>cached data and size"]

Start at the Jobs tab, drill down to stages and tasks, and use the SQL, Executors and Storage tabs for context.

What each tab tells you:

  • Jobs: every action creates a job. Failed jobs show up red with the error.
  • Stages: a stage is a set of tasks without a shuffle between them. Open a slow stage and look at the summary metrics table with min, 25th percentile, median, 75th percentile and max for task duration, input size, shuffle read and spill.
  • SQL / DataFrame: the plan of each query with row counts and time per operator. See Catalyst Optimizer and Query Plans for how to read the plan.
  • Executors: memory use, GC time (more than about 10% of task time is a warning), failed tasks and logs per executor.
  • Storage: which RDDs and DataFrames are cached, how much memory or disk they take and what fraction is cached.

The most valuable signal is the gap between median and max in a stage. When most tasks finish in two seconds and one takes four minutes, the stage is waiting on a straggler. That usually means data skew: one key or one partition holds far more rows than the rest.

Python Flagging skew from task metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Skew, spill and GC are the three numbers to read first. The fixes (salting, adaptive query execution, broadcast joins) are covered in Performance Tuning.

Logs and quick debugging tools

  • Driver logs show planning and the final error. Executor logs hold the output of print and logging calls made inside tasks, and live under the Executors tab, in YARN's yarn logs -applicationId <id> or in kubectl logs <pod>.
  • spark.sparkContext.setLogLevel("WARN") quiets noisy INFO output in development.
  • df.explain() prints the plan, df.printSchema() checks types, df.rdd.getNumPartitions() shows parallelism and rdd.toDebugString() shows lineage.
  • Debug on a sample: df.limit(1000) or df.sample(0.01) lets you iterate quickly, but remember that limit plus a shuffle can still scan a lot of data.
  • Make one change at a time and compare stage times in the UI instead of guessing.

Common errors

Symptom Likely cause Typical fix
OutOfMemoryError: Java heap space on the driver collect(), toPandas() or a huge broadcast Aggregate first, write to storage, raise --driver-memory
Executor OOM or Container killed ... exceeding memory limits Skewed or oversized partitions; too little memory overhead More partitions, fix skew, raise spark.executor.memoryOverhead
Task not serializable (Scala/Java) or PicklingError (Python) The closure captured a connection, lock or non-serializable object Create the resource inside mapPartitions or foreachPartition
AnalysisException: cannot resolve 'x' Misspelled column, or a column lost after a select or join Check df.columns and printSchema()
FetchFailedException An executor holding shuffle output died Fix the memory problem that killed it; enable external shuffle service
Job aborted due to stage failure ... failed 4 times A task kept failing (spark.task.maxFailures defaults to 4) Read the first error in the task's log, not the final summary
Python worker exited unexpectedly Native crash or OOM in a Python worker Raise memory overhead, reduce Python UDF memory

Always search for the root cause, which is usually the first exception in the executor log. The long stack trace the driver prints at the end often only says that a stage was aborted.

The serialization error deserves its own demonstration, because it is the one beginners meet first. The stand-in below uses Python's pickle (PySpark uses cloudpickle, which behaves the same way for objects like these).

Python Why capturing a connection fails and what to do instead

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. The unit test harness and the reduce checker are real techniques you can use as they are, but they run plain Python, not Spark. The task table imitates what the UI's stage page shows. The real UI adds timelines, DAG visualisations and per-operator SQL metrics, which only appear when a real application runs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should a function passed to reduceByKey be associative and commutative?

    The result must not depend on how data is split or in which order partials are merged. Otherwise the output changes with the number of partitions.
  2. 2. In the Stages tab, one task takes 240 seconds while the median is 2 seconds. What is the most likely cause?

    A huge gap between median and maximum task time, usually with larger shuffle read and spill for that task, points to skew.
  3. 3. Which setup keeps a Spark test suite fast?

    Starting a session is the expensive part. Reuse one, keep data small and lower the shuffle partition count.
  4. 4. A job fails with a Task not serializable error. What is the standard fix?

    The closure captured an object that cannot be serialized. Creating it inside the task means it never has to be sent.

Summary

  • Keep business logic in plain functions and DataFrame-in, DataFrame-out functions, so most tests need little or no Spark.
  • Test with one reused local-mode session, tiny data, a small shuffle partition count and order-insensitive comparisons.
  • Check that reduce functions give the same result for different partition counts; if not, they are not associative or commutative.
  • The Spark UI (port 4040, or the History Server) shows jobs, stages, SQL plans, executors and cached data; compare median and max task times to spot skew, and watch spill and GC.
  • Read the first exception in the executor logs, and match it to the usual suspects: driver or executor OOM, serialization, missing columns, lost shuffle files.

Next: put your skills to work in a hands-on build with Project: Web Server Log Analysis.