Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 48 ยท Under the Hood

Caching and Persistence

Learn when to cache a Spark DataFrame or RDD, how cache differs from persist, what storage levels mean, how to unpersist, and when checkpointing is better.

Spark is lazy: every action re-runs the whole chain of transformations that leads to the data, starting from the source. If two actions use the same expensive intermediate result, Spark computes it twice. Caching (also called persisting) tells Spark to keep a computed dataset around, in memory and optionally on disk, so later actions can reuse it.

Caching is one of the easiest speedups and one of the easiest to misuse. This lesson explains how it works, which storage level to choose, how to know whether it helped, and how checkpointing differs. It builds on Lazy Evaluation, Transformations and Actions and Shuffles and Stages.

The recomputation problem

Imagine a pipeline that reads a big file, parses it, filters it and joins it. You then call count() on the result and later write it out. Without caching, the second action starts again from the file: it parses, filters and joins all over again.

Python
cleaned = (spark.read.json("events/")
           .filter("status = 'ok'")
           .withColumn("day", F.to_date("ts")))

cleaned.count()                          # job 1: read, parse, filter
cleaned.groupBy("day").count().show()    # job 2: read, parse, filter again

Adding one line avoids the repeat:

Python
cleaned.cache()
cleaned.count()                          # job 1: computes and stores the partitions
cleaned.groupBy("day").count().show()    # job 2: reads the stored partitions

cache() is lazy too. It only marks the dataset; the partitions are stored the first time an action computes them. That is why people often follow it with a cheap action such as count() to fill the cache deliberately before the real work. Note that count() on some sources can be answered without reading every column, and then the cache is only partly filled; a full action such as writing to the noop format (df.write.format("noop").mode("overwrite").save()) materializes everything.

cache versus persist

cache() is shorthand for persist() with a default storage level. persist(level) lets you choose.

Python
from pyspark import StorageLevel

df.cache()                                   # DataFrame default: MEMORY_AND_DISK
df.persist(StorageLevel.DISK_ONLY)           # explicit level
rdd.cache()                                  # RDD default: MEMORY_ONLY

The defaults differ: RDD.cache() uses MEMORY_ONLY, while DataFrame.cache() uses MEMORY_AND_DISK. The main levels are:

Level Meaning
MEMORY_ONLY Keep partitions in memory. A partition that does not fit is not stored and is recomputed when needed.
MEMORY_AND_DISK Keep in memory; partitions that do not fit are written to local disk.
DISK_ONLY Keep only on local disk.
MEMORY_ONLY_2, MEMORY_AND_DISK_2 Same, with each partition replicated on two executors for faster recovery.
OFF_HEAP Store outside the JVM heap, which needs off-heap memory configured.

In Scala and Java there are also serialized variants such as MEMORY_ONLY_SER, which store data as byte arrays: smaller, but they cost CPU to read. In PySpark the data on the Python side is always serialized, so you will mostly choose among the memory, disk and replication options above.

Stored partitions are evicted in least-recently-used order when memory is needed for something else. Cached data is never required for correctness: if a cached partition is lost or evicted, Spark simply recomputes it from its lineage (see Fault Tolerance and Lineage).

When to cache, and when not to

Cache when all of these are true:

  • The dataset is used by two or more actions (or several branches of one query).
  • Recomputing it is expensive: heavy shuffles, many joins, slow sources.
  • It fits in the memory (and disk) you are willing to spend on it.

A classic case is iterative work, such as a machine-learning training loop that reads the same training data over and over (see MLlib Overview and Pipelines).

Do not cache:

  • A dataset used once. Storing it costs time and memory for no benefit.
  • Raw data that is cheap to re-read, for example a small Parquet file. Spark's column pruning and pushdown may even make reading the source faster than reading a cache of everything.
  • Huge datasets that will mostly spill or be evicted. Caching too much can slow the whole application, since cached blocks compete with execution memory.

Cache late in the pipeline, after filters and column selection, so you store the smallest useful version. Cache only what you reuse. Also remember that cached DataFrames are fixed snapshots of a query plan: if the underlying files change after caching, you may keep seeing old data until you unpersist and reload.

unpersist and the Storage tab

Release memory when you are done:

Python
cleaned.unpersist()                  # non-blocking by default
cleaned.unpersist(blocking=True)     # wait until blocks are removed

Spark also removes cached data when the dataset object is garbage-collected or the application ends, but long-running applications should unpersist explicitly. In the Spark UI the Storage tab lists every cached dataset with its storage level, number of cached partitions, fraction cached, and size in memory and on disk. Check it to confirm the cache is really filled and that its size is what you expected. The SQL tab also shows an InMemoryTableScan operator in plans that read from a cache.

A stand-in with a memoized pipeline

You cannot watch Spark's block manager here, but the idea is the same as memoization. The stand-in builds a lazy pipeline whose steps count how often they are computed. Without cache, every action recomputes the chain; with cache, only the first does. It models the behavior, not the storage or eviction.

Python Lazy pipeline with and without cache

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without the cache, both actions pay for both steps: 2 steps x 1,000 elements x 2 actions. With it, the second action costs nothing. After unpersist, the next action computes again, which is exactly how Spark behaves.

Checkpointing

Checkpointing also saves a computed dataset, but for a different purpose. It writes the data to reliable storage (a directory you set with spark.sparkContext.setCheckpointDir(...), typically on HDFS or cloud storage) and truncates the lineage: after a checkpoint, Spark no longer remembers how the data was built, only where it is stored.

Python
spark.sparkContext.setCheckpointDir("hdfs:///tmp/checkpoints")

stable = df.checkpoint()          # DataFrame: eager by default, runs a job now
rdd.checkpoint()                  # RDD: lazy, written when the next action runs

Use checkpointing when a lineage becomes very long, for example in an iterative algorithm that updates a dataset 100 times: the plan grows with each step and slows planning or risks a stack overflow. Checkpointing cuts it. It is also useful when recomputation is so costly that you want a durable copy that survives executor loss. Unlike a cache, checkpoint files are not deleted automatically when the application ends unless you configure cleanup.

Cache / persist Checkpoint
Stored where Executor memory and local disk Reliable shared storage
Lineage Kept (can recompute if lost) Truncated
Survives the application No Yes, if files are kept
Cost Cheap A full write of the data
Typical use Reuse in several actions Break very long lineages, durable save

A related option is localCheckpoint(), which truncates lineage but stores data on executors' local storage, so it is faster but not fault tolerant.

The stand-in below mimics the effect on a lineage, showing how a long chain collapses once a checkpoint is taken.

Python Lineage before and after a checkpoint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. The stand-ins show how reuse and lineage truncation behave. Real caching stores serialized or deserialized blocks across executors under a memory manager that evicts blocks, and real checkpoints are files written by tasks to the checkpoint directory.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does cache() actually store the data?

    cache() is lazy: it marks the dataset. The partitions are stored as the first action computes them.
  2. 2. What happens to a MEMORY_ONLY partition that does not fit in memory?

    MEMORY_ONLY drops partitions that do not fit; Spark recomputes them from lineage. MEMORY_AND_DISK would spill them to disk instead.
  3. 3. Which situation is the best candidate for caching?

    Caching pays off when an expensive result is reused. A single-use or oversized dataset only wastes resources.
  4. 4. What is the main difference between a checkpoint and a cache?

    Checkpoints persist data durably and forget how it was built. A cache is a performance aid that can always be recomputed from lineage.

Summary

  • Without caching, every action recomputes the chain from the source; cache() and persist() keep a dataset for reuse.
  • Caching is lazy, filled by the first action; DataFrame.cache() defaults to MEMORY_AND_DISK, RDD.cache() to MEMORY_ONLY.
  • Cache expensive, reused, reasonably small datasets, late in the pipeline, and call unpersist() when done; check the Storage tab.
  • Evicted or lost cached blocks are simply recomputed from lineage.
  • Checkpointing writes data to reliable storage and truncates lineage, which helps long iterative jobs.

Next: see how Spark recovers when work is lost in Fault Tolerance and Lineage.