Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 48 ยท RDDs

RDD Basics

Learn what a Resilient Distributed Dataset is, how to create RDDs with parallelize and textFile, why they are immutable, and how partitions and lineage work.

The Resilient Distributed Dataset (RDD) is Spark's original data abstraction, and the foundation that DataFrames are built on. An RDD is a collection of items, split into partitions that live on different machines, together with the recipe for how it was computed. You rarely need RDDs for ordinary table work (DataFrames are easier and faster), but they teach you how Spark really behaves, and they are the right tool for unstructured data and custom logic.

The name explains the idea: resilient because lost partitions can be rebuilt from their lineage, distributed because the data is spread across the cluster, and dataset because it is a collection you can process with functions.

Creating RDDs

You create RDDs from the SparkContext, available as spark.sparkContext (often named sc). The two most common sources:

Python
sc = spark.sparkContext

# 1. From a Python collection that is already in the driver's memory
numbers = sc.parallelize([1, 2, 3, 4, 5, 6, 7, 8], numSlices=4)

# 2. From files (local, HDFS, S3 ...): one item per line
lines = sc.textFile("data/access-*.log")
  • parallelize copies a local collection to the cluster. It is great for examples and tests, but not for big data, because the collection must fit in the driver first.
  • textFile reads text files lazily, splitting them into partitions (by default, one per block of the file). It accepts a path, a glob pattern, or a directory.
  • wholeTextFiles returns (filename, content) pairs, useful for many small files.
  • You can also get an RDD from a DataFrame with df.rdd.

Nothing is read or computed at creation; the RDD is just a plan, as you saw in Lazy Evaluation, Transformations and Actions.

Partitions

An RDD is divided into partitions. A partition is the unit of parallelism: Spark runs one task per partition. You can ask how many an RDD has and what is inside each one:

Python
print(numbers.getNumPartitions())   # 4
print(numbers.glom().collect())     # [[1, 2], [3, 4], [5, 6], [7, 8]]

glom() turns each partition into a list so you can see the layout. The number of partitions controls the amount of parallelism: too few leaves cores idle, too many creates tiny tasks with scheduling overhead. A common rule of thumb is two to four partitions per CPU core in the cluster. Partitions and Partitioning covers this fully.

flowchart LR
    subgraph RDD["RDD of 8 numbers"]
        P0["Partition 0<br/>1, 2"]
        P1["Partition 1<br/>3, 4"]
        P2["Partition 2<br/>5, 6"]
        P3["Partition 3<br/>7, 8"]
    end
    P0 --> E1["Executor A"]
    P1 --> E1
    P2 --> E2["Executor B"]
    P3 --> E2

Each partition is processed by one task; executors host many partitions.

Immutability

RDDs are immutable: you cannot change the items in an RDD. Every transformation returns a new RDD and leaves the old one as it was.

Python
doubled = numbers.map(lambda n: n * 2)   # a new RDD
print(numbers.collect())                  # [1, 2, 3, 4, 5, 6, 7, 8] (unchanged)

Immutability makes the system simpler and safer. Since no one can change a partition after it exists, any copy of it is always correct, and recomputing it from lineage always gives the same answer. This is also why your functions should be pure: same input, same output, no hidden state.

Lineage and resilience

Each RDD remembers its parent RDDs and the function that produced it. This chain is the lineage graph. If an executor dies and a partition is lost, Spark does not need a replica: it re-runs the recorded steps on the original input for just that partition.

flowchart LR
    A["textFile<br/>read blocks"] --> B["filter<br/>keep errors"]
    B --> C["map<br/>extract url"]
    C --> D["RDD: urls"]

The lineage of the "urls" RDD: read, filter, map. Lost partitions are rebuilt along this path.

You can print an RDD's lineage with toDebugString(). Indentation marks shuffle boundaries. For long lineages (for example iterative algorithms) it is sometimes cut off with a checkpoint; see Fault Tolerance and Lineage.

A MiniRDD to explore

The real RDD runs on a cluster. The class below is a stand-in that models the same three ideas in plain Python: items split into partitions, immutability, and a recorded lineage. It is not distributed and it computes eagerly on small lists; the point is to see how the structure behaves.

Python MiniRDD: partitions, immutability and lineage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that filter keeps the partition structure even when a partition becomes empty: partition 0 of big is an empty list. Real Spark does the same; empty partitions still produce (very cheap) tasks.

Recomputing a lost partition

The next stand-in simulates losing a partition and rebuilding it from lineage. The "source" data stays intact; each step is stored as a function, so replaying them for one partition rebuilds exactly that partition.

Python Rebuilding a lost partition from lineage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real RDD partitions are stored on executors and recomputed by tasks scheduled across the cluster; real lineage also records dependencies and partitioners. The stand-ins keep just the principle: partitions are independent, data is immutable, and the recipe can rebuild any partition.

When to use RDDs

In current Spark you should reach for DataFrames first (see DataFrames: Creating and Inspecting). They know the schema, so Spark can optimize and use compact binary storage. RDDs make sense when:

  • the data has no useful schema (raw text, binary blobs);
  • you need fine control over partitioning or low-level operations;
  • you are working with legacy code or learning how Spark works.

An RDD holds opaque Python or JVM objects, so Spark cannot look inside them to optimize. And in PySpark, RDD functions run in Python worker processes with data crossing between the JVM and Python, which is slower than DataFrame operations that stay in the JVM.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the 'resilient' in RDD refer to?

    Each RDD remembers how it was built from its parents, so Spark can rebuild a lost partition by replaying those steps.
  2. 2. How many tasks does Spark run for one stage over an RDD with 12 partitions?

    A task processes one partition, so a stage over 12 partitions has 12 tasks.
  3. 3. What happens to numbers after doubled = numbers.map(lambda n: n * 2)?

    RDDs are immutable. Transformations always produce a new RDD.
  4. 4. Why is parallelize not suitable for big data?

    parallelize copies an existing local collection into the cluster. For large data, read from distributed storage with textFile or the DataFrame readers.

Summary

  • An RDD is an immutable, partitioned collection plus the lineage that produced it.
  • Create RDDs with sc.parallelize (small local data) or sc.textFile (files); nothing runs until an action.
  • One task runs per partition; getNumPartitions() and glom() reveal the layout.
  • Lineage lets Spark rebuild lost partitions by replaying the recorded steps.
  • Prefer DataFrames for structured data; use RDDs for unstructured data or low-level control.
  • The MiniRDD classes here are stand-ins that model partitions, immutability and lineage.

Next: RDD Transformations.