Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 48 ยท RDDs

RDD Transformations

Learn the core RDD transformations map, filter, flatMap, distinct, union, sample and sortBy, which ones shuffle, and try them in a runnable MiniRDD.

Transformations are how you describe a computation on an RDD. Each one takes an RDD and returns a new RDD; none of them run until an action asks for a result (see Lazy Evaluation, Transformations and Actions). This lesson covers the everyday set: map, filter, flatMap, distinct, union, sample and sortBy, and tells you which of them are narrow and which cause a shuffle.

All the real examples below are read-only PySpark. After each group you get a runnable MiniRDD stand-in that behaves like the real transformations on small lists.

map and filter

map(f) applies a function to every item and returns exactly one output per input. filter(p) keeps the items for which the predicate returns true.

Python
nums = sc.parallelize([1, 2, 3, 4, 5, 6])

squares = nums.map(lambda n: n * n)            # 1, 4, 9, 16, 25, 36
evens   = nums.filter(lambda n: n % 2 == 0)    # 2, 4, 6

Both are narrow: every output partition is built from one input partition, no data moves between machines, and Spark pipelines them into one pass. map always keeps the count; filter can only shrink it.

flatMap

flatMap(f) applies a function that returns a sequence for each item, and then flattens all the sequences into one RDD. Use it when one input can produce zero, one or many outputs, such as splitting lines into words.

Python
lines = sc.parallelize(["to be or", "not to be"])

words_nested = lines.map(lambda s: s.split())       # [["to","be","or"], ["not","to","be"]]
words        = lines.flatMap(lambda s: s.split())   # ["to","be","or","not","to","be"]

map with split gives an RDD of lists, which is rarely what you want. flatMap gives an RDD of words. An empty list from the function simply produces no output, so flatMap can also filter. It is narrow.

flowchart LR
    A["'to be or'"] --> B["split"]
    C["'not to be'"] --> D["split"]
    B --> E["to, be, or"]
    D --> F["not, to, be"]
    E --> G["flatten: one RDD of 6 words"]
    F --> G

flatMap runs the function on each line and merges the resulting lists into one RDD.

A first MiniRDD

The stand-in class below models partitions as lists and implements map, filter and flatMap as narrow operations, one partition at a time.

Python MiniRDD: map, filter and flatMap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

distinct and union

distinct() removes duplicate items. To know that two equal items on different machines are duplicates, Spark must bring equal items together, so distinct is a wide transformation: it triggers a shuffle.

union(other) simply puts the partitions of two RDDs side by side. It does not remove duplicates and does not shuffle. It is narrow, and the result has as many partitions as both inputs combined.

Python
a = sc.parallelize([1, 2, 3, 3])
b = sc.parallelize([3, 4, 5])

a.union(b).collect()               # [1, 2, 3, 3, 3, 4, 5]   duplicates kept
a.union(b).distinct().collect()    # [1, 2, 3, 4, 5]         order is not guaranteed

If you want union semantics from SQL (no duplicates), chain distinct() yourself. Because partitions of the result of distinct arrive in an arbitrary order, never rely on the order of its output.

sample

sample(withReplacement, fraction, seed) returns a random subset. The fraction is the expected portion of items to keep, not an exact count; each item is kept independently with that probability (without replacement). Pass a seed to make the result repeatable on the same data and partitioning.

Python
tenth = sc.parallelize(range(1000)).sample(False, 0.1, seed=42)
print(tenth.count())      # about 100, not exactly 100

Sampling is narrow and cheap, which makes it a good way to develop and test on a small slice of big data before running the full job.

sortBy

sortBy(keyfunc, ascending=True, numPartitions=None) sorts the RDD by the value the key function returns. Sorting a distributed dataset needs a global ordering, so Spark first samples the data to pick range boundaries, shuffles items into ranges, and sorts each range. That makes sortBy a wide transformation and an expensive one. After it, partition 0 holds the smallest items, partition 1 the next range, and so on, so reading the partitions in order gives the full sorted result.

Python
people = sc.parallelize([("ana", 31), ("li", 25), ("raj", 40)])
people.sortBy(lambda p: p[1], ascending=False).collect()
# [("raj", 40), ("ana", 31), ("li", 25)]

Sort only when you need to, preferably late and on already reduced data.

flowchart TB
    A["map, filter, flatMap, union, sample<br/>narrow: no shuffle"] --> B["Same stage"]
    C["distinct, sortBy<br/>wide: shuffle"] --> D["Stage boundary"]

Narrow transformations are pipelined; wide ones start a new stage.

More MiniRDD: distinct, union, sample, sortBy

The second stand-in adds the wide operations. To keep the idea visible, distinct and sort_by do a small hash or range shuffle into new partitions, and sample uses a seeded random generator. It does not reproduce Spark's exact random numbers, so the sampled items will differ from a real cluster; only the behavior (a fraction, repeatable by seed) is the same.

Python MiniRDD: distinct, union, sample and sortBy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sort_by partitions are in range order, so concatenating them gives the full sort. That is the property real Spark's range partitioning provides.

Real versus stand-in. The real distinct, sortBy and sample run as distributed tasks with real shuffle files, a sampling step to choose range boundaries, and their own random number generators. The stand-in shows structure and behavior on small lists, not performance or exact outputs.

Tips for choosing transformations

  • Prefer fewer, wider steps over many passes when possible, but keep each function simple and pure.
  • Filter early to shrink the data before any wide step.
  • Use flatMap instead of map followed by manual flattening.
  • Avoid sortBy and distinct on huge data unless you need them; consider reduceByKey-style aggregation (see Key-Value (Pair) RDDs) or approximate methods instead.
  • Remember that DataFrames do many of these things better; see DataFrames: Creating and Inspecting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between map and flatMap when the function returns a list?

    map keeps one output per input, so an RDD of lists. flatMap concatenates all returned sequences into a single RDD.
  2. 2. Which transformation does NOT shuffle data?

    union just places partitions side by side. distinct and sortBy must regroup data across partitions, so they shuffle.
  3. 3. rdd.sample(False, 0.1, seed=1).count() on 1,000 items returns...

    The fraction is an expected proportion, not an exact count. The same seed on the same data gives a repeatable result.
  4. 4. What does a.union(b) do with duplicates?

    RDD union does not deduplicate. Chain distinct() if you need unique items.

Summary

  • map gives one output per input; flatMap allows zero or many and flattens the results; filter keeps matching items.
  • map, filter, flatMap, union and sample are narrow and pipelined.
  • distinct and sortBy are wide: they shuffle, and sortBy first samples to pick range boundaries.
  • union keeps duplicates; sample(withReplacement, fraction, seed) is approximate and repeatable with a seed.
  • Filter early, sort late, and prefer DataFrames for structured work.
  • The MiniRDD classes are stand-ins for structure and behavior, not for speed.

Next: get results back with RDD Actions.