Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 48 ยท RDDs

RDD Actions

Learn the Spark RDD actions collect, count, take, first, reduce, fold, aggregate, foreach and saveAsTextFile, plus the collect pitfalls, with runnable stand-ins.

Transformations only describe work. Actions are the calls that make Spark actually run a job and hand something back: a number, a list, a file on disk. Every Spark program ends in at least one action, because without one nothing would ever execute (see Lazy Evaluation, Transformations and Actions).

Actions fall into three groups. Some bring data back to the driver (collect, take, first), some combine all elements into one value (count, reduce, fold, aggregate), and some produce side effects on the executors (foreach, saveAsTextFile). Knowing which group an action belongs to tells you where the data ends up and how much memory the driver needs.

Bringing data back to the driver

These actions return ordinary Python objects in your driver program.

Python
rdd = spark.sparkContext.parallelize(range(1, 11), 3)

rdd.collect()      # [1, 2, 3, ..., 10]  every element
rdd.count()        # 10
rdd.first()        # 1                   the first element
rdd.take(4)        # [1, 2, 3, 4]        the first four elements
rdd.takeOrdered(3) # [1, 2, 3]           the three smallest
rdd.top(3)         # [10, 9, 8]          the three largest

collect() returns the whole dataset. That is fine for a few hundred rows and a disaster for a few hundred million, because every executor sends its partitions to the driver, and the driver has to hold the combined list in memory. We come back to that in a moment.

take(n) is the safe alternative. It does not compute every partition. Spark starts with the first partition, and only looks at further partitions if it has not yet found n elements. On a huge dataset where the first partition holds enough rows, take(5) finishes almost immediately, while count() has to read everything. first() is simply take(1) that raises an error if the RDD is empty.

Combining elements: reduce, fold and aggregate

reduce collapses an RDD into a single value by applying a two-argument function over and over.

Python
rdd.reduce(lambda a, b: a + b)    # 55

Spark runs the function inside each partition first, then combines the per-partition results on the driver. Because the order of partitions and the grouping of elements is not under your control, the function must be commutative and associative. Addition, multiplication, max and min qualify. Subtraction does not, and reduce(lambda a, b: a - b) can give different answers depending on how the data was split.

reduce fails on an empty RDD, because there is no element to start from. fold fixes this by taking a zero value:

Python
rdd.fold(0, lambda a, b: a + b)   # 55

The zero value must be neutral for the function (0 for addition, 1 for multiplication, an empty collection for set union). There is a subtle trap: Spark applies the zero value once per partition and once more when merging the partition results. A neutral zero is harmless, a non-neutral one is multiplied by the number of partitions plus one.

aggregate is the most general form, and the one to use when the result has a different type than the elements. You give it a zero value, a function that folds one element into the accumulator inside a partition (seqOp), and a function that merges two accumulators from different partitions (combOp). The classic example is computing an average, where the accumulator is a (sum, count) pair:

Python
total, n = rdd.aggregate(
    (0, 0),
    lambda acc, x: (acc[0] + x, acc[1] + 1),     # seqOp: inside a partition
    lambda a, b: (a[0] + b[0], a[1] + b[1]),     # combOp: across partitions
)
print(total / n)   # 5.5
flowchart LR
    P0["Partition 0<br/>seqOp"] --> M["Driver<br/>combOp"]
    P1["Partition 1<br/>seqOp"] --> M
    P2["Partition 2<br/>seqOp"] --> M
    M --> R["Final value"]

aggregate folds each partition separately, then merges the partial results on the driver.

A stand-in for actions

The real Spark cannot run here, so the code below uses a small MiniRDD that keeps its data in a list of partitions. It is not distributed and it is not Spark's code; it mimics the semantics of the actions, including take scanning only as many partitions as it needs. The class prints which partitions were computed, so you can see the cost of each action.

Python collect, count, take and first on a MiniRDD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

take(2) only touches one partition, while take(5) needs two. collect and count always touch all four. That is the reason to prefer take when you only want to peek at your data.

The second example demonstrates the three combining actions on the same data, including the zero-value trap.

Python reduce, fold and aggregate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fold(10, ...) line shows why a non-neutral zero value produces a surprising answer. With three partitions the zero is added four times.

Side effects: foreach and saveAsTextFile

foreach(fn) runs a function on every element on the executors, and returns nothing. It is the right place for side effects such as writing to a database or sending a metric, but there are two catches. First, anything the function prints goes to the executor's log, not to your notebook. Second, tasks can be retried, so the function may run more than once for the same element. Make the effect idempotent (safe to repeat), or use foreachPartition to open one connection per partition rather than one per element.

saveAsTextFile(path) writes the RDD out as a directory of files, one part-NNNNN file per partition, plus an empty _SUCCESS marker once the job finishes. The path must not exist already. Each element is written as str(element) on its own line.

Python
rdd.map(lambda n: f"{n},{n * n}").saveAsTextFile("out/squares")
# out/squares/part-00000, part-00001, part-00002, _SUCCESS

The stand-in below writes into a dictionary instead of a disk, so you can see the layout and the per-partition files.

Python foreach and saveAsTextFile on a stand-in

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. In real Spark each partition is processed by a task on some executor, the files are written by those executors directly into the file system (HDFS, S3, local disk), and the data never passes through the driver. The stand-ins run in a single Python process, but they follow the same rules about zero values, partial results and output layout.

The collect pitfalls

collect() is the most common cause of driver out-of-memory errors in beginner programs. Keep these habits:

  • Never collect a big RDD. Use take, takeSample, count or write the result to storage instead.
  • Reduce first, collect after. Aggregate on the cluster (reduceByKey, aggregate), and only collect the small summary.
  • Watch the size limit. The setting spark.driver.maxResultSize (1 GB by default) makes the job fail if collected results exceed it. That is a safety net, not a plan.
  • Do not collect in order to loop. for x in rdd.collect() turns a cluster job into a single-machine loop. Use foreach, or better, a transformation.
  • Remember that each action re-runs the plan. Calling count() and then collect() on an uncached RDD computes it twice. See Caching and Persistence.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is rdd.take(5) usually cheaper than rdd.collect()?

    take scans partitions one after another (starting with the first) and stops when it has enough elements. collect has to compute and ship every partition.
  2. 2. An RDD has 4 partitions. You call rdd.fold(5, lambda a, b: a + b) and the real sum of the elements is 100. What does Spark return?

    The zero value is applied once in every partition (4 times) and once more when merging, so 5 is added 5 times: 100 + 25 = 125.
  3. 3. Which action is the right choice for computing an average, where the accumulator is a (sum, count) pair?

    aggregate lets the result type (a tuple) differ from the element type, with one function for folding inside a partition and one for merging partitions.
  4. 4. What does saveAsTextFile('out') create?

    Every partition is written by its own task as part-NNNNN. The _SUCCESS file marks a finished job, and the path must not exist already.

Summary

  • Actions run a job. Some return data to the driver (collect, take, first), some combine elements (count, reduce, fold, aggregate) and some write or act on executors (foreach, saveAsTextFile).
  • take and first compute only the partitions they need; collect computes and ships everything.
  • reduce needs a commutative and associative function and fails on an empty RDD; fold adds a zero value that is applied once per partition plus once for the merge.
  • aggregate handles results of a different type than the elements with seqOp and combOp.
  • foreach runs on executors and may be retried, so keep side effects repeatable.
  • saveAsTextFile writes a directory of part-NNNNN files; avoid collect on large data.

Next: work with keyed data in Key-Value (Pair) RDDs.