Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 48 ยท RDDs

Key-Value (Pair) RDDs

Learn Spark key-value RDDs: reduceByKey, groupByKey, aggregateByKey, combineByKey, join, cogroup and countByKey, and why reduceByKey beats groupByKey.

Most real data has a natural key: a user id, a product code, a word, a date. When an RDD holds two-element tuples (key, value), Spark treats it as a pair RDD and unlocks a family of operations that work per key: summing values for each key, grouping them, or joining two datasets on the key. Nearly every classic Spark program, from word count to log analysis, is built from them.

A pair RDD is not a special class you have to create. Any RDD whose elements are tuples of two items gets the extra methods automatically. These operations are mostly wide transformations: all values for one key must meet on one partition, so data is shuffled across the network (see Lazy Evaluation, Transformations and Actions). How much data is shuffled is the whole story of this lesson.

Creating pairs

Python
words = spark.sparkContext.parallelize(["spark", "rdd", "spark", "key", "rdd", "spark"])

pairs = words.map(lambda w: (w, 1))          # (word, 1)
pairs.collect()
# [('spark', 1), ('rdd', 1), ('spark', 1), ('key', 1), ('rdd', 1), ('spark', 1)]

pairs.mapValues(lambda v: v * 10)            # change values, keep keys (no shuffle)
pairs.keys()                                 # just the keys
pairs.values()                               # just the values

mapValues is worth using whenever you only touch the value: it keeps the partitioner, so Spark knows the data is still grouped by key and can skip a later shuffle. A plain map hides that fact.

reduceByKey versus groupByKey

Both can compute a per-key sum, but they behave very differently.

Python
pairs.reduceByKey(lambda a, b: a + b).collect()
# [('spark', 3), ('rdd', 2), ('key', 1)]

pairs.groupByKey().mapValues(sum).collect()
# same result, much more data shuffled

groupByKey ships every (key, value) record across the network, collects all values of a key into one iterable on one machine, and only then do you sum. A single very frequent key can produce an iterable too big for memory.

reduceByKey first combines values locally inside each partition (a map-side combine), so each partition sends at most one record per key. Only those partial sums cross the network and get merged. For word count over a billion words with 50,000 distinct words, that is a difference of billions of records versus thousands per partition.

flowchart LR
    subgraph A["Partition A"]
        A1["spark,1 spark,1 rdd,1"]
    end
    subgraph B["Partition B"]
        B1["spark,1 rdd,1 key,1"]
    end
    A1 -->|"reduceByKey: combine first"| A2["spark,2 rdd,1"]
    B1 -->|"reduceByKey: combine first"| B2["spark,1 rdd,1 key,1"]
    A2 --> S["Shuffle: 5 records"]
    B2 --> S
    A1 -.->|"groupByKey: no combine"| T["Shuffle: 6 records"]
    B1 -.-> T

reduceByKey shrinks each partition's data before the shuffle; groupByKey sends every record.

The rule of thumb: use groupByKey only when you truly need all the values together (for example to sort them or take the median). For any aggregation that can be expressed as a merge, use reduceByKey, aggregateByKey or combineByKey.

A stand-in that counts shuffled records

The class below keeps pairs in partitions and routes keys with a hash partitioner: hash(key) % number_of_partitions. Python's built-in hash of strings changes between runs, so the stand-in uses a small stable hash. It counts how many records would cross the network. It is a model of the idea, not Spark's implementation.

Python reduceByKey versus groupByKey

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both give the same answer, but reduceByKey shuffled only a handful of partial sums while groupByKey moved every record. With real data the gap grows with the number of records per key.

aggregateByKey and combineByKey

reduceByKey requires the result type to equal the value type. Averages break that rule: you need (sum, count) while the values are plain numbers. Two more general operations solve it.

aggregateByKey(zero, seqOp, combOp) works like aggregate but per key. zero starts every key in every partition, seqOp folds one value into the accumulator inside a partition, and combOp merges accumulators from different partitions.

Python
scores = spark.sparkContext.parallelize([("ann", 90), ("bob", 70), ("ann", 80), ("bob", 60), ("ann", 100)])

sums = scores.aggregateByKey(
    (0, 0),
    lambda acc, v: (acc[0] + v, acc[1] + 1),
    lambda a, b: (a[0] + b[0], a[1] + b[1]),
)
sums.mapValues(lambda s: s[0] / s[1]).collect()
# [('ann', 90.0), ('bob', 65.0)]

combineByKey is the primitive underneath all of them. Instead of a zero value it takes three functions: createCombiner (turn the first value of a key in a partition into an accumulator), mergeValue (fold another value in) and mergeCombiners (merge accumulators across partitions). Use it when the starting accumulator depends on the first value, such as starting a list with [v].

Python Averages with aggregateByKey and combineByKey

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

join, cogroup and countByKey

Two pair RDDs can be combined on their keys.

Python
orders = sc.parallelize([("u1", "book"), ("u2", "pen"), ("u1", "lamp")])
users  = sc.parallelize([("u1", "Ann"), ("u3", "Cy")])

orders.join(users).collect()
# [('u1', ('book', 'Ann')), ('u1', ('lamp', 'Ann'))]        inner join

orders.leftOuterJoin(users).collect()
# u2 appears with None on the right side

orders.cogroup(users).mapValues(lambda p: (list(p[0]), list(p[1]))).collect()
# [('u1', (['book', 'lamp'], ['Ann'])), ('u2', (['pen'], [])), ('u3', ([], ['Cy']))]

join is an inner join: keys missing on either side vanish, and a key that appears twice on one side produces two output rows. leftOuterJoin, rightOuterJoin and fullOuterJoin keep unmatched keys with None. cogroup is the building block behind them: for each key it returns both lists of values, so nothing is dropped.

countByKey() is an action, not a transformation. It returns a dictionary {key: count} to the driver, so use it only when the number of distinct keys is small. lookup(key) returns all values of one key, and collectAsMap() collects the pairs into a dictionary (later duplicates overwrite earlier ones).

Python join, cogroup and countByKey on plain Python

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real Spark partitions the data by key with a HashPartitioner (using the key's hashCode, or Python's hash through PySpark), sorts and spills to disk during shuffles, and can skip the shuffle when both sides already share a partitioner. The stand-ins show the logic of combining and counting; they do not move anything over a network.

Practical advice

  • Prefer reduceByKey and friends over groupByKey followed by a map.
  • Use mapValues and flatMapValues instead of map when keys are unchanged, so the partitioner is preserved.
  • Pick keys carefully. Keys that are None or extremely frequent create skew: one task gets most of the data. We deal with it in Performance Tuning.
  • Join the smaller side wisely. RDD joins shuffle both sides; DataFrames can broadcast a small table instead (see Joins).
  • For structured data, prefer DataFrames. They give the optimizer more freedom than hand-written RDD code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does reduceByKey usually shuffle less data than groupByKey?

    The map-side combine means each partition sends at most one partial result per key. groupByKey sends every single record.
  2. 2. You need the average value per key. Which operation fits best?

    An average of averages is wrong, and reduceByKey cannot change the value type. aggregateByKey carries (sum, count) and divides at the end.
  3. 3. What does an inner join on pair RDDs return for a key that exists only in the left RDD?

    Inner joins keep only keys present on both sides. Use leftOuterJoin to keep unmatched left keys with None.
  4. 4. Which statement about countByKey is true?

    countByKey runs a job and returns a Python dict, so it is only safe when the number of distinct keys is small.

Summary

  • A pair RDD holds (key, value) tuples and gets per-key operations such as reduceByKey, groupByKey, join and cogroup.
  • reduceByKey combines inside each partition before the shuffle, so it moves far less data than groupByKey; use groupByKey only when you need every value together.
  • aggregateByKey and combineByKey allow the accumulator to have a different type than the values, which is how you compute averages.
  • join is an inner join; the outer variants keep unmatched keys with None, and cogroup returns both value lists per key.
  • countByKey and collectAsMap are actions that bring results to the driver; mapValues keeps the partitioner and avoids needless shuffles.

Next: learn how functions and data travel to executors in Broadcast Variables and Accumulators.