Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 48 ยท Practice and Review

Practice: Exercises with Solutions

12 Apache Spark practice exercises with solutions covering RDDs, DataFrames, aggregations, joins and window functions, with runnable Python stand-ins.

Reading about Spark and writing Spark are different skills. This page gives you 12 exercises that revisit the whole course: four on RDDs, four on DataFrame basics and aggregations, two on joins and two on window functions. Each one has a short task, an editor with starter data, and a hidden solution that shows both the real PySpark code and a stand-in you can run.

The real Spark engine cannot run inside this page, so every editor is a stand-in: plain Python (or pandas) that expresses the same idea on a few rows. That is deliberate. If you can solve the problem on six rows with a dictionary or a pandas group-by, you understand the logic, and the PySpark version is mostly a change of vocabulary. The PySpark code in each solution is read-only: copy it into a local pyspark session (see Installing Spark and Your First PySpark Program) to try it for real.

flowchart LR
    A["RDD exercises 1-4"] --> B["DataFrame exercises 5-8"]
    B --> C["Join exercises 9"]
    C --> D["Window exercises 10-12"]

The exercises follow the order of the course: from low-level RDDs up to window functions.

How to get the most out of them:

  • Try first, peek later. Give each task an honest attempt before opening the solution.
  • Compare, don't copy. A different solution can still be correct; read ours for ideas.
  • Say the plan in Spark terms. For each exercise ask yourself which steps are narrow and which cause a shuffle.

RDD exercises

These practice RDD basics, transformations, actions and pair RDDs.

1. Word count (Beginner)

Count how often each word appears in the lines below. Print the counts sorted alphabetically as (word, count) tuples.

Python Exercise 1: Word count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
counts = (sc.parallelize(lines)
            .flatMap(lambda line: line.split())
            .map(lambda w: (w, 1))
            .reduceByKey(lambda a, b: a + b))
print(sorted(counts.collect()))

Stand-in:

Python
from collections import Counter

words = [w for line in lines for w in line.split()]     # flatMap
counts = Counter(words)                                  # map + reduceByKey
print(sorted(counts.items()))

flatMap is the step that turns one line into many words. reduceByKey is preferred over groupByKey because it combines inside each partition first.

2. Squares of even numbers (Beginner)

From the numbers 1 to 10, keep the even ones, square them, and add the squares together.

Python Exercise 2: Squares of evens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
rdd = sc.parallelize(range(1, 11)).filter(lambda n: n % 2 == 0).map(lambda n: n * n)
print(rdd.collect(), rdd.sum())

Stand-in:

Python
from functools import reduce

squares = [n * n for n in nums if n % 2 == 0]
print(squares, reduce(lambda a, b: a + b, squares))

filter and map are lazy transformations; collect and sum are the actions that run the job. The stand-in runs eagerly, which is the main difference.

3. Average per key (Intermediate)

Given (name, score) pairs, compute each person's average score. Remember that an average of averages is wrong, so carry a sum and a count.

Python Exercise 3: Average per key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
sums = sc.parallelize(scores).aggregateByKey(
    (0, 0),
    lambda acc, v: (acc[0] + v, acc[1] + 1),
    lambda a, b: (a[0] + b[0], a[1] + b[1]),
)
print(sums.mapValues(lambda s: s[0] / s[1]).collect())

Stand-in:

Python
acc = {}
for name, value in scores:
    total, count = acc.get(name, (0, 0))
    acc[name] = (total + value, count + 1)
print({name: total / count for name, (total, count) in acc.items()})

The accumulator type (a tuple) differs from the value type (a number), which is exactly why reduceByKey is not enough here.

4. Top three words (Intermediate)

Using the same lines as exercise 1, print the three most frequent words. When counts tie, the alphabetically smaller word wins.

Python Exercise 4: Top three words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
top3 = counts.takeOrdered(3, key=lambda kv: (-kv[1], kv[0]))

Stand-in:

Python
from collections import Counter

counts = Counter(w for line in lines for w in line.split())
print(sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:3])

takeOrdered keeps only a small bounded set per partition, so it never has to sort the whole dataset on the driver. Avoid collect() followed by a Python sort on big data.

DataFrame exercises

These practice columns and expressions, filtering and sorting and aggregations. The stand-ins use pandas, whose vocabulary is close enough to see the idea.

5. Totals and labels (Beginner)

Add a total column (price * qty) and a bulk column that says "bulk" when qty >= 10 and "single" otherwise.

Python Exercise 5: Totals and labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
from pyspark.sql import functions as F

df.withColumn("total", F.col("price") * F.col("qty")) \
  .withColumn("bulk", F.when(F.col("qty") >= 10, "bulk").otherwise("single")) \
  .show()

Stand-in:

Python
import numpy as np

products["total"] = products["price"] * products["qty"]
products["bulk"] = np.where(products["qty"] >= 10, "bulk", "single")
print(products)

withColumn never changes the original DataFrame; it returns a new one. pandas mutates in place, which is a real difference.

6. Clean and rank (Beginner)

Employees with a missing salary must be ignored. Print the three best-paid of the rest.

Python Exercise 6: Clean and rank

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
df.dropna(subset=["salary"]).orderBy(F.col("salary").desc()).limit(3).show()

Stand-in:

Python
print(emp.dropna(subset=["salary"]).sort_values("salary", ascending=False).head(3))

The three winners are Ed (6100), Ann (5200) and Di (4800). In Spark, comparing with null never gives true, so a filter like salary > 0 would also drop Bob silently.

7. Per-department summary (Intermediate)

For each department print the number of employees, the average salary and the highest salary. Ignore missing salaries in the average, as Spark does.

Python Exercise 7: Per-department summary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
df.groupBy("dept").agg(
    F.count("*").alias("n"),
    F.avg("salary").alias("avg"),
    F.max("salary").alias("top"),
).show()

Stand-in:

Python
print(emp.groupby("dept").agg(n=("name", "count"), avg=("salary", "mean"), top=("salary", "max")).reset_index())

IT has three people but an average of 5650, because Bob's missing salary is skipped by avg and max. Note count("*") counts rows while count("salary") would count only non-null salaries.

8. Pivot the sales (Intermediate)

Sum amount per region and quarter, with regions as rows and quarters as columns. Missing combinations should show 0.

Python Exercise 8: Pivot the sales

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
df.groupBy("region").pivot("quarter", ["Q1", "Q2"]).sum("amount").na.fill(0).show()

Stand-in:

Python
print(sales.pivot_table(index="region", columns="quarter", values="amount", aggfunc="sum", fill_value=0))

Passing the list of pivot values (["Q1", "Q2"]) saves Spark an extra job that would otherwise discover the distinct values.

Join exercises

These practice joins.

9. Orders and customers (Intermediate)

Given orders and customers, print (a) the orders that have a matching customer, with the customer name, (b) the orders whose customer is unknown, and (c) the customers who never ordered.

Python Exercise 9: Orders and customers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
inner = orders.join(customers, "cust", "inner")
unknown = orders.join(customers, "cust", "left_anti")
never_ordered = customers.join(orders, "cust", "left_anti")

Stand-in:

Python
print(orders.merge(customers, on="cust", how="inner"))

merged = orders.merge(customers, on="cust", how="left", indicator=True)
print(merged[merged["_merge"] == "left_only"][["order_id", "cust", "amount"]])

print(customers[~customers["cust"].isin(orders["cust"])])

Part (a) returns orders 1, 2 and 3. Part (b) returns order 4 (customer c9). Part (c) returns Cy. A left_anti join keeps only left rows without a match and, unlike a left join followed by a null filter, does not add the right-hand columns.

Window exercises

These practice window functions.

10. Rank within a department (Intermediate)

Rank the employees with a salary inside each department, highest first.

Python Exercise 10: Rank within a department

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
from pyspark.sql.window import Window

w = Window.partitionBy("dept").orderBy(F.col("salary").desc())
df.dropna(subset=["salary"]).withColumn("rank", F.rank().over(w)).show()

Stand-in:

Python
d = emp.dropna(subset=["salary"]).copy()
d["rank"] = d.groupby("dept")["salary"].rank(method="min", ascending=False).astype(int)
print(d.sort_values(["dept", "rank"]))

method="min" mirrors rank(), which gives tied rows the same rank and leaves a gap afterwards. Use row_number() when you need a unique number per row.

11. Running balance (Intermediate)

Each account has transactions on days 1, 2, 3. Add a balance column that is the running sum of amt per account, ordered by day.

Python Exercise 11: Running balance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
w = (Window.partitionBy("acct").orderBy("day")
           .rowsBetween(Window.unboundedPreceding, Window.currentRow))
df.withColumn("balance", F.sum("amt").over(w)).show()

Stand-in:

Python
t = tx.sort_values(["acct", "day"]).copy()
t["balance"] = t.groupby("acct")["amt"].cumsum()
print(t)

Account a ends at 90 (100, 70, 90) and account b at 40. Spell out the frame with rowsBetween: the default frame with an orderBy uses a RANGE frame, which treats rows with equal sort values together.

12. Latest record per user (Advanced)

Each row is a plan change for a user. Keep only the most recent row (highest ts) per user. This is the classic deduplication pattern in ETL jobs.

Python Exercise 12: Latest record per user

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution

Real PySpark:

Python
w = Window.partitionBy("user").orderBy(F.col("ts").desc())
latest = df.withColumn("rn", F.row_number().over(w)).filter("rn = 1").drop("rn")

Stand-in:

Python
print(events.sort_values("ts").drop_duplicates("user", keep="last").sort_values("user"))

dropDuplicates(["user"]) in Spark keeps an arbitrary row per user, so it is wrong here. A row_number window gives you control over which row survives.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which Spark call keeps the single latest row per key in a controlled way?

    dropDuplicates keeps an arbitrary row. A row_number window with an explicit ordering decides exactly which row survives.
  2. 2. Why is an average of per-partition averages usually wrong?

    Each partial average needs its own weight (the row count). Carrying (sum, count) and dividing at the end gives the true mean.
  3. 3. What does a left_anti join return?

    left_anti keeps left rows with no match and returns only the left columns, which makes it the clean way to find missing keys.
  4. 4. In these exercises, why are the editors stand-ins rather than real PySpark?

    A real Spark runtime needs a JVM and a cluster or local installation. The stand-ins teach the logic; the PySpark code in each solution is for your own machine.

Summary

  • RDD tasks combine flatMap, map, filter, reduceByKey and aggregateByKey; averages need a (sum, count) accumulator.
  • DataFrame tasks use withColumn, when/otherwise, dropna, groupBy().agg() and pivot, and null values are skipped by aggregates.
  • inner and left_anti joins answer "which rows match" and "which rows are missing".
  • Window functions (rank, row_number, running sums) solve per-group ranking and de-duplication without collapsing rows.
  • Solve the logic on a few rows first; translating it to PySpark is the easy part.

Next: rehearse explaining these ideas out loud in Interview Questions.