Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 48 ยท Practice and Review

Spark Cheat Sheet

A one-page Apache Spark and PySpark cheat sheet: RDD, DataFrame, SQL, joins, window functions, key configs and spark-submit flags, with runnable stand-ins.

This is a quick reference for the Spark things you look up most often: creating data, transforming it, joining, windowing, tuning with configuration, and submitting a job. It is meant to sit open in another tab while you work, so each section is a compact table or snippet with links to the full lesson.

All Spark commands below are read-only: real Spark needs a JVM, which cannot run inside this page. Two runnable stand-ins at the end show the same ideas in plain Python and pandas: one query written as DataFrame calls and as SQL, and a small spark-submit command builder. The code assumes the usual imports and a session:

Python
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.window import Window

spark = SparkSession.builder.appName("demo").getOrCreate()
sc = spark.sparkContext
flowchart LR
    R["Read<br/>csv, json, parquet"] --> T["Transform<br/>select, filter, join"]
    T --> A["Aggregate<br/>groupBy, window"]
    A --> W["Write<br/>parquet, table"]

Most Spark jobs follow this shape: read, transform, aggregate, write.

RDD quick reference

See RDD Basics, RDD Transformations and RDD Actions.

Task Code Kind
Create from a list sc.parallelize(data, 4) source
Read text files sc.textFile("path/*.txt") source
Apply a function rdd.map(f) narrow
Keep matching items rdd.filter(f) narrow
One item to many rdd.flatMap(f) narrow
Remove duplicates rdd.distinct() wide
Combine two RDDs rdd.union(other) narrow
Sort rdd.sortBy(f) wide
Sum per key pairs.reduceByKey(lambda a, b: a + b) wide
Per-key custom type pairs.aggregateByKey(zero, seqOp, combOp) wide
Join pair RDDs a.join(b), leftOuterJoin, cogroup wide
Bring to the driver collect(), take(n), first() action
Count and fold count(), reduce(f), fold(zero, f) action
Save rdd.saveAsTextFile("out") action

Prefer reduceByKey over groupByKey, use take(n) instead of collect() on big data, and see Broadcast Variables and Accumulators for shared state.

DataFrame quick reference

See Columns and Expressions, Filtering, Sorting and Deduplicating and Grouping and Aggregations.

Python
df = spark.createDataFrame([("ann", 30), ("bob", 25)], ["name", "age"])
df.printSchema()                     # column names and types
df.show(5, truncate=False)           # print rows

df.select("name", (F.col("age") + 1).alias("next_age"))
df.withColumn("adult", F.col("age") >= 18)
df.withColumnRenamed("age", "years")
df.drop("age")
df.filter((F.col("age") > 20) & F.col("name").isNotNull())
df.orderBy(F.col("age").desc())
df.distinct()
df.dropDuplicates(["name"])          # keeps an arbitrary row per name
df.dropna(subset=["age"]).fillna({"age": 0})
df.limit(10)
df.count()                           # action
Expression Meaning
F.when(cond, a).otherwise(b) if / else column
F.col("x").cast("int") change the type
F.coalesce(a, b) first non-null value
F.col("x").isin(1, 2) membership test
F.concat_ws("-", a, b) join strings with a separator
F.to_date("d", "yyyy-MM-dd") parse a date
F.explode("arr") one row per array item
F.expr("a * 2 AS b") embed a SQL expression

Comparisons with null are never true, so use isNull() and isNotNull().

Aggregations and pivots

Python
df.groupBy("dept").agg(
    F.count("*").alias("n"),
    F.avg("salary").alias("avg_salary"),
    F.max("salary").alias("top"),
)
df.groupBy("region").pivot("quarter", ["Q1", "Q2"]).sum("amount")
df.rollup("region", "product").sum("amount")      # subtotals and grand total
df.agg(F.approx_count_distinct("user_id"))        # fast, approximate

count("*") counts rows; count("col") counts non-null values. Aggregates skip nulls.

SQL quick reference

See Spark SQL and Temp Views.

Python
df.createOrReplaceTempView("people")
spark.sql("""
    SELECT dept, AVG(salary) AS avg_salary
    FROM people
    WHERE salary IS NOT NULL
    GROUP BY dept
    ORDER BY avg_salary DESC
""").show()
spark.catalog.listTables()

SQL and the DataFrame API compile to the same optimized plan, so pick whichever reads best.

Joins

See Joins.

Call Result
a.join(b, "id") inner join on a shared column (the key appears once)
a.join(b, a.id == b.id, "left") left outer; unmatched right columns are null
"right", "full" right outer, full outer
"left_semi" left rows that have a match, left columns only
"left_anti" left rows with no match, left columns only
a.crossJoin(b) every combination (can explode in size)
a.join(F.broadcast(small), "id") broadcast the small side, no shuffle of a

Joining on an expression keeps both key columns, which makes later references ambiguous: join on the column name, or drop one side.

Window functions

See Window Functions.

Python
w = Window.partitionBy("dept").orderBy(F.col("salary").desc())

df.withColumn("rn", F.row_number().over(w))      # 1, 2, 3 ... unique
df.withColumn("rk", F.rank().over(w))            # ties share a rank, gaps follow
df.withColumn("dr", F.dense_rank().over(w))      # ties share a rank, no gaps
df.withColumn("prev", F.lag("salary", 1).over(w))
df.withColumn("next", F.lead("salary", 1).over(w))

running = (Window.partitionBy("acct").orderBy("day")
                 .rowsBetween(Window.unboundedPreceding, Window.currentRow))
df.withColumn("balance", F.sum("amt").over(running))

Reading and writing

See Reading and Writing Data.

Python
spark.read.option("header", True).option("inferSchema", True).csv("data.csv")
spark.read.json("events.json")
spark.read.parquet("lake/orders")

(df.write
   .mode("overwrite")                  # append, overwrite, ignore, errorifexists
   .partitionBy("year", "month")
   .parquet("lake/orders"))

Prefer an explicit schema over inferSchema for production jobs, because inference reads the data twice and can guess differently from run to run.

Caching, partitions and plans

Task Code
Cache for reuse df.cache() or df.persist(StorageLevel.MEMORY_AND_DISK)
Release the cache df.unpersist()
Number of partitions df.rdd.getNumPartitions()
Full shuffle to n partitions df.repartition(200, "key")
Merge without a shuffle df.coalesce(10)
Show the plan df.explain() or df.explain("formatted")
Cut a long lineage sc.setCheckpointDir("dir"), then df.checkpoint()

Important configuration

Set options when building the session, with spark.conf.set(key, value) for SQL settings, or with --conf on spark-submit. See Performance Tuning.

Setting Default Purpose
spark.sql.shuffle.partitions 200 partitions after a shuffle
spark.sql.adaptive.enabled true (Spark 3.2+) adaptive query execution
spark.sql.autoBroadcastJoinThreshold 10 MB automatic broadcast join size limit
spark.executor.memory 1g heap per executor
spark.executor.cores 1 on YARN, all on standalone cores per executor
spark.driver.memory 1g driver heap (raise it before heavy collect)
spark.dynamicAllocation.enabled false grow and shrink executors with load
spark.serializer Java serializer set to org.apache.spark.serializer.KryoSerializer for RDDs

spark-submit

See Deploying Spark.

Terminal
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 10 \
  --executor-cores 4 \
  --executor-memory 8g \
  --driver-memory 4g \
  --conf spark.sql.shuffle.partitions=400 \
  --py-files deps.zip \
  job.py arg1 arg2
Flag Meaning
--master local[*], spark://host:7077, yarn, k8s://https://host:443
--deploy-mode client (driver on the submitting machine) or cluster (driver on the cluster)
--packages Maven coordinates of extra jars, such as connectors
--files, --py-files ship files or Python dependencies to executors
--conf key=value any Spark property

Two runnable stand-ins

The first stand-in writes one query twice, as chained DataFrame-style calls and as SQL, and shows that the answers match. It uses pandas and SQLite in place of Spark.

Python Same query: DataFrame style and SQL style

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second stand-in turns a cluster description into executor settings with a common rule of thumb (leave one core and 1 GB per node for the system, about five cores per executor, one slot for the driver or application master) and prints the matching spark-submit command. Treat the numbers as a starting point, not a law: the right values depend on your job.

Python A spark-submit command builder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. The stand-ins show the logic only. Real Spark plans queries with Catalyst, runs them in parallel on executors and reads files from distributed storage. The sizing function is a heuristic, not something Spark computes for you.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which join type returns left rows that have NO match on the right, with only the left columns?

    left_anti keeps unmatched left rows. left_semi keeps matched ones, and both return only the left columns.
  2. 2. What does spark.sql.shuffle.partitions control?

    It sets the number of partitions after a shuffle such as a groupBy or a join (200 by default); AQE may coalesce them at runtime.
  3. 3. Which call gives tied rows the same rank WITHOUT leaving gaps afterwards?

    rank() leaves gaps after ties (1, 1, 3), dense_rank() does not (1, 1, 2), and row_number() never ties.
  4. 4. What is the difference between --deploy-mode client and cluster?

    The deploy mode decides where the driver process lives. Cluster mode suits production, because the job survives your laptop disconnecting.

Summary

  • RDDs: map, filter, flatMap are narrow; reduceByKey, join, distinct shuffle; actions such as collect and count run the job.
  • DataFrames: select, withColumn, filter, groupBy().agg(), pivot; nulls need isNull() rather than ==.
  • SQL and the DataFrame API produce the same plan; register views with createOrReplaceTempView.
  • Joins: use left_semi and left_anti for existence checks and broadcast() for small tables.
  • Windows: row_number, rank, dense_rank, lag, lead and explicit frames.
  • Tune with spark.sql.shuffle.partitions, AQE, memory settings and spark-submit flags.

This is the last lesson of the course. Revisit any topic from the Practice: Exercises with Solutions page, or start again at What Is Apache Spark?.