Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 48 ยท Practice and Review

Interview Questions

Common Apache Spark interview questions with short answers on architecture, lazy evaluation, shuffles, joins, partitions, skew, tuning and streaming.

Spark interviews rarely ask you to recite API names. They ask whether you understand what happens inside the cluster when you write a line of code: where data moves, what is recomputed, and why a job is slow. This page collects questions that come up again and again, grouped by topic, with short answers you can expand in your own words.

Try to answer each question out loud before you open it. If you cannot explain an idea in two or three sentences, follow the link to its lesson. Four of the answers come with a runnable demo. Real Spark cannot run in the browser, so those demos are stand-ins: small plain-Python models of the idea. They show the logic, not Spark's actual implementation or speed.

flowchart TD
    Q["Interview question"] --> A["Name the concept"]
    A --> B["Explain what Spark does internally"]
    B --> C["Give a cost or example"]
    C --> D["Mention a trade-off"]

A good answer follows four steps: concept, mechanism, example, trade-off.

Architecture and execution

1. What are the driver, the executors and the cluster manager?

Show answer

The driver runs your main program, builds the plan (the DAG), splits it into stages and tasks, and schedules them. Executors are worker processes on the cluster that run tasks, hold cached data and write shuffle files. The cluster manager (standalone, YARN, Kubernetes) only hands out resources. It does not run your code. See Spark Architecture.

2. What is the difference between a job, a stage and a task?

Show answer

Each action triggers one job. A job is cut into stages at shuffle boundaries (wide dependencies). A stage is made of one task per partition, and each task runs the same pipeline of narrow transformations on its partition. More partitions means more tasks, which can run in parallel if there are free cores.

3. What is lazy evaluation and why does Spark use it?

Show answer

Transformations such as map and filter only record steps. Nothing runs until an action such as count or write is called. Because Spark sees the whole plan before running it, it can pipeline narrow steps together, skip unneeded columns, push filters down to the data source and recompute lost partitions from the lineage. See Lazy Evaluation, Transformations and Actions.

The demo below records steps and runs nothing until collect().

Python Demo: lazy plan, eager action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Building the plan called double zero times. Only the action ran it, five times, once per element, in a single pass.

4. RDD, DataFrame or Dataset: which one should you use?

Show answer

Default to DataFrames. They carry a schema, so the Catalyst optimizer and Tungsten can plan and execute the query efficiently. RDDs give full control but no optimizer: use them for unstructured data or custom partitioning. Datasets add compile-time types and exist only in Scala and Java; in Python a DataFrame is the equivalent. See DataFrames: Creating and Inspecting.

Shuffles, joins and partitions

5. What is a shuffle, and what triggers one?

Show answer

A shuffle redistributes data across partitions so that rows with the same key meet. Map tasks write partitioned files to local disk, and the next stage reads them over the network. It is the most expensive step in most jobs. Triggers include groupBy/reduceByKey, join (unless broadcast), distinct, repartition and orderBy. Narrow steps like map, filter and coalesce do not shuffle. See Key-Value (Pair) RDDs.

6. Why is reduceByKey better than groupByKey?

Show answer

reduceByKey combines values inside each partition before the shuffle, so each partition sends at most one record per key. groupByKey ships every record and builds a full list per key, which can exhaust memory on a hot key. Use groupByKey only when you really need all values together.

7. What is the difference between repartition and coalesce?

Show answer

repartition(n) does a full shuffle and can increase or decrease the number of partitions, producing evenly sized ones. coalesce(n) only merges existing partitions without a shuffle, so it is cheap but can only decrease the count and may leave partitions uneven. See Partitions and Partitioning.

Python Demo: coalesce versus repartition

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The stand-in shows the trade-off: coalescing moves fewer rows but leaves sizes of 7 and 3, while repartitioning balances them at 5 and 5 by moving more rows.

8. Which join strategies does Spark use?

Show answer

A broadcast hash join sends the small table to every executor, so the big table is not shuffled. A sort-merge join shuffles and sorts both sides on the join key and is the default for two large tables. A shuffled hash join builds hash tables per partition. A table under spark.sql.autoBroadcastJoinThreshold (10 MB by default) is broadcast automatically, and you can force it with broadcast(df). See Joins.

9. What is data skew and how do you fix it?

Show answer

Skew means one key (or a few) holds most of the rows, so one task does most of the work while the others finish early. Fixes: enable adaptive query execution (AQE) skew-join handling, broadcast the smaller side, filter out null or junk keys, or salt the hot key by appending a random suffix, aggregating, and then merging the partial results. See Performance Tuning.

Python Demo: skew and salting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without salting one of four partitions holds 92 of 100 rows. After salting the rows spread out almost evenly, and merging the salted keys back gives the same totals.

10. How many partitions should a job have?

Show answer

A common target is 2 to 4 tasks per available core, with partitions of roughly 100 to 200 MB. Too few partitions waste cores and risk out-of-memory errors; too many create scheduling overhead and small files. For shuffles the setting is spark.sql.shuffle.partitions (200 by default), and AQE can coalesce small shuffle partitions at runtime.

Memory, caching and fault tolerance

11. When should you cache a DataFrame?

Show answer

Cache when the same expensive result is used by more than one action and fits in memory or disk. Do not cache data used once: caching has a cost and evicts other data. Remember that cache() is lazy, so the first action fills it, and call unpersist() when you are done. See Caching and Persistence.

12. How does Spark recover from a failed task or a lost executor?

Show answer

Each RDD remembers its lineage, the recipe that produced it. If a partition is lost, Spark recomputes only that partition from its parents; if a shuffle file disappeared, the map stage that produced it reruns. Failed tasks are retried (4 attempts by default), and speculative execution can start a duplicate of a slow task. Checkpointing cuts long lineages. See Fault Tolerance and Lineage.

Tuning and optimization

13. What does the Catalyst optimizer do?

Show answer

Catalyst turns your query into a logical plan, applies rules such as predicate pushdown and column pruning, picks a physical plan (including the join strategy using statistics) and then generates efficient code with Tungsten whole-stage code generation. df.explain() shows the result. See Catalyst Optimizer and Query Plans.

14. Why are Python UDFs slow, and what should you use instead?

Show answer

A regular Python UDF moves each row between the JVM and a Python process and hides the logic from Catalyst. Prefer built-in functions in pyspark.sql.functions; if you need Python, use a pandas UDF, which transfers columnar batches with Apache Arrow. See User-Defined Functions and Pandas UDFs.

Streaming

15. What is a watermark in Structured Streaming?

Show answer

Streaming aggregations over event time keep state for each window. The watermark is a threshold (the latest event time seen minus the allowed lateness) after which Spark may close old windows, free their state and drop events that arrive too late. It trades completeness for bounded memory. See Event Time, Windows and Watermarks.

Python Demo: a watermark dropping late events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Event 4 (time 5) is late but still accepted, because its window had not yet closed. Event 6 (time 3) arrives after the watermark passed 20, so its window is gone and it is dropped.

16. What are the output modes of a streaming query?

Show answer

Append writes only new final rows, update writes rows that changed since the last trigger, and complete rewrites the whole result table each time (only valid for aggregations). Which one is allowed depends on the query: a windowed aggregation in append mode needs a watermark so Spark knows when a window is final. See Structured Streaming Basics.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which operation does NOT cause a shuffle?

    coalesce merges existing partitions without moving data between executors in a full shuffle. The other three redistribute rows by key or evenly.
  2. 2. A job has one task that runs for an hour while the others finish in a minute. What is the most likely cause?

    A single straggler task points at skew. Salting, AQE skew handling or a broadcast join are the usual remedies.
  3. 3. What does the cluster manager do?

    The driver plans and schedules, executors run tasks and cache data, and the cluster manager only provides the resources.
  4. 4. In a streaming aggregation, what does the watermark allow Spark to do?

    The watermark bounds state: once a window is older than the watermark it is finalized and later events for it are dropped.
  5. 5. Which is the best answer to 'why does Spark use lazy evaluation'?

    Seeing the full plan enables pipelining, pushdown and pruning, and the recorded lineage gives fault tolerance.

Summary

  • Know the roles: the driver plans, executors run tasks, the cluster manager hands out resources.
  • Actions create jobs, shuffles split jobs into stages, and each stage runs one task per partition.
  • Shuffles are the main cost: prefer reduceByKey, broadcast joins and sensible partition counts.
  • Skew shows up as straggler tasks; salting, AQE and broadcasting are the standard fixes.
  • Streaming answers revolve around event time, watermarks, state and output modes.
  • In every answer name the concept, the mechanism, an example and a trade-off.

Next: keep the essentials at hand in the Spark Cheat Sheet.