Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 48 ยท Getting Started

Spark Architecture

Understand how Spark runs a program: the driver, executors, cluster manager, jobs, stages, tasks and the DAG, with a runnable driver and executor simulator.

When you run a Spark program, your code does not simply execute line by line on one computer. A small number of cooperating processes divide the work, send it to other machines, collect the answers and keep track of what failed. Knowing who does what turns many confusing error messages and slow jobs into things you can reason about.

This lesson introduces the cast: the driver, the executors and the cluster manager. It then follows one request down the hierarchy of job, stage and task, and finishes with a runnable simulator of a driver handing tasks to a pool of workers.

The three main roles

A Spark application is made of these parts:

  • Driver: the process that runs your main program (your PySpark script or notebook). It creates the SparkSession, turns your code into a plan, splits the plan into tasks, schedules them and gathers results. There is exactly one driver per application.
  • Executors: worker processes that run tasks and keep data in memory or on disk. Each executor has a fixed number of CPU cores and a slice of memory. An application usually has many executors, spread over machines.
  • Cluster manager: the service that hands out machines and resources. Spark can use its own standalone manager, YARN, Kubernetes, or (in older setups) Mesos. It decides where executors start; it does not decide what they compute.
flowchart LR
    subgraph Driver["Driver process"]
        SC["SparkSession<br/>plan and scheduler"]
    end
    CM["Cluster manager<br/>standalone, YARN, Kubernetes"]
    subgraph W1["Worker node 1"]
        E1["Executor<br/>tasks and cache"]
    end
    subgraph W2["Worker node 2"]
        E2["Executor<br/>tasks and cache"]
    end
    SC -->|"1. request resources"| CM
    CM -->|"2. launch executors"| E1
    CM -->|"2. launch executors"| E2
    SC -->|"3. send tasks"| E1
    SC -->|"3. send tasks"| E2
    E1 -->|"4. results"| SC
    E2 -->|"4. results"| SC

The driver asks the cluster manager for executors, then talks to the executors directly.

Once executors are running, the driver communicates with them directly. The cluster manager is out of the loop for the actual work, which is why a busy manager does not slow down a running job.

Local mode

You do not need a cluster to learn. In local mode the driver and one or more executors live inside a single process on your machine, using threads as cores (in a real JVM). local[4] means four threads; local[*] means one per CPU core. The architecture is identical; only the placement differs. That is why code written on a laptop can run on a cluster with a changed configuration and no code changes.

Job, stage and task

Spark organizes the work in a hierarchy. Understanding it is the key to reading the Spark UI later.

  • A job is triggered by an action such as count() or write. Calling an action sends one job to the scheduler. (Transformations only build a plan; see Lazy Evaluation, Transformations and Actions.)
  • A job is divided into stages. A stage is a group of steps that can run without moving data between machines. A new stage begins wherever data must be shuffled: regrouped across the network, for example by groupBy or a join.
  • A stage is divided into tasks: one per partition of the data. A task is the smallest unit of work, and it runs on one core of one executor.
flowchart TB
    J["Job<br/>triggered by an action"] --> S1["Stage 1<br/>read, filter, map"]
    J --> S2["Stage 2<br/>aggregate after shuffle"]
    S1 --> T1["Task: partition 0"]
    S1 --> T2["Task: partition 1"]
    S1 --> T3["Task: partition 2"]
    S2 --> T4["Task: partition 0"]
    S2 --> T5["Task: partition 1"]
    S1 -.->|"shuffle"| S2

One job, two stages split by a shuffle, and one task per partition inside each stage.

If a dataset has 200 partitions and 20 executor cores are free, Spark runs 20 tasks at a time and the rest wait in line, in waves. More partitions than cores is normal and healthy. It keeps all cores busy and makes each task small.

The DAG

Spark records your transformations as a directed acyclic graph (DAG): nodes are datasets, edges are operations, and there are no cycles. The DAG scheduler in the driver cuts this graph at shuffle boundaries into stages, and the task scheduler sends the tasks of each stage to executors. Because the whole graph is known before anything runs, Spark can optimize it, run independent branches at once, and recompute a lost piece from its recipe (see Fault Tolerance and Lineage).

A short real example shows how code maps onto this picture. It is read-only here.

Python
rows = spark.range(1_000_000)                      # narrow: no stage yet
evens = rows.filter("id % 2 = 0")                   # narrow: same stage
grouped = evens.groupBy((evens.id % 10).alias("k")) # needs a shuffle
result = grouped.count()                            # still lazy
result.show()                                       # action: 1 job, 2 stages

The show() call is the action. The scheduler finds one shuffle (for the groupBy), so the job has two stages.

A driver and executor simulator

The real scheduler is JVM code and cannot run here. The stand-in below models the shape of it in plain Python: a driver cuts data into partitions, makes one task per partition, and hands them to "executor cores" in waves. Browsers cannot start real threads in this runner, so the waves run one after another, but the scheduling logic is the same idea.

Python A tiny driver and executor pool

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The answer matches a plain single-threaded sum, however many partitions or cores you use. The second stand-in adds the stage boundary. Stage 1 turns numbers into (key, value) pairs on every partition; the shuffle moves all pairs with the same key to the same place; stage 2 adds them up.

Python Two stages and a shuffle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. The real driver builds a DAG, serializes tasks, ships them over the network to executor JVMs, and tracks failures and retries. Real shuffles write files to disk on the map side and fetch them over the network on the reduce side. The stand-ins above keep only the shape: partitions, one task per partition, limited cores, and a regroup between two stages.

Why this matters in practice

Many practical problems come straight from this picture:

  • Out of memory on the driver usually means you pulled too much data back with an action like collect().
  • A slow stage often hides one very large partition (data skew) holding up an entire wave of tasks.
  • "Too many small tasks" happens when there are thousands of tiny partitions, and scheduling costs more than the work.
  • Executor lost errors point to executors killed for exceeding memory or to dying machines; Spark re-runs the lost tasks elsewhere.

You will learn to diagnose each of these in the later chapters.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which process turns your program into tasks and collects the final results?

    The driver runs your main program, builds the plan, schedules tasks and gathers results. Executors only run tasks.
  2. 2. What typically starts a new stage inside a job?

    Narrow steps are pipelined into one stage. A shuffle boundary forces the data to be redistributed, so a new stage begins.
  3. 3. A stage has 100 partitions and the cluster has 10 free cores. How does Spark run it?

    One task is created per partition. A core runs one task at a time, so extra tasks wait for a free core.
  4. 4. What is the job of the cluster manager?

    YARN, Kubernetes or the standalone manager allocate resources. After that the driver talks to the executors directly.

Summary

  • The driver plans and schedules; executors run tasks and hold cached data; the cluster manager allocates resources.
  • An action triggers a job; shuffles split a job into stages; each stage runs one task per partition.
  • Spark builds a DAG of operations, which lets it optimize the plan and recompute lost data.
  • Local mode uses threads in one process with the same architecture as a cluster.
  • Typical problems (driver OOM, skew, tiny tasks, lost executors) map directly onto these parts.

Next: Installing Spark and Your First PySpark Program.