Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 48 ยท RDDs

Broadcast Variables and Accumulators

Learn how Spark ships closures to executors, why driver variables are copied, and how broadcast variables and accumulators share data safely, with runnable stand-ins.

When you pass a function to map or foreach, that function does not run on your laptop. Spark packages it up, sends it to executors on other machines, and runs it there. Anything the function refers to from the surrounding code travels along with it. This is the source of two of the most common beginner bugs: a counter that stays at zero, and a huge lookup table that makes every job slow.

Spark offers two shared variables for these cases. A broadcast variable sends a read-only value to each executor once. An accumulator lets tasks add to a number that only the driver can read. This lesson explains closures first, because both tools only make sense once you see why plain variables fail.

Closures and serialization

A closure is a function together with the variables it uses from its surrounding scope. Before a task runs, Spark serializes the closure (PySpark uses cloudpickle, Scala and Java use Java or Kryo serialization) and sends a copy to every task. Each task works on its own copy. Changes made to that copy are never sent back.

Python
counter = 0

def count_it(x):
    global counter
    counter += 1

rdd.foreach(count_it)
print(counter)      # 0 in cluster mode: each task incremented its own copy

In local mode you may sometimes see a different result because everything shares one process, which is exactly why the bug survives testing and appears on the cluster. The behavior is undefined by design: never mutate a driver variable from inside a transformation or action.

Serialization has a second consequence. Everything a closure touches must be serializable, so capturing a database connection, a file handle, or an object holding one raises an error such as PicklingError (Python) or Task not serializable (Scala). Create such resources inside the task, once per partition, with mapPartitions or foreachPartition.

The stand-in below makes the copying visible. Each simulated task receives a deep copy of the state, so what it changes is lost, exactly as in a cluster.

Python Why a driver variable stays at zero

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The right way to move data back to the driver is through return values (reduce, aggregate, collect) or through an accumulator.

Broadcast variables

Suppose you enrich a billion records with a country-code lookup table of 50 MB. If the dictionary is simply captured by the closure, Spark sends a copy with every task. With 2,000 tasks that is 100 GB shipped over the network, and the driver spends its time serializing the same table again and again.

A broadcast variable fixes that. You wrap the value once on the driver; Spark sends it to each executor a single time (using an efficient peer-to-peer scheme), caches it there, and all tasks on that executor share the one copy.

Python
countries = {"DE": "Germany", "FR": "France", "JP": "Japan"}    # imagine 50 MB
bc = spark.sparkContext.broadcast(countries)

def enrich(record):
    code, amount = record
    return (bc.value.get(code, "unknown"), amount)

sales.map(enrich).take(3)
bc.unpersist()      # drop the copies from executors (can be re-sent if used again)
bc.destroy()        # release it for good

Rules for broadcast variables:

  • They are read-only. Changing bc.value inside a task changes only that task's view and is not shared.
  • Read them through .value inside tasks. If you use the original variable instead, you lose the benefit.
  • Broadcast only values that fit comfortably in each executor's memory. A multi-gigabyte table is a job for a join (see Joins).
  • DataFrames have the same idea built in: a broadcast join sends a small table to every executor automatically.
Python Bytes shipped: closure copy per task versus broadcast

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

flowchart TB
    D["Driver<br/>holds the table"] -->|"once"| E1["Executor 1<br/>cached copy"]
    D -->|"once"| E2["Executor 2<br/>cached copy"]
    E1 --> T1["Task A"]
    E1 --> T2["Task B"]
    E2 --> T3["Task C"]
    E2 --> T4["Task D"]

A broadcast variable travels once per executor; every task on that executor reads the same copy.

Accumulators

An accumulator is the opposite direction: tasks can only add to it, and only the driver can read it. It suits counters and sums that are a side product of your main job, such as the number of malformed lines you skipped.

Python
bad_lines = spark.sparkContext.accumulator(0)

def parse(line):
    parts = line.split(",")
    if len(parts) != 3:
        bad_lines.add(1)         # tasks add...
        return None
    return parts

good = lines.map(parse).filter(lambda r: r is not None)
print(good.count())              # an action runs the job
print(bad_lines.value)           # ...the driver reads (only after the action)

Reading .value inside a task is not allowed. In Scala, a named accumulator such as sc.longAccumulator("bad lines") also shows up in the web UI next to the stage that updated it; PySpark accumulators cannot be named. For something other than numbers you can write a custom accumulator, for example one that collects a small set of error types.

The caveat: retries and recomputation

Accumulators come with a guarantee that is easy to misread. Spark guarantees that updates made inside an action (such as foreach) are applied exactly once per task, even when a task is retried. Updates made inside a transformation such as map carry no such promise, because a transformation can run more than once: when a stage is re-executed after a failure, when an uncached RDD is used by a second action, or when a cached partition is evicted and recomputed. Each rerun adds again.

So use accumulators in transformations only for diagnostics where approximate numbers are acceptable. If the count drives a decision or ends up in a report, compute it with a real action such as count() or aggregate.

The next stand-in models the rule. Each task buffers its additions locally and the buffer is only merged into the accumulator when that attempt succeeds. We then run the same pipeline twice to see the transformation caveat.

Python Accumulators: exactly once in actions, repeated in transformations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real Spark serializes closures with cloudpickle or Java serialization, distributes broadcasts with a BitTorrent-like protocol between executors, and merges accumulator updates when it receives each task's result. The stand-ins use copy.deepcopy, simple arithmetic and a per-task buffer to show the same rules; no network, executor or retry logic is real.

Choosing the right tool

You want to... Use
Send a read-only table or model to all tasks Broadcast variable
Count or sum something as a side effect Accumulator (prefer inside actions)
Return a result computed on the cluster reduce, aggregate or collect, never a mutated variable
Use a connection or client in tasks Create it per partition with mapPartitions or foreachPartition

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You increment a global Python counter inside rdd.foreach() on a cluster. What does the driver see afterwards?

    Closures are serialized and copied to each task. Changes to the copies are never sent back, so use an accumulator or return values instead.
  2. 2. What is the main benefit of a broadcast variable?

    Broadcasting ships one copy per executor, shared by all tasks there, which saves network traffic and driver serialization time.
  3. 3. Where does Spark guarantee that accumulator updates are applied exactly once per task?

    Transformations can be re-executed (second action, lost cached partitions, re-run stages), which adds again. Inside actions the update of each task is applied once.
  4. 4. A task needs a database connection. What is the safest way to provide it?

    Connections cannot be serialized and sent between machines. Open them on the executor where they will be used.

Summary

  • Spark serializes closures and sends a copy to each task, so mutating a driver variable inside a task does not work on a cluster.
  • Return data to the driver with actions (reduce, aggregate, collect) or with accumulators.
  • A broadcast variable ships a read-only value once per executor; read it with .value inside tasks and unpersist() when finished.
  • An accumulator is add-only for tasks and readable only on the driver.
  • Accumulator updates inside actions are applied once per task; inside transformations they can be repeated by retries and recomputation, so treat them as diagnostics.
  • Create non-serializable resources such as connections inside the task, per partition.

Next: move on from RDDs to structured data in DataFrames: Creating and Inspecting.