Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 48 ยท Getting Started

Lazy Evaluation, Transformations and Actions

Learn why Spark is lazy, the difference between transformations and actions, narrow and wide transformations, and lineage, with a runnable lazy pipeline.

The most surprising thing about Spark for newcomers is that most lines of code do nothing at all. You can write a filter, a map and a join on a terabyte of data, and the line returns instantly. The work only happens when you ask for a result. This design is called lazy evaluation, and it is the reason Spark can optimize your program and recover from failures.

This lesson explains the two kinds of operations (transformations and actions), splits transformations into narrow and wide ones, shows what a lineage is, and gives you a runnable lazy pipeline to experiment with.

Transformations and actions

Every Spark operation is one of two kinds:

  • A transformation describes a new dataset in terms of an old one: filter, map, select, join, groupBy. It returns immediately with a new dataset object and does not touch the data. Datasets are immutable: a transformation never changes its input, it returns a new one.
  • An action asks for a result and therefore forces the work to run: count, collect, show, take, write, reduce.
Python
numbers = spark.sparkContext.parallelize(range(1, 1_000_001))

evens   = numbers.filter(lambda n: n % 2 == 0)    # transformation: instant
squares = evens.map(lambda n: n * n)               # transformation: instant

print(squares.count())                             # action: the work runs now
print(squares.take(3))                             # action: runs again

The first two lines only record a recipe. count() triggers a job that runs the whole recipe. The second action, take(3), runs the recipe again from the start unless you cache the data (see Caching and Persistence).

Why Spark is lazy

Laziness is not laziness for its own sake. It buys three things.

  1. Optimization. Because Spark sees the entire chain before running, it can reorder and merge steps. A filter written after a join can be pushed before it, so fewer rows are joined. Column selections can be pushed down into the file reader so unused columns are never read.
  2. Pipelining. Consecutive simple steps run together in one pass over each partition, with no intermediate dataset stored between them. filter then map is one loop, not two.
  3. Fault tolerance. The recipe itself is the backup. If a machine fails, Spark replays the recipe for the lost partitions only.
flowchart LR
    A["parallelize"] --> B["filter"]
    B --> C["map"]
    C --> D{"Action?"}
    D -->|"no"| E["Nothing runs<br/>plan is recorded"]
    D -->|"count()"| F["Plan is optimized<br/>and executed"]

Transformations extend the plan; an action makes Spark run it.

A practical consequence: errors can appear late. A typo inside a map function may not show up until the action runs, and the stack trace points at the action, not the line with the bug.

Narrow and wide transformations

Transformations differ in how much data movement they need.

  • Narrow transformations: each output partition depends on one input partition. Examples: map, filter, flatMap, select, withColumn, union. They need no network and are pipelined together.
  • Wide transformations: an output partition depends on many input partitions, so data must be shuffled across the cluster. Examples: groupByKey, reduceByKey, distinct, join (in general), repartition, orderBy. They are far more expensive and mark stage boundaries.
flowchart TB
    subgraph Narrow["Narrow: one input partition feeds one output partition"]
        N1["P0"] --> N2["P0'"]
        N3["P1"] --> N4["P1'"]
    end
    subgraph Wide["Wide: outputs read from many inputs (shuffle)"]
        W1["P0"] --> W3["Q0"]
        W1 --> W4["Q1"]
        W2["P1"] --> W3
        W2 --> W4
    end

Narrow steps stay inside a partition; wide steps cross partition boundaries.

When you write Spark code, a good habit is to count the wide transformations. Each one is a shuffle, and shuffles dominate the running time of most jobs. Shuffles and Stages goes deep on this.

Lineage

Because every dataset is defined by a transformation of its parents, Spark can draw the complete chain back to the source data. This chain is the dataset's lineage. You can print it for an RDD:

Python
print(squares.toDebugString().decode())

It shows each step and where the shuffle boundaries are. For DataFrames, the equivalent tool is explain() (see Catalyst Optimizer and Query Plans). Lineage is the record Spark uses to rebuild lost partitions.

Seeing laziness: a lazy pipeline stand-in

The class below behaves like a tiny lazy dataset. Transformations only record a step in a list; an action replays the list over the data. The data is a plain list and nothing runs in parallel; the point is the behavior.

Python Nothing runs until an action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two action calls each print [running job ...]: with no cache, every action replays the full plan. The original numbers object is untouched, because transformations returned new objects. This also shows immutability.

The next example proves the "late error" behavior and the pipelining idea. A bad function sits in the middle of a plan; the error only appears at the action. Then we count how many times a function runs for a pipelined plan versus a plan that materializes each step.

Python Late errors and pipelining

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real Spark builds a DAG, cuts it into stages at wide transformations, runs tasks on many partitions in parallel and optimizes the plan first. The stand-ins only model the key behavior: recording steps, running them on demand, immutability and late errors.

Common mistakes

  • Calling an action in a loop. Each count() re-runs the entire plan. Cache the data first if you need it more than once.
  • Using collect() on large results. It pulls everything into the driver's memory and can crash it. Prefer take, show or write.
  • Putting a side effect in a transformation. A map that writes to a database may run several times (retries, re-computation) or never. Do side effects in actions such as foreach, and make them safe to repeat.
  • Expecting print output from lazy code. Print statements inside functions run on executors, not on your console.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these is an action?

    count returns a value to the driver, so it forces the job to run. The others only extend the plan.
  2. 2. What is a wide transformation?

    Wide transformations such as reduceByKey and join regroup data across partitions, which requires moving data over the network.
  3. 3. You call evens = numbers.filter(...) on a huge dataset and it returns instantly. Why?

    Transformations are lazy. The filter runs only when an action asks for a result.
  4. 4. How does Spark recover a lost partition?

    The lineage is the recipe for building each partition from the source, so Spark recomputes only what was lost.

Summary

  • Transformations are lazy and return new immutable datasets; actions run the plan and return results.
  • Laziness lets Spark optimize, pipeline narrow steps and recover from failures.
  • Narrow transformations stay inside a partition; wide ones need a shuffle and start a new stage.
  • The lineage is the chain of transformations back to the source; toDebugString() and explain() print it.
  • Each action replays the plan unless the data is cached.
  • Errors and print output can surface late or on executors, not at the line you wrote.

Next: build on these ideas with RDD Basics.