Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 36 ยท Spark and DataFrames

Transformations, Actions and Lazy Evaluation

Learn why Spark is lazy: what transformations and actions are, how the query plan and the Catalyst optimizer work, and when caching and explain() help.

When you run df.filter(...).select(...) in Spark, nothing happens. No data is read, and the cell finishes instantly. This surprises people coming from pandas, but it is the key to how Spark can be fast: it waits until you ask for a result, looks at the whole chain of steps and then optimises the entire plan before running it.

Two kinds of operations

  • Transformations describe a new DataFrame from an old one: select, filter, withColumn, groupBy().agg(), join, orderBy, union. They are lazy.
  • Actions produce a result or a side effect and force execution: count(), show(), collect(), first(), take(n), write..., display(), toPandas().
Python
from pyspark.sql import functions as F

orders = spark.read.table("main.sales.orders")          # lazy: no data read yet

big = (orders
       .filter(F.col("amount") > 100)                    # lazy
       .select("order_id", "city", "amount")             # lazy
       .withColumn("tax", F.col("amount") * 0.2))        # lazy

print(big.count())                                       # action: now Spark runs the job
big.show(5)                                              # another action: runs again

Each action starts a fresh run of the whole chain, so two actions on the same DataFrame read and compute the data twice unless you cache it.

What Spark builds: the plan

Behind the scenes Spark keeps a logical plan (what to do). The Catalyst optimizer rewrites it to be cheaper, then creates a physical plan (how to do it on the cluster). Examples of what Catalyst does:

  • Predicate pushdown: pushes filters down to the file reader so unneeded files and row groups are skipped.
  • Column pruning: reads only the columns that are used.
  • Reordering and combining projections and filters.
  • Choosing join strategies such as broadcast joins.

You can see the plans (read-only Databricks code):

Python
big.explain()            # physical plan
big.explain(True)        # parsed, analysed, optimised logical plan and physical plan

In the result you would look for PushedFilters and a ReadSchema that lists only the columns you use. That is the optimizer at work.

flowchart LR
    A["Your code<br/>transformations"] --> B["Logical plan"]
    B --> C["Catalyst<br/>optimizer"]
    C --> D["Physical plan"]
    D --> E["Tasks on executors"]
    F["Action: count(), write()"] -.triggers.-> E

Transformations only extend the plan. The dashed arrow shows an action starting the real work.

Benefits of laziness

  1. Optimisation across steps. Because Spark sees the full chain, it can reorder and prune.
  2. No wasted work. If you only take(5), Spark may read a tiny part of the data.
  3. Fault tolerance. The plan records how each partition is computed, so a lost partition can be recomputed.

The cost: errors may appear late (a bad column name in line 3 can fail only at the action) and timing is confusing. A cell with only transformations is fast; the cell with the action seems slow because it does everything.

Caching

If you use a DataFrame several times, you can keep its result around:

Python
big.cache()             # lazy: marks it for caching
big.count()             # the first action fills the cache
big.show(5)             # served from memory
big.unpersist()         # release when done

Caching helps when the same expensive result is reused several times in one session. It does not help a single pass. On serverless compute cache() is not supported, and Delta's own disk cache on classic compute already speeds repeated reads of files, so measure before you cache.

Stand-in: a lazy pipeline

The stand-in below builds a tiny lazy pipeline: each transformation only records a step, and the action runs the whole list. It also counts how many times the data is "read", showing why two actions mean two runs.

Python Stand-in: transformations record, actions run

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second stand-in adds a simple cache. The first action computes and stores the result; later actions reuse it.

Python Stand-in: a cache avoids recomputing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Debugging lazy code

  • Run df.printSchema() and df.limit(5).show() after each step to catch mistakes early; both are cheap.
  • Remember that an exception in a transformation is reported when the action runs. Read the stack trace from the bottom to find your line.
  • Avoid actions inside loops. A loop calling count() on every iteration reruns the whole plan each time.

Common mistakes

  • Calling count() just to "check something" before every step on big data.
  • Caching everything. Memory is limited, and caching a DataFrame used once wastes it.
  • Believing that a fast cell means the job is done. It may only have built a plan.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these is an action?

    count() returns a result and forces Spark to run the plan. The others are lazy transformations.
  2. 2. What does predicate pushdown do?

    The optimizer applies filters as early as possible, often inside the file reader.
  3. 3. You call count() and then show() on the same uncached DataFrame. What happens?

    Each action triggers its own execution unless the intermediate result is cached.
  4. 4. When does cache() actually fill the cache?

    cache() only marks the DataFrame. The next action computes and stores it.

Summary

  • Transformations build a plan; actions run it.
  • Catalyst optimises the whole plan (pushdown, pruning, join choice) before execution.
  • Each action reruns the plan unless the data is cached; cache only data you reuse.
  • Check schemas and small samples as you go, and avoid actions inside loops.

Next: Joins, Aggregations and Window Functions