Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 48 ยท Under the Hood

Catalyst Optimizer and Query Plans

See how Spark turns DataFrame code into logical and physical plans, how Catalyst pushes filters and prunes columns, and how to read explain() output.

When you write a DataFrame or SQL query, you describe what result you want, not how to compute it. Spark's query optimizer, called Catalyst, decides the how. It rewrites your query into an equivalent but cheaper one: filters move closer to the data, unused columns are never read, constant expressions are computed once. This is the main reason DataFrames usually beat hand-written RDD code.

Understanding Catalyst helps you read explain() output, find why a query is slow, and write code the optimizer can work with. You should know Lazy Evaluation, Transformations and Actions and the DataFrame basics from Columns and Expressions.

From code to plan

Because DataFrame operations are lazy, Spark has your whole query in hand before it runs anything. It then moves through four phases.

flowchart LR
    Q["DataFrame or SQL query"] --> A["Analysis: resolve names and types"]
    A --> L["Logical optimization: rewrite rules"]
    L --> P["Physical planning: choose strategies and cost them"]
    P --> C["Code generation: run on executors"]

Catalyst turns a query into an analyzed plan, an optimized logical plan, a physical plan and finally generated code.

  1. Analysis. Names such as amount are matched against the catalog and the schema (see Datasets, Schemas and Data Types). A typo in a column name fails here, with an AnalysisException, before any data is touched.
  2. Logical optimization. A logical plan is a tree of operators (scan, filter, project, join, aggregate) that says what to do without saying how. Catalyst applies rule-based rewrites until nothing changes.
  3. Physical planning. For each logical operator Spark generates one or more physical plans, for instance a broadcast hash join or a sort-merge join, and picks one using statistics and cost rules.
  4. Code generation. The chosen plan is compiled to JVM bytecode, covered below.

Reading explain()

Call explain() on any DataFrame. With no argument it shows the physical plan; explain(True) (or explain("extended")) shows the parsed, analyzed and optimized logical plans plus the physical plan, and explain("formatted") produces a more readable layout.

Python
orders = spark.read.parquet("orders/")

result = (orders
          .select("customer", "amount", "note")
          .filter("amount > 100")
          .select("customer", "amount"))

result.explain()

A simplified physical plan for this query looks like the following. The exact text varies with Spark's version, settings and data source:

TEXT
== Physical Plan ==
*(1) Project [customer#12, amount#13]
+- *(1) Filter (isnotnull(amount#13) AND (amount#13 > 100.0))
   +- *(1) ColumnarToRow
      +- FileScan parquet [customer#12,amount#13]
           PushedFilters: [IsNotNull(amount), GreaterThan(amount,100.0)],
           ReadSchema: struct<customer:string,amount:double>

Read it from the bottom up, because data flows from the scan upward. Notice what happened to the code you wrote. You asked for three columns first, but ReadSchema lists only two: Spark noticed note is never used and did not read it. The filter appears in PushedFilters, so the Parquet reader can skip row groups whose statistics show no value over 100. The *(1) prefix marks operators fused into one code-generated stage.

The rules that matter most

Predicate pushdown. A filter is moved as close to the data source as possible: below projections, below joins when it only involves one side, and into the file reader or database (JDBC) when the source supports it. Less data enters the pipeline in the first place.

Column pruning. Only columns that the query actually needs are read. For columnar formats like Parquet and ORC this saves real I/O, because unused columns are physically separate on disk (see Reading and Writing Data).

Constant folding and simplification. An expression like price * (1 + 0.2) becomes price * 1.2, 1 = 1 disappears, and a filter like x > 5 AND x > 5 collapses to one condition.

Join selection. If one side is small, Spark picks a broadcast hash join and avoids a shuffle; otherwise a sort-merge join. With Adaptive Query Execution (AQE), this choice can change at run time using real statistics from finished stages (see Performance Tuning).

Project and filter reordering. Rules combine adjacent projections and push projections below filters when it helps, as long as results stay identical.

Pushdown can fail. If you hide the column in a UDF, such as filter(my_udf("amount")), the optimizer cannot translate the condition for the data source and cannot push it down (see User-Defined Functions and Pandas UDFs). The same happens when you cast the column in a way the source cannot evaluate. Prefer plain comparisons on plain columns.

Tungsten and whole-stage code generation

After planning, the Tungsten engine decides how the work executes. It stores rows in a compact binary format, off the regular Java object heap where possible, which reduces memory use and garbage collection. It also uses cache-friendly layouts.

Whole-stage code generation is Tungsten's biggest trick. A naive engine calls an operator for every row: ask the filter for a row, ask the scan for a row, and so on, with virtual function calls in between. Spark instead fuses a chain of simple operators (scan, filter, project) into one generated Java function with a tight loop, which the JVM's JIT compiler can optimize as hand-written code. In explain() the operators with the same *(n) number belong to one generated function. Operators that need a shuffle, or cannot be generated, break the chain.

A stand-in toy optimizer

Real Catalyst has hundreds of rules working on rich trees. The stand-in below shows the shape of the idea: a plan is a small tree of dictionaries, and three rules (constant folding, column pruning and filter pushdown into the scan) are applied repeatedly until the plan stops changing. It is a model of the technique, not Spark's code, and its plan format is invented for this lesson.

Python A toy rule-based optimizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Constant folding turned 50 * 2 into 100, column pruning removed note and region from the scan, and pushdown handed the filter to the scan. The loop ran a second round only to confirm that no rule changed anything: that is how Catalyst's "run until fixed point" batches behave. The Filter stays in the plan even after pushdown, just as in real Spark, because a source may skip only some data and Spark still checks the rest.

Real versus stand-in. Real Catalyst works on immutable trees of expression objects with statistics, runs rule batches in a fixed order (some once, some to a fixed point), and its physical planner compares alternative strategies. The toy has no statistics and no physical planning.

Helping the optimizer

  • Use built-in column expressions instead of UDFs or Python lambdas, so filters and projections stay visible.
  • Filter early and select only needed columns; the optimizer often does this for you, but explicit code is easier to reason about.
  • Choose columnar formats such as Parquet or ORC so pushdown and pruning have something to work with.
  • Collect table statistics with ANALYZE TABLE ... COMPUTE STATISTICS for catalog tables, so the planner can pick sensible join strategies.
  • Read explain() before and after a change. Look at ReadSchema, PushedFilters, the join type and the number of exchanges (shuffles; see Shuffles and Stages).

An Exchange operator in a physical plan marks a shuffle. Fewer exchanges almost always mean a cheaper query.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In which phase does a misspelled column name raise an AnalysisException?

    The analyzer resolves names and types before optimization and before any task runs, so the error is raised early.
  2. 2. What does column pruning achieve for a Parquet table?

    Parquet stores columns separately, so unused ones are simply never read. Nothing is deleted from the source.
  3. 3. Which change is most likely to stop predicate pushdown?

    A UDF is opaque, so Catalyst cannot translate the condition into something the data source understands.
  4. 4. What does an Exchange node in a physical plan indicate?

    Exchange is the physical operator for a shuffle, such as hash partitioning before an aggregation or a sort-merge join.
  5. 5. What does whole-stage code generation do?

    Operators such as scan, filter and project are compiled together, avoiding per-row virtual calls between operators.

Summary

  • Catalyst analyzes your query, rewrites the logical plan with rules, chooses a physical plan and generates code.
  • explain() shows the plan; read it bottom up and look for ReadSchema, PushedFilters, join type and Exchange nodes.
  • Predicate pushdown, column pruning, constant folding and join selection are the rules that matter most day to day.
  • Tungsten stores rows compactly and whole-stage code generation fuses operators into one loop.
  • UDFs, odd casts and non-columnar formats limit what the optimizer can do.

Next: see how data is split across the cluster in Partitions and Partitioning.