Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Lazy Evaluation

Lazy Frames: the Basics

Learn Polars lazy evaluation: create a LazyFrame with lazy() or scan_csv, chain operations, check schemas, and run the optimized plan with collect.

So far every Polars call has run immediately and returned a result. That is eager evaluation. Polars has a second mode, lazy evaluation, that is the recommended way to write real pipelines. A lazy query does not run when you write it. It only records what you want. When you finally ask for the result, Polars looks at the whole plan, rewrites it into a cheaper one, and then executes it.

Think of the difference between a cook who starts chopping the moment you say the first ingredient, and a cook who listens to the whole order first and then plans the fastest way to prepare it.

Making a LazyFrame

There are two common ways in.

  1. Turn an existing DataFrame into a lazy one with df.lazy().
  2. Start lazy from a file with a scan_* function: pl.scan_csv, pl.scan_parquet, pl.scan_ndjson. The file is not read yet; only its schema may be peeked at.
Python From eager to lazy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The methods you already know (filter, select, with_columns, group_by, sort, join, ...) exist on LazyFrame with the same names and the same expressions. Only the moment of execution changes: collect() is the "go" button and returns an ordinary DataFrame.

Printing a LazyFrame shows the plan

Printing a LazyFrame does not show data, because there is none yet. You can still get useful information about it without running it.

Python Inspecting without running

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

collect_schema() is the recommended way to ask "what columns and types will the result have?" because it works out the answer from the plan without computing any data. (Older code used lf.schema; in recent versions that triggers a warning for expensive plans.) explain() prints the plan as text; the next lesson, Query Plans and the Optimizer, reads such plans in detail.

Scanning files lazily

For a file, scan_csv returns a LazyFrame without reading the whole file. Polars needs only the first rows to guess the schema.

Python scan_csv: schema and plan

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plan shows PROJECT 2/4 COLUMNS: the scan will read only the two columns that the query needs, and the filter appears on the scan itself as a SELECTION. That is the optimizer at work.

On a normal installation you finish such a query with collect():

Python
result = (
    pl.scan_csv("sales.csv")
    .filter(pl.col("amount") > 30)
    .group_by("region")
    .agg(pl.col("amount").sum())
    .collect()
)

The browser build of Polars used on this site cannot execute file scans (scan_csv(...).collect()), although it can plan them, which is why the example above stops at explain(). Everything else about lazy mode, including DataFrame.lazy() queries, runs here. On your own machine, scans work normally.

Why lazy can be much faster

  1. Skip work you never needed. If you read a 50-column CSV but use two columns, the optimizer reads only those two.
  2. Filter early. A filter that could run at the start of the pipeline is moved there, so every later step handles fewer rows.
  3. Merge and reuse. Duplicate sub-computations are computed once; constant expressions such as 2 + 3 are folded to 5.
  4. Choose how to run. Independent branches can run in parallel, and the streaming engine can process data in chunks, so a file larger than memory can still be handled.

None of this changes your answer. The optimizer is only allowed to make rewrites that give identical results.

Eager or lazy: how to choose

Situation Choose
Exploring a small table in a notebook Eager is fine
Reading a file you will filter or aggregate Lazy (scan_*)
A multi-step pipeline you will run repeatedly Lazy
Data bigger than RAM Lazy with the streaming engine
You need to look at intermediate data Eager, or collect() at that step

A good habit: write pipelines on a LazyFrame, call collect() once at the end, and keep the result as an eager DataFrame for display, plotting or conversion.

What you cannot do on a LazyFrame

Because there is no data yet, a few eager conveniences are missing: no lf["col"] indexing, no lf.row(0), no print(lf) of values, no to_list. If you need one of those, collect() first. Most methods you will use in pipelines are there.

Python What fails on a LazyFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does a LazyFrame query actually execute?

    Lazy methods only record the plan. collect() optimizes and runs it, returning a DataFrame.
  2. 2. How do you get the column names and types of a LazyFrame without running it?

    collect_schema() resolves the schema from the plan alone.
  3. 3. Which is a reason lazy mode can be faster for CSV files?

    Projection pushdown lets the scan read only the needed columns, and predicate pushdown filters early.
  4. 4. What type does collect() return?

    collect() executes the plan and returns an eager DataFrame.

Summary

  • A LazyFrame records a plan; collect() optimizes and runs it and returns a DataFrame.
  • Create one with df.lazy() or pl.scan_csv/scan_parquet/...; the same methods and expressions work.
  • collect_schema() and explain() inspect a query without running it.
  • Lazy mode skips unused columns, filters early and merges work; results are identical to eager mode.
  • The in-browser build can plan file scans but not execute them; they run normally on your machine.

Next: Query Plans and the Optimizer