Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 37 ยท Lazy Evaluation

Lazy Pipelines and Streaming

Build reusable lazy pipelines in Polars with pipe and functions, run several queries with collect_all, and process data larger than memory with the streaming engine and sinks.

Lazy queries are ordinary Python objects, so you can pass them to functions, return them from functions, store them in lists, and combine them. This lesson shows how to organize a lazy pipeline into clean, testable pieces, how to run several queries together, and how Polars handles data that does not fit in memory.

Pipelines as functions

A function that takes a LazyFrame and returns a LazyFrame is a pipeline step. LazyFrame.pipe(func, *args) calls it and passes the frame as the first argument, so steps chain without nesting.

Python Steps as functions with pipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each step is small, named and testable on its own with a tiny frame. Because nothing runs until collect(), Polars still sees and optimizes the whole chain as one plan. Splitting code into functions does not cost performance.

You can also write reusable expressions as functions. They return pl.Expr and slot into any context:

Python
def with_vat(price: pl.Expr, rate: float = 0.2) -> pl.Expr:
    return (price * (1 + rate)).round(2)

df.select(with_vat(pl.col("price")).alias("gross"))

Branching: one source, several results

Often you start from one dataset and need several outputs. With lazy frames you can build each query separately and collect them together with pl.collect_all, which runs them in parallel and shares common work where it can.

Python collect_all

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If several queries start with the same expensive steps (reading and cleaning a big file), collect_all can avoid repeating them. You can also mark a sub-plan with .cache() to say "compute this once and reuse it".

Checking a pipeline before running it

Because a LazyFrame knows its schema, you can validate a pipeline without computing it. This is a cheap, fast test.

Python Schema checks as tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If a step refers to a column that does not exist, the error appears when the plan is resolved (at collect_schema() or collect()), not after minutes of data processing.

Data bigger than memory: the streaming engine

The normal (in-memory) engine loads the data it needs entirely into RAM. The streaming engine instead processes data in batches: read a chunk, process it, keep only the running result, then read the next. Operations such as filters, projections, many aggregations, and joins can work this way, so a 100 GB file can be summarized on a laptop.

On a normal installation you ask for it when collecting:

Python
result = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("status") == "ok")
    .group_by("user_id")
    .agg(pl.col("amount").sum())
    .collect(engine="streaming")
)

And if the result itself is too big for memory, write it straight to disk with a sink, which never builds the whole DataFrame:

Python
(
    pl.scan_csv("huge.csv")
    .filter(pl.col("country") == "NO")
    .select("id", "amount")
    .sink_parquet("norway.parquet")      # also sink_csv, sink_ndjson
)

The in-browser Polars build has no streaming engine, no file scans that execute, and no Parquet writer, so the two snippets above are read-only here. The ideas behind them are runnable, as the next example shows.

Why streaming works: partial aggregates

Streaming relies on the fact that many results can be computed chunk by chunk and merged. A sum is the sum of chunk sums; a count is the sum of chunk counts; a mean needs the sum and the count. Here is the idea done by hand with slice:

Python Aggregating in chunks by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The streaming engine does this for you, much more efficiently, for the operations it supports. Operations that need all the data at once (some sorts, some window functions, anything with a Python function) can force Polars to fall back to the in-memory engine for that part of the plan; explain and the documentation show which ones.

Practical guidance

  • Keep your first draft lazy from the start: scan_*, transformations, then collect().
  • Use sink_* when the result is big, and collect(engine="streaming") when the input is big but the result is small.
  • Select columns and filter rows as early as you can in the code. The optimizer does it too, but it makes the pipeline easier to read.
  • Test pipelines on a small sample first: lf.head(1000).collect() is cheap thanks to slice pushdown (on a normal install).
  • Save intermediate results as Parquet (compressed, typed, columnar) instead of CSV. The IO chapter covers formats.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does LazyFrame.pipe(func, 5) do?

    pipe passes the frame as the first argument to your function and returns what it returns, enabling readable chains of functions.
  2. 2. Why use pl.collect_all([q1, q2, q3])?

    collect_all executes multiple LazyFrames at once and returns a list of DataFrames.
  3. 3. When should you use a sink (such as sink_parquet) instead of collect()?

    A sink writes the result to disk in batches without building the full DataFrame in memory.
  4. 4. Which statement about splitting a lazy pipeline into several Python functions is true?

    Functions only build the plan. Nothing runs until collect(), and the optimizer sees the entire combined plan.

Summary

  • Write pipeline steps as functions from LazyFrame to LazyFrame and chain them with pipe; reusable expressions return pl.Expr.
  • pl.collect_all runs several queries together; .cache() reuses a sub-plan; collect_schema() validates a pipeline cheaply.
  • The streaming engine (collect(engine="streaming")) and sink_* handle data bigger than memory by working in batches.
  • Streaming works because many aggregations can be computed chunk by chunk and merged.
  • The browser build runs the in-memory engine only, so streaming and sinks are read-only here.

Next: Joins in Polars