Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Performance

Fast Patterns and Common Slow Spots

Write fast Polars code. Avoid row loops and map_elements, batch work into one with_columns, use when/then, joins instead of lookups, and know streaming and thread settings.

Polars is fast if you let it do the work. Its speed comes from running compiled Rust code over whole columns, in parallel, with an optimizer in front. Every time your code drops back into a Python loop or a per-row Python function, you leave that fast path and pay Python's per-row cost instead.

This lesson is a field guide: the patterns that keep you on the fast path, the traps that silently slow you down, and a few settings for big data. Timings in the examples are measured live in your browser, so your numbers will differ from run to run, but the ranking (fast, slower, slowest) stays the same.

The speed ladder

From fastest to slowest, three ways to double a column of numbers:

  1. A native expression: pl.col("v") * 2. Compiled code, vectorized, parallel on real hardware.
  2. map_elements with a Python function: Polars calls your function once per value.
  3. A Python loop over rows: for row in df.iter_rows(named=True).
Python Expression versus map_elements versus loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three produce the same numbers, but the expression is faster, and run it a second time (the first call includes one-time warm-up) to see the gap widen. On real hardware with several cores and data sets of tens of millions of rows, the difference is the difference between a second and many minutes.

map_elements prints a PolarsInefficientMapWarning when it recognizes that a native expression could do the same job, and it often suggests the replacement. Read those warnings: they are free performance advice.

Replace Python logic with expressions

Most "I need a Python function" cases have a native form:

Python habit Polars expression
if x > 10: "big" else "small" pl.when(pl.col("x") > 10).then(pl.lit("big")).otherwise(pl.lit("small"))
s.upper(), s.strip(), "a" in s .str.to_uppercase(), .str.strip_chars(), .str.contains("a", literal=True)
x in [1, 2, 3] pl.col("x").is_in([1, 2, 3])
dict.get(x, default) pl.col("x").replace_strict(mapping, default=...) or a join
sum(row[a] * row[b] ...) (pl.col("a") * pl.col("b")).sum()
datetime.strptime(...) .str.to_datetime(format)
nested if/elif chain chained pl.when().then().when().then().otherwise()
Python when/then instead of a Python function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you truly need arbitrary Python (a third-party library, odd parsing), prefer map_batches, which passes a whole Series to your function once, over map_elements, which calls it once per value. Inside the function, use vectorized NumPy or Polars calls.

One with_columns, not ten

Each expression in a single with_columns or select runs independently and in parallel, and the optimizer can see them all at once. A chain of ten separate .with_columns(...) calls also works (the lazy optimizer often merges them), but a single call with ten expressions states the independence clearly. Expressions in the same call cannot see each other's new columns, so dependent steps need a second call.

Python Independent expressions in one call

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lookups: join, do not loop

Looking up a value for every row in a dictionary-like table is a join, never a Python loop with a filter inside.

Python Join instead of repeated filtering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Python loop builds 2,000 tiny queries; the join does the whole job once with a hash table.

Habits that keep memory and time down

  • Select only what you need. In an eager pipeline call select early; in a lazy one the optimizer does it for you.
  • Filter early, before joins and sorts.
  • Prefer pl.len() over pl.col("x").count() when you just want the number of rows.
  • Avoid to_pandas() / to_list() in the middle of a pipeline; each one copies the data out of Polars' fast representation.
  • Avoid appending rows in a loop. Gather rows in a Python list, then build one DataFrame; or use pl.concat([...]) once.
  • Cast to smaller types (see data types and memory) and use Categorical for repeated strings.
  • Sort once. If a column is already sorted, tell Polars with set_sorted("col"), so operations like group_by_dynamic, join_asof and search_sorted can skip work. Telling it wrongly gives wrong answers, so do it only when you are certain.
  • unique and n_unique are cheaper than sorting and slicing; top_k and bottom_k are cheaper than a full sort when you only need a few rows.
Python top_k and set_sorted

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Big data settings (read-only)

These matter on real hardware and cannot be demonstrated in a single-threaded browser build:

Python
import os
os.environ["POLARS_MAX_THREADS"] = "4"   # before importing polars: cap the thread pool

import polars as pl
print(pl.thread_pool_size())

# process data larger than memory with the streaming engine
result = (
    pl.scan_parquet("big/*.parquet")
    .group_by("user")
    .agg(pl.col("amount").sum())
    .collect(engine="streaming")
)
  • Threads: Polars uses all cores by default. Set POLARS_MAX_THREADS before the import when sharing a machine.
  • Streaming engine: collect(engine="streaming") processes data in batches, so the whole data set need not fit in RAM. Not every operation supports streaming yet; Polars falls back to the in-memory engine for the rest, and explain(engine="streaming") shows what will happen.
  • Write results with sink_parquet instead of collecting a huge frame in memory first.
  • Do not fork after using Polars (for example with multiprocessing fork): use the spawn start method, because the thread pool does not survive a fork.

Measure before you tune

Optimize what the profile shows, not what you suspect. Use lazy().profile() (previous lesson), time whole steps with time.perf_counter(), and repeat measurements: the first run includes warm-up. A simple, expression-based query is usually fast enough; clever tricks matter only on the one step that dominates.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which is normally the fastest way to compute a new column v * 2?

    A native expression runs as compiled, vectorized code. The others call Python for each row.
  2. 2. You need 'big'/'small' labels based on a threshold. Which approach should you use?

    when/then/otherwise is the vectorized conditional, and the optimizer can see through it.
  3. 3. How should you attach a tier from a small lookup table to a large orders table?

    A join does the whole lookup at once with a hash table; it is faster and clearer.
  4. 4. When is map_batches preferable to map_elements?

    map_batches calls your function once with a Series, so a vectorized NumPy or Polars call inside it stays fast.

Summary

  • Keep work inside native expressions: they are vectorized, optimizable and parallel on real hardware.
  • Replace Python conditionals with when/then/otherwise, string and date work with namespaces, lookups with joins.
  • Avoid iter_rows and map_elements for computation; if unavoidable, use map_batches.
  • Filter and select early, cast to small types, build frames once, sort once.
  • On real hardware: POLARS_MAX_THREADS, collect(engine="streaming") and sink_* handle large data.
  • Measure with profile() and timers before optimizing.

Next: pandas to Polars: The Cheat Sheet