Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Migrating from pandas

Concepts That Differ from pandas

Learn the real differences between Polars and pandas: no index, null instead of NaN, strict types, immutable frames, expression contexts, null ordering and ddof defaults.

Translating method names is the easy part of moving from pandas to Polars. The harder part is a handful of concepts that work differently, because code that "looks right" can then behave differently or fail with an unfamiliar error. This lesson walks through the ones that cause the most surprises, with a runnable comparison for each.

Every runnable example here imports numpy and pyarrow and pandas before polars, which this browser runner needs so that Polars can see them. The pandas version in the runner is 3.x, which already uses copy-on-write and a dedicated string type, so some older pandas habits (chained assignment, SettingWithCopyWarning) no longer apply there either.

1. There is no index

A pandas frame has an index: labels attached to rows that follow filters, joins and sorts, and that many operations align on. Polars has none. Rows are identified by position, and if a label matters, it is just a column.

Python Index labels versus plain columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What this means in practice:

  • No reset_index(), set_index() or MultiIndex gymnastics; there is nothing to reset.
  • Where pandas relies on index alignment, Polars makes you join on a key, which is explicit and visible.
  • Never write code that assumes row labels survive: in Polars they do not exist.

2. null is the one missing value

pandas has several markers: NaN for floats, None in object columns, NaT for datetimes, pd.NA for the nullable types. An integer column that receives a missing value is silently converted to floats. Polars has a single null for every dtype, tracked in a bitmask, and the column keeps its type.

Python Integers stay integers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Careful points:

  • A comparison involving null yields null, and filter drops rows where the condition is null (not true). To keep them, use .fill_null(False) or .eq_missing(...).
  • NaN still exists in Polars as an ordinary float value (0.0 / 0.0); is_null() does not see it, is_nan() does.
  • Sorting: null sorts first in ascending order by default in Polars; pandas puts NaN last. Pass nulls_last=True to match pandas.
Python Where do the missing values sort?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

3. Strict, explicit types

pandas will often quietly coerce between types. Polars refuses ambiguous operations and tells you to cast, which surfaces bugs early.

Python Errors instead of silent coercion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A few more type habits:

  • / is always true division (float); // is floor division. Integer columns do not overflow silently into floats.
  • String columns are really String: there is no object dtype of mixed Python things (a pl.Object exists but is a last resort).
  • Joins require compatible key types; cast first when one side is a string and the other an integer.

4. Frames are immutable

In pandas, df["c"] = ... and df.loc[mask, "c"] = ... change the object you already have. Polars frames do not support that; every method returns a new frame. This removes a whole family of bugs (views versus copies, SettingWithCopyWarning, "did that modify my original?") and enables the optimizer.

Python Nothing changes in place

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Memory is not wasted: the unchanged columns of bigger share their data with df, so with_columns copies only what it creates. Single elements of a Series can be assigned with s[0] = 5, but this is the exception and is rarely the right tool; prefer expressions.

5. Expressions run in a context, in parallel

A Polars expression such as pl.col("a") * 2 is a description; it does nothing until you hand it to a context: select, with_columns, filter, group_by().agg(). All expressions in one context are computed independently, so they cannot see each other's new columns:

Python No peeking at columns created in the same call

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the opposite of pandas, where each assignment line sees the previous one. It also explains why df["a"] (a Series, evaluated right now) and pl.col("a") (an expression, evaluated later in a context) are different things: use pl.col inside queries and df["a"] for quick peeks.

6. Other differences worth knowing

Topic pandas Polars
Truth value bool(df) is an error too, but df.empty helps bool(df) raises; use df.is_empty() or df.height
Standard deviation default ddof=1 ddof=1 (same; pass ddof=0 for the population value)
Row access df[0] Column named 0 or error A one-row frame
Single value df.iloc[0, 0] df.item() or df["a"][0]
Duplicate column names Allowed Not allowed; raises DuplicateError
Group order Sorted keys by default (sort=True) Unspecified unless you sort or set maintain_order=True
Row order after join Mostly left order Left join keeps left order; other joins may not, so sort if needed
Column labels Any hashable Strings only
Lazy mode None .lazy() and scan_* with an optimizer
Plotting df.plot df.plot via hvplot/altair, or pass columns to matplotlib/seaborn

The group-order row deserves emphasis: pandas sorts group keys, so code that accidentally relies on that order will produce shuffled output in Polars. Add .sort(...) after group_by.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In pandas, a filtered frame keeps its original index labels. What is the Polars equivalent?

    Polars deliberately has no index. If a label matters, store it as a normal column and join or filter on it.
  2. 2. You build a Series from [1, None, 3]. What are the dtypes in pandas and Polars?

    pandas upcasts integers to float to hold NaN; Polars keeps Int64 and stores the missing value as null.
  3. 3. Why does df.with_columns(b=pl.col('a') * 2, c=pl.col('b') + 1) fail?

    All expressions in a context see the original frame. Use a second with_columns call for dependent steps.
  4. 4. Where do null values appear after df.sort('a') in Polars by default?

    Polars sorts nulls first in ascending order unless nulls_last=True is passed; pandas puts NaN last.

Summary

  • No index: labels are columns, alignment is a join.
  • One null for every type; integers stay integers; comparisons with null give null.
  • Strict types: cast explicitly; mismatched operations raise errors.
  • Frames are immutable; use with_columns and when/then/otherwise instead of in-place edits.
  • Expressions in one context are independent: chain a second call for dependent columns.
  • Do not rely on group or join output order; sort when it matters.

Next: Migrating a pandas Pipeline Step by Step