Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Migrating from pandas

Migrating a pandas Pipeline Step by Step

Port a real pandas cleaning and reporting script to Polars step by step, check that the results match, convert it to a lazy query, and plan a safe gradual migration.

You do not have to rewrite everything on day one, and you should not. The safest migrations move one pipeline at a time, prove that the new output equals the old one, and only then switch over. This lesson walks through a realistic example: a small pandas script that loads orders, cleans them, adds derived columns, and builds a monthly report. We port it to Polars in steps, compare the results at each stage, and end with a lazy version and a checklist for your own code.

All examples import numpy, pyarrow and pandas before polars, which this browser runner needs. The data is built from text in memory because the sandbox has no files; on a real project you would replace io.StringIO(text) with a file path.

The starting point: a pandas script

Raw data from a shop: some missing quantities, inconsistent country spelling, text dates, and a "cancelled" flag.

Python The original pandas pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is typical pandas: assign to columns, filter by boolean mask, groupby with as_index=False, sort_values.

Step 1: read and clean

Polars does reading and cleaning with with_columns. Each pandas assignment becomes a named expression:

pandas line Polars expression
read_csv(..., parse_dates=["date"]) read_csv(..., try_parse_dates=True)
.str.strip().str.upper() .str.strip_chars().str.to_uppercase()
.replace({"GERMANY": "DE"}) .replace({"GERMANY": "DE"})
.fillna(1).astype(int) .fill_null(1).cast(pl.Int64)
Python Clean the data in Polars

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check the details. Rows with qty missing now have a 1; the cancelled order is gone; the date column has the real Date type. The pandas qty column was read as float because of the missing value, and only became an integer after fillna. Polars read it as an integer with a null and kept it that way (the fill_null then removed the null).

Step 2: derived columns and the report

New columns are more expressions. The grouped report uses group_by(...).agg(...) and sorts at the end, because group order is not guaranteed.

Python The complete Polars version

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The whole pipeline is one chain, with no intermediate variable being mutated. Reading top to bottom, every step says what it does.

Step 3: prove the two are equal

Never trust a migration by eye. Run both versions on the same input and compare. polars.testing.assert_frame_equal raises with a clear message if the frames differ in values, order, dtypes or column names; convert the pandas result with pl.from_pandas first (and tell it how to treat dtypes with check_dtypes=False when pandas and Polars choose different widths).

Python Compare pandas and Polars results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice check_dtypes=False. The values are the same, but pl.len() returns an unsigned 32-bit integer while pandas counts in 64-bit. Decide deliberately whether such differences matter downstream, and cast to the type your consumers expect (.cast(pl.Int64)) if they do.

Step 4: make it lazy

Change read_csv to scan_csv (for a real file), add .collect() at the end, and the same chain becomes a lazy query that the optimizer rewrites: only the needed columns are read and the filter runs as early as possible. In this runner we keep the in-memory data, so we use .lazy() on the frame instead; everything after it is identical.

Python The lazy version and its plan

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plan shows the unneeded columns (product) are never carried through and that the filter sits near the data. The optimizer also let us drop the separate cast: we only needed qty inside the multiplication.

A safe migration plan

  1. Pick one pipeline with clear inputs and outputs, ideally slow or memory-hungry.
  2. Freeze a sample of the input and the pandas output; keep them as test fixtures.
  3. Port step by step, comparing with assert_frame_equal at each stage (as above).
  4. Convert at the boundary. Use pl.from_pandas(...) and .to_pandas() where the rest of the code (plotting, scikit-learn) still needs pandas. Only the heavy middle moves first.
  5. Replace apply and loops with expressions as you go; they are where the biggest speed-ups come from.
  6. Compare time and memory on real-sized data before you declare victory (see fast patterns).
  7. Keep the pandas version for one release cycle as a reference, then delete it.

Not everything must move. If a script handles a thousand rows and is not slow, leaving it in pandas is a perfectly good decision.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the best way to verify that a ported Polars pipeline matches the original pandas one?

    assert_frame_equal compares values, order, names and (optionally) dtypes, and prints a precise diff on mismatch.
  2. 2. Why did the comparison use check_dtypes=False?

    pl.len() returns UInt32 while pandas count is 64-bit. The values match; decide separately whether the width matters.
  3. 3. What is the main change needed to turn the eager Polars chain into a lazy one for a file?

    The expressions are identical; scan_* starts a lazy query and collect() runs the optimized plan.
  4. 4. Which part of a pandas script usually deserves the most careful rewriting?

    Row-wise Python code has no one-to-one translation; replacing it with expressions is also where the big speed-ups come from.

Summary

  • Port one pipeline at a time and keep the old result as a test oracle.
  • Each pandas assignment becomes an expression in with_columns; boolean-mask filters become filter; groupby becomes group_by, followed by an explicit sort.
  • Compare with polars.testing.assert_frame_equal, deciding consciously about dtype differences.
  • Lazy mode is the same chain with scan_* and collect().
  • Convert at boundaries (from_pandas, to_pandas) while migration is in progress.

Next: Project: Sales Report