Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท DataFrames and Series

Filtering and Sorting

Keep and order rows in Polars with filter, boolean expressions, is_in, is_between, sort with multiple keys, unique, duplicates, sample and top_k.

Most analysis begins by narrowing the rows to the ones you care about and putting them in a sensible order. Polars does both with a small set of methods built on expressions: filter for conditions, sort for ordering, and a few helpers for duplicates and samples.

Python The staff table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Filtering with boolean expressions

A filter takes an expression that evaluates to Boolean, one value per row, and keeps the rows where it is True.

Python Comparison filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Several conditions can also be passed as separate arguments to filter; they are combined with AND, which often reads better than a chain of &:

Python
staff.filter(pl.col("team") == "dev", pl.col("years") > 3)

Keyword form works too: staff.filter(team="dev") is shorthand for equality on a column.

Useful condition helpers

Expression Meaning
col.is_in([...]) Value is in a list
col.is_between(lo, hi) Inclusive range by default (closed="both")
col.str.contains("x") Text contains a pattern
col.str.starts_with("A") Text prefix
col.is_null() / is_not_null() Missing or not
col.is_duplicated() / is_unique() Appears more than once, or only once
col.is_first_distinct() First appearance of each value
Python Helpers in filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sorting

sort accepts one or several columns, and descending can be a single flag or one flag per column.

Python Sort by one or several keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sorting is stable only if you ask: pass maintain_order=True when ties must keep their original order. Nulls go first by default; use nulls_last=True to move them to the end.

top_k(n, by=...) and bottom_k find the n largest or smallest rows without a full sort. They are faster than sort(...).head(n) for large data.

Duplicates and unique rows

Python unique and is_duplicated

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

unique(subset=...) considers only some columns when deciding what counts as a duplicate and keep picks which copy survives ("first", "last", "any", "none"). Without maintain_order=True, the output order is not specified.

Random samples

sample draws rows at random; set seed for repeatable results.

Python Samples and shuffles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Filtering is not row-by-row

A common beginner pattern is a Python loop that tests each row. Do not. A filter is evaluated for the whole column in one fast pass, and in lazy mode Polars can move it as close to the data source as possible. Another trap is comparing a column to a missing value: pl.col("x") == None is not the way to test for null; use .is_null(). In Polars, comparing anything to null gives null, and filter drops rows where the condition is null. The next lesson, Null and NaN Values, explains this.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you keep rows where team is 'dev' AND years is greater than 5?

    Use & between boolean expressions and parenthesize each comparison. Python's and does not work on expressions.
  2. 2. Which call sorts by team ascending, then by salary descending?

    sort accepts several keys and a list of descending flags, one per key. Sorting twice would discard the first order.
  3. 3. What does df.unique(subset='user', keep='first', maintain_order=True) return?

    unique with subset deduplicates on that column; keep picks which row survives and maintain_order preserves the original order.
  4. 4. A filter condition evaluates to null for some rows. What happens to them?

    filter keeps only rows where the condition is True. Null (unknown) is not True, so those rows are dropped.

Summary

  • filter takes Boolean expressions combined with &, |, ~; several arguments are ANDed.
  • Helpers such as is_in, is_between, str.contains and is_null express common conditions.
  • sort takes multiple keys, per-key descending, nulls_last and maintain_order; top_k avoids a full sort.
  • unique, is_duplicated and sample handle duplicates and random subsets.

Next: Null and NaN Values