Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท DataFrames and Series

Null and NaN Values

Handle missing data in Polars: the difference between null and NaN, detecting, counting, filling and dropping nulls, and how nulls behave in comparisons and sums.

Real data has gaps: a form field left blank, a sensor that failed, a join with no match. Polars represents "no value" in two different ways, and mixing them up causes subtle errors. Null means missing, in any data type. NaN ("not a number") is a floating-point value produced by invalid arithmetic such as 0.0 / 0.0. They are different things, and Polars treats them differently.

Null versus NaN

Python Two kinds of missing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Key rules:

  • Null exists in every column type (strings, dates, booleans, integers). Aggregations such as sum, mean and min skip nulls.
  • NaN exists only in float columns. NaN is a real value, so mean of a column with a NaN is NaN.
  • Integer columns can never hold NaN; they hold null instead.
  • Reading data from pandas or CSV may give NaN where you want null. Convert with fill_nan(None) or nan_to_null=True in some readers.

Detecting and counting nulls

Python Finding nulls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

pl.any_horizontal and pl.all_horizontal combine Boolean columns across a row, which is how you ask "does this row contain any null?".

Filling nulls

fill_null accepts a value, a strategy, or an expression.

Python fill_null

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Strategies include "forward", "backward", "min", "max", "mean" and "zero"/"one". forward is common for time series: carry the last known value. interpolate draws a straight line between known neighbours (numbers only). Choose the fill that matches reality: filling missing salaries with 0 distorts averages, while filling a missing "quantity" with 0 can be exactly right.

Dropping nulls

Python drop_nulls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

DataFrame.drop_nulls removes whole rows. Inside select, pl.col("a").drop_nulls() shortens a single column, so it can produce a column of a different length than the others, which then raises an error if you combine it with full-length columns.

Null in comparisons and logic

Comparing anything with null yields null (unknown), not False.

Python Three-valued logic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Polars follows SQL-style three-valued logic: null AND false is false, null OR true is true, everything else involving null stays null. eq_missing and ne_missing treat two nulls as equal, which is useful in joins and deduplication. When a filter condition is null, the row is dropped; fill the condition with False or True explicitly if you want a different choice.

Null-aware aggregation

Python Aggregations and nulls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sum of an all-null group is 0, but the mean is null: there is nothing to average. pl.len() counts rows, while col.count() counts only non-null values.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statement about NaN and null in Polars is true?

    Null marks missing data in every dtype. NaN is a floating-point value and exists only in float columns.
  2. 2. What does pl.col('x') == None evaluate to for every row?

    Any comparison with null is null. Use is_null() to test for missing values, or eq_missing(None) for null-aware equality.
  3. 3. How do you carry the last known value forward into missing entries?

    The forward strategy replaces each null with the most recent non-null value above it.
  4. 4. What does df.drop_nulls(subset='a') do?

    With subset, only those columns are checked, and whole rows with a null there are removed.

Summary

  • Null is missing data in any type and is skipped by aggregations; NaN is a float value that spreads through math.
  • Detect with is_null, null_count and any_horizontal; convert NaN with fill_nan(None).
  • Fill with values, strategies (forward, backward, mean...), expressions or interpolate; drop with drop_nulls.
  • Comparisons with null give null, and filter drops those rows; use is_null or eq_missing.

Next: Expression Basics