Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Performance

Data Types and Memory

Shrink Polars DataFrames and speed up queries by choosing smaller integer and float types, Categorical and Enum for repeated text, and measuring size with estimated_size.

Every column in a Polars DataFrame has one strict data type, and that type decides how much memory the column needs and how fast operations on it run. Polars picks safe defaults (Int64, Float64, String), but you often know more than the defaults: a column of ages never needs 64 bits, and a column holding only four city names should not store the text "Kyoto" fifty thousand times.

This lesson shows how to measure memory, choose smaller types, and use Categorical and Enum for repeated text. These are the cheapest performance wins in Polars: no new algorithm, just a better description of your data.

Measuring size

df.estimated_size() returns the approximate heap size in bytes (pass "kb", "mb" or "gb" for a unit). It is an estimate, especially for strings, but it is perfect for comparing "before" and "after".

Python How big is a DataFrame?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Integers default to Int64 (8 bytes per value), floats to Float64 (8 bytes), and strings are variable-length, so 100,000 short city names cost more than the numbers do.

The numeric types

Type Bytes Range
Int8 / UInt8 1 -128..127 / 0..255
Int16 / UInt16 2 -32,768..32,767 / 0..65,535
Int32 / UInt32 4 about +-2.1 billion / 0..4.29 billion
Int64 / UInt64 8 very large
Float32 4 about 7 significant digits
Float64 8 about 15 significant digits

Casting down is pl.col("age").cast(pl.UInt8). By default cast is strict: if a value does not fit, Polars raises an error instead of silently wrapping around.

Python Strict casts protect you

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Be careful with small integer types in arithmetic: sums and products can overflow the type. Polars aggregations such as sum() widen small integers automatically, but things like pl.col("a") * pl.col("b") on two Int8 columns stay Int8. When in doubt, cast to a wider type first.

The helper Series.shrink_dtype() picks the smallest integer type that fits the actual values, which is handy for exploration:

Python Shrink to the smallest type

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For Float64 to Float32, remember you lose precision (about 7 digits). That is fine for model features or coordinates, but not for money, where you should use integers of cents or the Decimal type.

Categorical and Enum for repeated text

A column of country names, product codes or statuses repeats a handful of strings millions of times. Categorical stores each distinct string once in a dictionary and keeps a small integer code per row. Comparisons, joins, group-bys and sorts on the codes are much faster than on strings.

flowchart LR
    S["String column<br/>Oslo, Lima, Oslo, Kyoto, Lima"] --> D["dictionary<br/>0=Oslo, 1=Lima, 2=Kyoto"]
    S --> C["codes<br/>0, 1, 0, 2, 1"]

A categorical column stores a small dictionary plus one integer per row.

Python String to Categorical

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Enum is the strict cousin: you declare the allowed values up front. It prevents typos (a value outside the list is an error), gives a meaningful sort order (the order you declared, not alphabetical), and lets different frames share the same categories without extra work.

Python Enum: fixed categories, custom sort order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When should you use which?

  • Enum when the set of values is known and fixed (statuses, weekdays, sizes, enums from your application).
  • Categorical when the values are open-ended but repeated (cities, user agents, product names).
  • String when most values are unique (names, addresses, free text, IDs): a dictionary would be as big as the column.

A useful rule of thumb: if the number of distinct values is below about 5 to 10% of the number of rows, try a categorical type.

Other dtype habits

  • Dates: store real Date / Datetime columns, never strings; they are smaller, sortable and carry the .dt methods.
  • Booleans use one bit per value; flags stored as 0/1 integers waste space.
  • Nulls are tracked in a separate bitmask and cost almost nothing, so prefer null to sentinel values such as -1.
  • Decimals (pl.Decimal) give exact base-10 arithmetic for money; check the current status in the docs because the type has been evolving.
  • Read files with the right types from the start: schema_overrides in read_csv, or use Parquet, which stores the types.

Applying types to a whole frame at once

DataFrame.cast accepts a mapping from column names to types, and pl.selectors pick columns by dtype:

Python Cast many columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens by default when you cast a value that does not fit the target integer type, e.g. 300 to UInt8?

    cast is strict by default. Pass strict=False if you want non-fitting values to become null.
  2. 2. Which type suits a column that has only the values 'draft', 'review' and 'published' in a defined workflow order?

    Enum declares the allowed values and a sort order, and rejects anything else.
  3. 3. When is Categorical a poor choice?

    With mostly unique values the dictionary is as large as the data, so you gain nothing and add overhead.
  4. 4. Which method reports the approximate memory of a DataFrame?

    estimated_size(unit) returns the approximate heap size; memory_usage is a pandas method.

Summary

  • Measure with estimated_size(); compare before and after.
  • Downsize numbers with cast; casts are strict, so a value that does not fit raises an error.
  • Categorical for repeated open-ended text, Enum for a fixed list with a meaningful order.
  • Real Date/Datetime types, booleans and nulls beat strings, integers and sentinels.
  • Choose types at read time (schema_overrides) or store typed formats such as Parquet.

Next: Query Plans and Optimizations