Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 64 ยท Performance and Scale

Memory Usage and Smaller Dtypes

Measure how much memory a DataFrame uses and shrink it with smaller numeric dtypes, category columns, sparse arrays, usecols and chunked reading.

A pandas DataFrame lives entirely in memory. A CSV file of 500 MB can easily need several gigabytes once loaded, because every number gets a default 64-bit slot and every text value becomes a separate Python object. When a program becomes slow or crashes with a memory error, the cause is often just the wrong dtypes.

This lesson shows how to measure memory, how to pick smaller dtypes safely, and how to avoid loading data you do not need. The first rule comes before every technique: measure first, optimise second. A table that fits comfortably in memory does not need any of this.

Measuring memory

df.memory_usage() returns the bytes used by each column and the index. By default it counts only the array itself. For text columns that hides the real cost, so use deep=True, which also counts the Python objects behind the values.

Python Where do the bytes go?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numeric columns use exactly rows x bytes per value: 50,000 rows of an 8-byte integer is 400,000 bytes. The country column holds only five different values, yet it needs far more memory than a number column, because each row refers to a text object. df.info(memory_usage="deep") prints the same totals in one summary and is a quick way to look at the whole frame (its exact numbers vary between machines, so we do not print it here).

flowchart TD
    A["Measure with memory_usage(deep=True)"] --> B{"Which column is big?"}
    B -->|"numbers (int64, float64)"| C["downcast to int8/int16/int32 or float32"]
    B -->|"text with few distinct values"| D["convert to category"]
    B -->|"mostly zeros or missing"| E["use a sparse dtype"]
    B -->|"columns you never use"| F["leave them out with usecols"]
    B -->|"file larger than memory"| G["read in chunks"]
    C --> H["Measure again"]
    D --> H
    E --> H
    F --> H
    G --> H

A repeatable routine: find the heavy column, apply the matching technique, measure again.

Smaller numeric dtypes

Integers default to int64 and floats to float64, which can hold huge values. If an age column only holds 18 to 89, an int8 (range -128 to 127) is enough and uses an eighth of the memory. pd.to_numeric(..., downcast=...) picks the smallest type that fits the actual values:

Python Downcasting numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

age becomes int8, visits becomes int32 (its values go above 32,767, which is too much for int16), and rating becomes float32. Integers are exact after downcasting. Floats lose precision: a float32 keeps about seven significant digits, which is fine for a rating, but not for money totals or scientific values that need full accuracy. Also remember that small integer types overflow. Adding to an int8 column can wrap around silently, so downcast values that you only store and read, or convert back before heavy arithmetic.

category for repeated text

A column with many repeats, such as country, status or product type, can be stored as a category: pandas keeps each distinct value once and stores a small integer code per row. See Categorical Data for the full feature set.

Python category versus plain text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The saving is largest when the number of distinct values is small compared with the number of rows. If almost every value is unique, such as an email address, a category column can be larger than plain text, so test it. Since pandas 3, text columns already use the dedicated str dtype by default, which is cleaner and often smaller than the old object columns, but it does not de-duplicate values the way a category does.

Sparse data

If a column is mostly one value, usually zero or missing, a sparse array stores only the other values and their positions. This is common after one-hot encoding (see Reshaping), where each row has a single 1 among many zeros.

Python Sparse columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only 0.5 percent of the values are non-zero, so the sparse version is roughly a hundred times smaller. Arithmetic still works. Sparse types pay off only for really sparse data; with a density above about 30 percent they are slower and not smaller.

Loading less: usecols and dtype

The best saving is data you never load. read_csv can skip columns with usecols, and set compact dtypes at the start with dtype, so the big 64-bit version never exists:

Python usecols and dtype in read_csv

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The note column has no value for the analysis, so skipping it saves the most. usecols also makes parsing faster, because pandas ignores the other fields. More options for reading are in Reading CSV Data.

Reading in chunks

If the file is larger than memory, process it piece by piece with chunksize. read_csv then returns an iterator of DataFrames. You keep only a running result, not the data:

Python Aggregating a file in chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each chunk has up to 8,000 rows. Per-chunk sums combine exactly because a sum of sums is a sum. Averages need a sum and a count kept separately, and anything that needs all rows at once, such as a median or a sort, does not fit this pattern. The next-but-one lesson, When Data Gets Big, covers tools for those cases.

Measure before you optimise

Memory tuning costs readability. Downcast types cause overflow surprises, categories add behavior you must remember, and sparse arrays are slower for many operations. Use this order:

  1. Check whether the data really is a problem: df.info(memory_usage="deep").
  2. Load less: usecols, nrows, filtering while reading.
  3. Fix the biggest column first, usually text.
  4. Re-measure, and stop when the table is comfortably small.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do you pass deep=True to memory_usage() for a DataFrame with text columns?

    Without deep=True pandas only counts the array of references, which hides most of the real cost of text.
  2. 2. Which column is the best candidate for the category dtype?

    Categories save the most when there are few distinct values compared with the number of rows.
  3. 3. What is a risk of downcasting an integer column to int8?

    int8 only holds -128 to 127. Arithmetic that goes outside that range wraps around instead of failing.
  4. 4. How do you process a CSV file that is larger than memory?

    With chunksize, read_csv yields DataFrames of N rows. You aggregate each one and keep only the running result.
  5. 5. Which should you do first when a pandas script uses too much memory?

    Measuring shows where the memory goes, so you spend effort on the column that matters.

Summary

  • Use memory_usage(deep=True) or info(memory_usage="deep") to see where the bytes go.
  • pd.to_numeric(downcast=...) or dtype= picks the smallest numeric type that fits; watch for overflow and float precision.
  • category stores repeated text as small codes and is best for few distinct values.
  • Sparse arrays store only the non-default values of mostly empty columns.
  • Load less with usecols, dtype and nrows; process huge files with chunksize.
  • Measure first, optimise the heaviest column, then measure again.

Next: Writing Fast pandas Code