Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 64 ยท Data Types

Categorical Data

Use the pandas category dtype for repeated labels: save memory, define an order, rename and reorder categories, and handle unobserved categories in groupby.

Many columns hold only a handful of distinct values repeated over and over: shirt sizes, countries, order statuses, survey answers. Stored as plain text, each row keeps its own copy of the string. pandas offers a better way: the category dtype stores each distinct label once and represents every row as a small integer code that points to it.

That gives you three things. First, memory savings on large columns. Second, a defined order (small < medium < large) that sorting and comparisons respect. Third, a fixed list of allowed values, which catches typos and lets you show categories that have no rows. This lesson builds on Data Types and Conversion.

Creating a categorical column

Convert with astype("category"). The distinct values become the categories, and .cat.codes shows the integer stored for each row.

Python From text to category

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

flowchart LR
    A["Rows<br/>small, large, small"] --> B["Codes<br/>1, 0, 1"]
    B --> C["Categories<br/>0: large<br/>1: small"]

Each row stores a small integer code; the distinct labels live once in the categories list.

By default the categories are sorted alphabetically, which is rarely the order you mean (large < medium < small is nonsense for sizes). The next section fixes that.

Memory savings

The saving grows with the number of rows and shrinks with the number of distinct values. A column of 100,000 values drawn from three labels shows the effect clearly.

Python Measuring memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The codes use the smallest integer type that fits (int8 for up to 127 categories). A category column with thousands of distinct values, or one where nearly every value is unique, saves little or nothing and is slower to build. Memory Usage and Smaller Dtypes measures more cases.

Ordered categories

Declare a CategoricalDtype to choose the categories, set their order and make the order meaningful with ordered=True.

Python Sorting and comparing with an order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because sorting follows the categories, sort_values() on an ordered column gives a natural result, and comparisons like sizes >= "L" become valid filters. A value that is not one of the categories is not allowed: converting it gives a missing value.

Python Values outside the categories

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This strictness is a feature. A typo like "mediun" becomes visible as a missing value instead of quietly creating a fourth size.

The .cat accessor

The .cat accessor manages categories without touching the data:

Method What it does
rename_categories(...) change labels (a dict or list)
reorder_categories([...]) change the order (must list all existing categories)
set_categories([...]) replace the whole set; values outside become missing
add_categories([...]) allow extra values (with zero rows for now)
remove_unused_categories() drop categories no row uses
as_ordered(), as_unordered() switch ordering on or off
codes, categories, ordered inspect
Python Renaming, reordering and adding categories

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Renaming changes labels in one step for all rows, which is much cheaper than a replace on text. Note that value_counts() on a categorical shows every category, including those with zero rows; that is useful for reports that must always list the same rows, such as every size or every weekday.

Categories and groupby

Grouping by a categorical column groups by its categories. In pandas 3, only categories that actually occur are shown by default (observed=True). Pass observed=False to keep unobserved categories in the result, with zero or missing values.

Python Unobserved categories in groupby

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keeping every category (observed=False) is what you want for fixed reports and charts where "no orders for size M" is information. The default is better when you only care about what happened. Grouping is covered in groupby Basics.

Where categories come from

Several tools produce categoricals directly. pd.cut and pd.qcut (see Binning and Discretising) return ordered categories of intervals, and read_csv(..., dtype={"col": "category"}) loads a column as a category from the start, saving memory before the full text column ever exists.

When to use category (and when not to)

Use it when:

  • the column has few distinct values compared with the number of rows;
  • you need a custom sort order, or comparisons such as "at least medium";
  • you want every category shown in counts and group results;
  • you want a fixed vocabulary so typos become missing values.

Skip it when:

  • values are mostly unique (ids, names, free text);
  • the table is small and memory is not a concern;
  • you will often combine columns with different category sets: operations such as concatenating two categoricals with different categories fall back to plain text, which removes the benefit.

Tip: Convert to category after cleaning. Whitespace and case variants like "Small", "small " and "SMALL" would each become their own category.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a category column store for each row?

    Each row keeps an integer code, and the distinct labels are stored once in the categories list. That is where the memory saving comes from.
  2. 2. How do you make 'S' < 'M' < 'L' sort and compare in that order?

    A plain category sorts alphabetically. Declaring the categories in your order with ordered=True makes sorting, min/max and comparisons follow it.
  3. 3. You assign a value that is not among the categories. What happens?

    Assignment raises a TypeError. Use cat.add_categories first. This guards against typos.
  4. 4. When is a category column a poor choice?

    With mostly unique values there is little repetition to share, so a category saves almost no memory and costs extra to build.

Summary

  • astype("category") stores each distinct label once plus integer codes per row; large repetitive columns shrink dramatically.
  • pd.CategoricalDtype([...], ordered=True) defines the allowed values and their order, so sorting, min/max and comparisons work.
  • Values outside the categories become missing on conversion and raise on assignment.
  • The .cat accessor renames, reorders, adds and removes categories without touching rows.
  • value_counts() and groupby(..., observed=False) can show categories with zero rows; pandas 3 defaults to observed=True.
  • Skip it for mostly unique columns, and clean text before converting.

Next: Creating and Changing Columns.