Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 64 ยท Grouping and Reshaping

groupby Basics

Learn pandas groupby from scratch: split-apply-combine, grouping by one or many keys, size versus count, iterating groups, as_index, dropna and observed.

"Total revenue per region." "Average score per class." "How many orders per customer?" Every one of these questions has the same shape: split the rows into groups, calculate something for each group, and put the answers in one table. In pandas that shape is groupby, and it is probably the most important method in the library after loc.

This lesson teaches the core idea, split-apply-combine, then the everyday forms: one key or several, the difference between size and count, looping over groups, and the options (as_index, dropna, observed) that change what the result looks like. The next lesson, agg, transform and filter, builds on this.

Split, apply, combine

df.groupby("region") does not calculate anything yet. It creates a GroupBy object that remembers how the rows are divided. Calling a method on it (.sum(), .mean(), ...) performs the three steps:

  1. Split the rows by the values of the key column.
  2. Apply a function to each group separately.
  3. Combine the per-group results into one Series or DataFrame.
flowchart LR
    A["df<br/>8 rows, 3 regions"] --> S["split by region"]
    S --> N["North rows"]
    S --> So["South rows"]
    S --> E["East rows"]
    N --> F1["sum of revenue"]
    So --> F2["sum of revenue"]
    E --> F3["sum of revenue"]
    F1 --> C["combine:<br/>one row per region"]
    F2 --> C
    F3 --> C

groupby splits the rows by key, applies a function to each piece, and combines the answers into one table.

Your first groupby

The sample data below is a few sales rows. One value is deliberately missing, so you can see how groupby treats it.

Python Sum, mean and size per region

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice three things. The result is indexed by the group key, and the groups are sorted by key (East, North, South), not in order of first appearance. Choosing one column with g["revenue"] gives a Series; choosing a list of columns gives a DataFrame. And the East mean is simply 54.0, because the missing revenue is skipped.

Common reductions you can call directly: sum, mean, median, min, max, std, var, first, last, nunique and count. If you call df.groupby("region").mean() on a frame that still has text columns, pandas 3 raises an error, because it cannot average words. Select the numeric columns first, or pass numeric_only=True.

size versus count

These two look alike and answer different questions:

  • size() counts rows in each group, missing values included.
  • count() counts non-missing values, column by column.
Python size counts rows, count counts values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

East has two rows (size 2) but one known units value (count 1). Also look at the second block: the group East + mug has a single row whose revenue is missing, and its sum is 0.0, not NaN. Summing nothing gives zero, which can fool you into thinking a value was recorded. Use min_count=1 (.sum(min_count=1)) to get NaN for all-missing groups instead.

Several keys

Pass a list to group by combinations. The result has a MultiIndex with one level per key (see MultiIndex):

Python
df.groupby(["region", "product"])["revenue"].sum()

You can turn the index levels back into ordinary columns with reset_index(), or avoid the MultiIndex by using as_index=False (next section).

Iterating over groups

A GroupBy object is iterable. Each step gives you the key and the sub-DataFrame. Use it for quick checks, or when each group needs custom handling such as saving a file per group.

Python Loop over groups and pick one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Looping is for inspection. If you find yourself computing a number per group inside the loop, there is nearly always a built-in method that does it in one line and much faster.

The options that change the result

as_index

By default the group keys become the index of the result. With as_index=False they stay as a normal column and the result has a plain 0, 1, 2... index. This is convenient when you plan to merge or plot the result.

sort

Groups are sorted by key by default. sort=False keeps them in order of first appearance, which can be slightly faster and is useful when order matters. To order by the result, sort the output: .sort_values(ascending=False).

dropna

Rows whose key is missing are dropped by default: they belong to no group. That can silently lose data, so check the totals. dropna=False gives missing keys a group of their own.

observed

If the key is a categorical column (see Categorical Data), pandas can show a row for every possible category, including ones that never occur. observed=True shows only categories that appear in the data; observed=False shows all of them with zero or empty results. Choose observed=False when you want a complete table for a chart or report, and always write the argument explicitly so the behaviour is obvious.

Python as_index, dropna and observed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a groupby surprises you, print df.groupby(key).size() first. It shows how many groups there are, how big each one is, and whether a key you expected is missing.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does df.groupby('region') calculate by itself, with no method after it?

    groupby only prepares the split. The work happens when you call an aggregation such as sum() or size() on the result.
  2. 2. A group has 2 rows, and one value in the 'units' column is missing. What do g.size() and g['units'].count() return for it?

    size counts rows, including missing values. count counts only non-missing values in that column.
  3. 3. Some rows have NaN in the key column. What happens to them in a default groupby?

    By default dropna=True, so rows with a missing key are left out. Pass dropna=False to keep them as their own group.
  4. 4. What does as_index=False do?

    The keys stay as a column, so the result is an ordinary DataFrame that is easy to merge or plot.
  5. 5. When grouping by a categorical column that has an unused category, which setting shows that category with a zero total?

    observed=False includes every category, even those with no rows. observed=True lists only categories that appear in the data.

Summary

  • groupby splits rows by key, applies a function to each group and combines the results.
  • df.groupby("key")["col"].sum() is the usual pattern; groups come back sorted by key.
  • size() counts rows, count() counts non-missing values; an all-missing group sums to 0.
  • Pass a list of keys for a MultiIndex result, or use as_index=False to keep keys as columns.
  • Iterate with for key, part in df.groupby(...) for inspection, and use get_group for a single piece.
  • dropna controls missing keys and observed controls unused categories; set both on purpose.

Next: agg, transform and filter.