Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 64 ยท Transforming Data

Binning and Discretising

Learn pandas binning with cut and qcut: custom edges and labels, open and closed bin edges, equal-width versus equal-count bins, and counting values per bin.

Binning (also called discretising or bucketing) turns a continuous number into a small set of ranges: ages into "minor / adult / senior", prices into "cheap / mid / premium", scores into letter grades. You bin data to summarise it, to draw a histogram or bar chart, to group rows for groupby, or to build features for a model.

pandas has two main functions. pd.cut makes bins from edges you choose (or from equal-width ranges). pd.qcut makes bins that each hold about the same number of rows. This lesson covers both, the edge rules that cause off-by-one surprises, and a NumPy alternative.

cut: bins from edges

Give pd.cut a Series and a list of edges. Three edges make two bins, four edges make three bins, and so on. By default each bin is open on the left and closed on the right: (18, 65] excludes 18 and includes 65.

flowchart LR
    E0["edge 0"] --- B1["(0, 18]"] --- E1["edge 18"] --- B2["(18, 65]"] --- E2["edge 65"] --- B3["(65, 120]"] --- E3["edge 120"]

N edges make N - 1 bins. Each bin includes its right edge and excludes its left edge by default.

Python Edges, labels and the right= switch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the first two lines. The age 18 lands in (0, 18] by default, but in [18, 65) when you pass right=False. Pick the convention that matches your rule ("adult means 18 or older" is right=False) and write it down, because every boundary value depends on it.

The result is a category column (see Categorical Data), so it stores each label once and keeps your label order. Passing labels replaces the interval text with names; there must be exactly one label per bin.

Values outside the edges

A value that falls outside the outermost edges gets NaN, not an error. The lowest edge is also excluded by the default rule, so a value equal to the first edge goes missing too. include_lowest=True fixes that.

Python Values that fall outside the bins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Silent NaN is the most common binning bug. After a cut, check labels.isna().sum(), or make the outer edges wide enough, for example [-np.inf, 18, 65, np.inf]. Infinite edges catch every value, and you can still give the bins readable labels.

Letting pandas choose equal-width bins

If you pass a number instead of a list, cut splits the range between the minimum and maximum into that many equal-width bins:

Python
pd.cut(ages, bins=3)   # three bins of equal width over the data range

The edges are computed from the data (and widened by about 0.1% so the minimum is included), so they look like (2.902, 35.667]. For a report, pass your own round edges instead.

qcut: bins with equal counts

Equal-width bins can be badly unbalanced when the data is skewed: most values pile up in the first bin. pd.qcut cuts at quantiles instead, so every bin gets roughly the same number of rows. Pass the number of bins (q=4 gives quartiles) or a list of quantile fractions between 0 and 1.

Python cut versus qcut on skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The equal-width version puts 707 of the 1,000 people into the first bin, which tells you little. The quartile version gives exactly 250 per bin. The last call uses custom quantiles: the lower half is "base", the next 40% "upper", and the top 10% "top".

qcut fails when the data has so many repeated values that two quantile edges are equal, because a bin of zero width is impossible. Pass duplicates="drop" to merge those bins:

Python qcut with repeated values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here the repeated 1s make four requested bins collapse into a single one. When bins are dropped you get fewer than you asked for, so count what you actually received before you attach labels.

Counting values per bin

value_counts can bin for you with its bins= argument, which gives a quick histogram table. It is handy for exploring, but it does not let you set labels. Combine cut with value_counts (and sort=False to keep the bins in order) when you want control.

NumPy alternatives: digitize and histogram

NumPy offers np.digitize(values, edges), which returns the bin number for each value (1 for the first bin, and 0 or len(edges) for values outside). It follows its own edge rule: by default a bin includes its left edge, the opposite of cut. right=True flips it. np.histogram(values, bins=edges) returns the count per bin and uses left-closed bins too, except that the last bin is closed on both ends.

Python cut, digitize and histogram side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the score 50 and the score 100. cut puts 50 in the first bin (0, 50] and 100 in the last. np.digitize by default puts 50 in the second bin and gives 100 the number 4, "past the last edge". The functions are right at their own rule, so choose one and check your boundary values. For labelled DataFrame columns, cut is usually the better fit.

Bucketing for groups and charts

Because the result is a categorical column, you can group by it directly. Pass observed=True to list only the bins that occur (see groupby Basics); observed=False also shows empty bins, which keeps a chart's axis complete.

Python Group and summarise by bin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a plot, the per-bin counts from value_counts(sort=False) or this groupby output can go straight into a bar chart; see Plotting with pandas.

Keep the bin edges in a named list at the top of your script (AGE_BINS = [0, 18, 65, 120]) so the same definition feeds the cut, the labels, and any chart axis.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. With pd.cut(s, bins=[0, 10, 20]) and default settings, which bin does the value 10 fall into?

    By default bins are closed on the right, so (0, 10] includes 10. With right=False the bins become [0, 10) and [10, 20), and 10 moves to the second bin.
  2. 2. A value lies below the lowest edge passed to pd.cut. What do you get?

    Values outside the outer edges are not in any bin, so they become NaN. Check the count of missing labels after binning, or use infinite outer edges.
  3. 3. Which function makes bins that each contain about the same number of rows?

    qcut cuts at quantiles. cut with a number gives equal-width ranges, which can hold very different numbers of rows.
  4. 4. You pass labels=['low', 'mid', 'high'] to pd.cut with the edges [0, 50, 100]. What happens?

    Two bins need exactly two labels. N edges make N - 1 bins, so a mismatch raises a ValueError.
  5. 5. Why might qcut raise 'Bin edges must be unique'?

    If a single value fills more than a bin's share of the data, neighbouring quantiles coincide. duplicates='drop' merges the empty bins.

Summary

  • pd.cut bins by edges you choose; N edges give N - 1 bins, closed on the right by default (right=False flips it, include_lowest=True keeps the first edge).
  • Values outside the edges become NaN; count them, or use -np.inf and np.inf as outer edges.
  • labels= names the bins (one per bin); labels=False returns bin numbers.
  • pd.qcut makes equal-count bins at quantiles and needs duplicates="drop" when many values repeat.
  • np.digitize and np.histogram are left-closed by default, the opposite of cut, so test boundary values.
  • Binned columns are categoricals that group, count and plot well.

Next: Duplicates and Outliers.