Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 49 ยท Shapes and Combining

Unique Values, Sets and Counting

Find distinct values, do set operations and count occurrences with NumPy. Covers unique, isin, union1d, intersect1d, setdiff1d, bincount, histogram and digitize.

Data analysis keeps asking the same questions: Which distinct values are there? How often does each one occur? Which items of this list also appear in that list? Which bin does each measurement fall into? NumPy answers all of them with a small set of functions that work on whole arrays, with no dictionaries or loops.

You will learn np.unique and its three optional extras, the set operations (union1d, intersect1d, setdiff1d) and the fast membership test np.isin, counting integers with bincount, summarizing numbers with histogram and digitize, and a few cumulative counting tricks. The sorting ideas from Sorting and Searching are used behind the scenes, since most of these functions sort first.

np.unique

np.unique(a) returns the sorted distinct values of an array. For multi-dimensional input it flattens first (unless you pass axis). Three keywords add extra outputs, and when you ask for any of them the function returns a tuple:

  • return_counts=True: how often each distinct value occurs.
  • return_index=True: the index of the first occurrence of each value in the original array.
  • return_inverse=True: for every element of the original array, the position of its value in the unique array. So unique_values[inverse] rebuilds the original.
flowchart LR
    O["original<br/>red blue red green blue red"] --> U["unique values<br/>blue green red"]
    O --> I["inverse<br/>2 0 2 1 0 2"]
    U --> R["unique[inverse]<br/>rebuilds the original"]
    I --> R
    O --> C["counts<br/>2 1 3"]

The inverse array is a compact "code" for every element; the unique array is the dictionary.

Python unique with counts, index and inverse

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Using unique with return_counts and argmax is the standard NumPy way to compute the mode of an array. The axis=0 form finds distinct rows, which is useful for removing duplicate records from a table.

Unique returns sorted values, not "in order of first appearance". If you want first-appearance order, combine return_index with sorting: colors[np.sort(first_idx)].

The inverse array is the foundation of encoding categories as integers: strings become small codes that other functions (such as bincount below) can use directly.

Set operations

Treating arrays as mathematical sets takes one function call each. All of them return sorted, duplicate-free arrays.

Function Set operation
np.union1d(a, b) everything in a or b
np.intersect1d(a, b) values in both
np.setdiff1d(a, b) values in a that are not in b
np.setxor1d(a, b) values in exactly one of them
np.isin(a, b) boolean array: which elements of a appear in b

The older name np.in1d was replaced by np.isin, which keeps the shape of the first argument and has an invert option. Use np.isin in new code.

Python Set operations and isin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

np.isin is the right tool when you want to filter one array by membership in another, as in the ids example. It keeps duplicates and the original order, while intersect1d returns each common value once, sorted. The return_indices=True option tells you where the common values sit in each input.

A Python loop with in on a list is slow because every lookup scans the list. np.isin sorts or uses a lookup table internally and handles millions of elements quickly.

Note: The set functions work on NumPy arrays of numbers or strings. They compare by value, so 1 and 1.0 count as the same element.

bincount: counting non-negative integers

np.bincount(x) counts how many times each integer 0, 1, 2, ..., max(x) occurs. The result at position i is the count of i. It is much faster than unique for small non-negative integers because it needs no sorting.

Two options make it far more useful:

  • minlength=n guarantees at least n result slots (so counts of rare categories show up as zeros).
  • weights=w adds the weight of each element instead of 1, which turns bincount into a group-by-sum.
Python bincount for counts and group sums

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

bincount accepts only non-negative integers. Floats and negative numbers raise an error. If your categories are strings or arbitrary integers, first convert them to small codes with np.unique(..., return_inverse=True), then feed the inverse array to bincount. The group-by-sum pattern (bincount with weights) replaces what would otherwise be a Python loop with a dictionary.

The last lines count letters in a word: frombuffer turns the bytes of the string into an array of small integers, ord("a") shifts them to start at 0, and flatnonzero skips the letters that never occur.

histogram and digitize: counting numbers in bins

For continuous values, you count how many fall into each bin (interval). np.histogram(x, bins) does it:

  • bins can be a number (that many equal-width bins between the minimum and maximum) or a sequence of edges.
  • It returns (counts, edges). Note len(edges) == len(counts) + 1.
  • Every bin is half-open [left, right) except the last, which also includes its right edge.
  • density=True normalizes so the area under the histogram is 1.

np.digitize(x, bins) answers a slightly different question: which bin does each element belong to? It returns a bin number for every element, which you can use for labels, colors or group-by operations.

Python histogram, digitize and a text chart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

digitize and searchsorted are close relatives: np.digitize(x, bins) equals np.searchsorted(bins, x, side="right") for increasing bins. With the default right=False, an age of exactly 18 lands in bin 1 (adult), and with right=True the boundary value falls into the lower bin, as the last digitize call shows.

Combining digitize with bincount is a compact way to build your own histogram with custom labels: np.bincount(which) counted the children, adults and seniors above. Plotting histograms with matplotlib is shown in Arrays as Images and Plots, and a two-dimensional histogram appears in Statistics, Correlation and Covariance.

Cumulative counting

Sometimes you want a running count rather than totals. np.cumsum over a boolean (or 0/1) array counts how many conditions have been met so far. np.maximum.accumulate gives a running maximum. And a loop over the few distinct categories, not over the data, can number the occurrences of each value.

Python Running counts and occurrence numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loop runs once per distinct value, not once per element, so it stays cheap even for large arrays with few categories. The "distinct so far" line is a handy trick: mark the first appearance of each value, then take the cumulative sum of those marks.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does np.unique(a, return_counts=True) return?

    You get a tuple of two arrays: sorted unique values and their counts, in the same order.
  2. 2. Which call keeps duplicates and the original order while telling you which elements of ids are in vip?

    isin returns a boolean array with the shape of ids. intersect1d, setdiff1d and union1d return sorted, duplicate-free arrays.
  3. 3. What is the main limitation of np.bincount?

    The values are used as positions in the result, so they must be non-negative integers. Encode other data with np.unique(..., return_inverse=True) first.
  4. 4. A histogram is computed with np.histogram(x, bins=5). How many edges are returned?

    Five bins need six edges, because each bin has a left and right edge and neighbors share them.
  5. 5. How can bincount compute the total sales per store?

    With weights, each element adds its weight to the bin given by its integer label, which is a group-by-sum.

Summary

  • np.unique returns sorted distinct values, with optional return_counts, return_index and return_inverse; axis=0 finds distinct rows.
  • union1d, intersect1d, setdiff1d and setxor1d are the set operations; np.isin is the fast membership test (it replaces in1d).
  • np.bincount counts non-negative integers and, with weights, sums values per group.
  • np.histogram counts values per bin (edges are one more than counts); np.digitize returns the bin index of each element.
  • cumsum on booleans and np.maximum.accumulate give running counts and running extremes.

Time to move on to matrices: Vectors, Dot Products and the @ Operator.