Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 49 ยท Indexing and Selection

Boolean Masks and Filtering

Filter NumPy arrays with boolean masks, combine conditions with &, | and ~, use np.where, nonzero, any, all, select and clip to pick and replace values.

Slices pick elements by position. Very often, though, you do not care where a value sits; you care what it is: "all temperatures above 30", "every negative balance", "the rows where the status is error". NumPy answers these questions with boolean masks: an array of True and False values that says, element by element, which items to keep.

Masks are the tool you will use most for cleaning and filtering data. This lesson shows how to build them from comparisons, how to combine several conditions without hitting the classic precedence trap, how to use masks to read and to assign, and how np.where, np.nonzero, np.select and np.clip build on the same idea. If you have not met arrays yet, start with Your First Arrays, and if you want a refresher on positional selection, see Indexing and Slicing.

From a comparison to a mask

Comparing an array with a number compares every element and returns a new array of booleans, the same shape as the input. Using that boolean array as an index keeps only the elements where it is True.

flowchart LR
    A["data<br/>[12 35 7 28 41 3]"] --> B["data > 20"]
    B --> C["mask<br/>[F T F T T F]"]
    A --> D["data[mask]"]
    C --> D
    D --> E["[35 28 41]"]

A comparison produces a mask; indexing with the mask keeps the elements at the True positions.

Python Compare, then filter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because True is 1 and False is 0 when you do arithmetic, mask.sum() counts the matches and mask.mean() gives the fraction. Both are handy one-liners for quick data checks.

The mask must have the same length as the axis it indexes. A mask that is too short or too long raises an IndexError. Also remember that filtering with a mask always returns a copy, never a view; more on that in Views vs Copies.

Masks on 2D arrays

On a 2D array a full-shape mask returns a flat 1D array of the matching elements, because the surviving items no longer form a rectangle. A 1D mask can instead select whole rows or columns:

Python Masks on a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Combining conditions: &, | and ~

Python's and, or and not work on a single truth value, so they fail on arrays. Use the elementwise operators instead:

Operator Meaning
& AND: both conditions true
| OR: at least one true
~ NOT: flip every value

The one rule you must remember: wrap each comparison in parentheses. The & and | operators bind more tightly than > or ==, so a > 1 & a < 5 is parsed as a > (1 & a) < 5, which is not what you meant.

Python AND, OR and NOT

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error message mentions an ambiguous truth value. That is NumPy refusing to turn a whole array into a single True or False. If you see it, look for a missing pair of parentheses or an accidental and/or.

Tip: Build complicated filters in steps. Give each condition a name (is_adult, is_member) and combine the named masks. The code reads like a sentence and each piece is easy to test on its own.

Assigning through a mask

A mask can sit on the left side of = too. NumPy then writes into exactly the selected positions of the original array. This is the clean way to patch bad values:

Python Masked assignment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Missing values are a big topic of their own; see Missing Data and NaN.

np.where: choose between two arrays

np.where(condition, x, y) builds a new array. Wherever the condition is true it takes the value from x, otherwise from y. Scalars and arrays both work, and they are broadcast to a common shape (see Broadcasting).

Python np.where with three arguments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Unlike masked assignment, np.where does not change its input; it returns a new array. Note that both x and y are fully computed before the choice is made, so np.where(d != 0, 1 / d, 0) still evaluates the division for zeros and may emit a warning.

np.where with one argument

With only a condition, np.where returns the positions of the true values, which is the same as np.nonzero. For a 1D array you get a tuple holding one index array; for a 2D array you get one array per axis (row indexes and column indexes).

Python Finding positions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

np.nonzero treats any non-zero value as true, so it also finds the positions of non-zero numbers directly. To get the position of the single largest or smallest value, use argmax and argmin from Aggregations and Axes.

any and all

Two reductions turn a mask into a single answer. np.any(mask) is true if at least one element is true; np.all(mask) is true only when every element is. Both accept an axis argument to answer the question per row or per column.

Python any and all

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

np.select: more than two outcomes

np.where handles two outcomes. For a ladder of conditions, np.select(condlist, choicelist, default) checks the conditions in order and uses the first one that matches, like an if / elif / else chain applied to the whole array.

Python np.select

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Order matters: because 95 >= 80 is also true, the stricter condition must come first.

np.clip: limit a range

np.clip(a, low, high) forces every value into the interval [low, high]. It is a shortcut for two masked assignments, and either bound may be None to leave that side open.

Python np.clip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which expression selects the elements of a that are between 10 and 20, inclusive?

    Use the elementwise & and put each comparison in parentheses. and and chained comparisons need a single truth value, and without parentheses & binds tighter than the comparisons.
  2. 2. What does np.where(a > 0) (one argument) return?

    With a single argument, np.where behaves like np.nonzero and returns the positions of the true elements as a tuple of arrays.
  3. 3. After a[a < 0] = 0, what happens to the array a?

    A mask on the left side of an assignment writes into the original array at the selected positions.
  4. 4. In np.select, why must the stricter condition (score >= 90) come before the looser one (score >= 80)?

    np.select uses the first condition that is true for each element, so a looser test placed earlier would claim values that belong to the stricter one.

Summary

  • Comparing an array with a value gives a boolean mask of the same shape; a[mask] keeps the True positions and returns a copy.
  • Combine masks with &, | and ~, and put each comparison in parentheses. and, or and not do not work on arrays.
  • a[mask] = value patches the original array in place.
  • np.where(cond, x, y) picks between two arrays; np.where(cond) and np.nonzero return positions.
  • any and all collapse a mask to one answer, or one per axis; sum and mean count and measure matches.
  • np.select handles several ranked conditions and np.clip limits values to a range.

Next up: choose elements by lists of positions in Fancy Indexing.