Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 64 ยท Selecting and Filtering

Filtering with Conditions

Build boolean masks, combine them with &, | and ~, use isin, between, str.contains, isna and query, and learn where and mask for conditional replacement.

Most real questions about data are filters: "orders over $100", "customers in Germany or France", "rows with no email address". In pandas a filter is a boolean mask: a Series of True and False values, one per row, that says which rows to keep. You build the mask from a condition, then use it to select.

This lesson shows how to create masks from comparisons, combine several conditions without hitting the usual operator trap, use the helper methods isin, between, str.contains and isna, and finish with query, where and mask. If you know NumPy, this is the same idea as Boolean Masks, with row labels attached.

From a comparison to a mask

Comparing a column with a value is applied to every row and returns a boolean Series with the same index. Putting that Series in square brackets, or in loc, keeps the rows where it is True.

flowchart LR
    A["df['price']<br/>12, 55, 8, 40"] --> B["> 20"]
    B --> C["mask<br/>F, T, F, T"]
    C --> D["df[mask]<br/>rows 1 and 3"]

The comparison produces one True or False per row; the mask keeps the True rows.

Python Compare, then filter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same operators work as in Python: ==, !=, <, <=, >, >=. Use mask.sum() to count matches and mask.mean() for the fraction of rows that match; both are quick data checks.

A filtered DataFrame keeps the original row labels, so the index after filtering is usually not 0, 1, 2.... Call reset_index(drop=True) if you want fresh numbers (see The Index).

Combining conditions: &, | and ~

Python's and, or and not need one truth value, so they raise an error when given a Series. pandas uses the elementwise operators instead:

Operator Meaning
& AND: both conditions true
| OR: at least one true
~ NOT: flip every value

Always wrap each comparison in parentheses. The operators & and | bind more tightly than > or ==, so without parentheses Python reads a > 1 & b < 5 in a way you did not intend.

Python AND, OR and NOT

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ValueError message says the truth value of a Series is ambiguous. It is pandas refusing to collapse a whole column into one True or False. When you see it, look for an accidental and, or or not, or a missing pair of parentheses.

Name your conditions. is_vip = orders["total"] > 500 and is_recent = orders["year"] == 2025 give you orders[is_vip & is_recent], which reads like a sentence and lets you test each part on its own.

isin: membership in a list

Chaining many == tests with | is clumsy. isin checks each value against a collection and returns a mask. Negate it with ~ for "not in".

Python isin and ~isin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

between: ranges in one call

between(low, high) is shorthand for (s >= low) & (s <= high). Both ends are included by default; pass inclusive="left", "right" or "neither" to change that.

Python between with inclusive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Text conditions

On string columns, the .str accessor gives masks too. str.contains takes a regular expression by default; pass regex=False to search for plain text, and case=False to ignore case. Other useful tests are str.startswith, str.endswith and str.len. All string tools get their own lesson in Working with Text.

Python Filtering text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Missing values

A comparison with a missing value is never True, so rows with NaN silently drop out of both x > 5 and x <= 5. To find or keep missing values explicitly, use isna() and notna().

Python Filtering around missing data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Missing values are covered in depth in Missing Data. For now, remember that a filter never mentions them unless you ask.

query: filters as text

DataFrame.query takes the condition as a string. Column names are used directly, so no parentheses or repeated df[...] are needed, and you can use and, or and not. Use @name to refer to a Python variable.

Python query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

query is neat for short, readable filters, and it is part of the method-chaining style. Column names with spaces need backticks (`unit price` > 3). Masks remain more flexible, since they can be built from any Python code.

where and mask: replace instead of drop

Filtering removes rows. Sometimes you want to keep all rows and change values only where a condition holds. where(cond, other) keeps values where cond is True and replaces the rest with other (default NaN); mask(cond, other) does the reverse.

Python where and mask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For choosing between two values with a rule, np.where is also common; see Vectorised Operations.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which expression keeps rows where age is at least 18 and member is True?

    Use the elementwise & and parenthesize each comparison. and raises a ValueError for a Series, and && is not Python.
  2. 2. What does s.between(1, 5) return for the value 5?

    between includes both ends unless you pass inclusive='left', 'right' or 'neither'.
  3. 3. A column has a NaN in some rows. Rows with NaN are missing from both df[df['x'] > 5] and df[df['x'] <= 5]. Why?

    Any ordering comparison with a missing value is False. Use isna() to select those rows explicitly.
  4. 4. How do you select rows whose city is NOT in a list called big_cities?

    isin builds a membership mask and ~ flips it.
  5. 5. What is the difference between s.where(s > 0) and s[s > 0]?

    Filtering with brackets shortens the Series. where keeps the shape and replaces the non-matching values.

Summary

  • A condition on a column gives a boolean Series; df[mask] or df.loc[mask, cols] keeps the True rows and preserves their labels.
  • Combine masks with &, | and ~, with parentheses around each comparison; never use and, or or not.
  • isin, between, str.contains and isna produce masks for common cases.
  • Comparisons with missing values are False, so handle NaN explicitly.
  • query("price > @limit and stock > 0") is a readable alternative for simple filters.
  • where and mask keep the shape and replace values instead of dropping rows.

Next up: put your filtered rows in order in Sorting and Ranking.