Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 39 ยท Selectors and Data Extraction

CSS Selectors with select()

Use CSS selectors in BeautifulSoup: select and select_one, combinators, attribute selectors, nth-child, :not, :has, :-soup-contains and a selector cheat sheet for scrapers.

You already know CSS from styling pages: .card, #price, ul > li. The same selectors can find elements for a scraper. BeautifulSoup supports them through select() and select_one(), powered by the soupsieve library. A single selector string often replaces several lines of find_all and navigation calls, and you can test it directly in your browser's Console with document.querySelectorAll(...) or by copying it from the Elements panel.

select and select_one

  • soup.select("css") returns a list of all matches (empty if none).
  • soup.select_one("css") returns the first match or None.
  • Both work on any tag too, searching only inside it.
Python The basics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The selector toolbox

Simple selectors

Selector Matches
p every p
.card class card
#main id main
p.note p with class note
.a.b both classes a and b, in any order
* every element

Combinators (relationships)

Selector Meaning
ul li li anywhere inside ul (descendant)
ul > li li that is a direct child of ul
h2 + p the p immediately after an h2 (adjacent sibling)
h2 ~ p every p after an h2 with the same parent (general sibling)
a, b either selector (list)

Attribute selectors

Selector Matches
[href] has an href
[type="text"] exact value
[href^="https"] starts with
[href$=".pdf"] ends with
[href*="/product/"] contains
[class~="sale"] value in a space-separated list
[lang|="en"] en or en-US
[type="TEXT" i] case-insensitive

Position and state pseudo-classes

Selector Matches
li:first-child, :last-child first or last child
li:nth-child(2) second child
li:nth-child(odd), :nth-child(3n+1) patterns
li:nth-of-type(2) second li among its siblings of that type
:not(.sale) does not match
:has(a) contains a matching descendant
:empty no children and no text
:is(h1, h2) any of the listed selectors

Soup Sieve extensions (not in browsers)

Selector Matches
p:-soup-contains("Price") elements whose text contains this (also :-soup-contains-own)

Combinators and attributes in action

Python Combinators and attribute selectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every line replaces what would be a small loop with find_all and string tests. Scan the output to learn how each selector behaves; h2 + p returns only the paragraph directly after each heading, while h2 ~ p returns all following paragraphs.

:has(), :-soup-contains() and :is()

These make selectors powerful enough to find a record by what it contains:

Python Find by content and structure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The label-to-value trick, th:-soup-contains("Color") + td, finds the table cell next to a header whose text contains "Color". It replaces a find-then-navigate pair with one expression.

:-soup-contains() is specific to Soup Sieve. A selector using it will not work in a browser console, so test the rest of the selector there and add the text condition in Python.

Selecting inside a tag

Call select on a record to stay inside it:

Python
for card in soup.select("div.card"):
    name = card.select_one("h3").get_text(strip=True)
    price = card.select_one("b").get_text(strip=True)
    badge = card.select_one(".badge")                  # may be None
    print(name, price, badge.text if badge else "-")

Selectors are evaluated relative to the whole document for matching purposes but only return elements inside the tag you called them on, so something like card.select("div h3") can still match when the div ancestor is outside the card. Keep that in mind when selectors seem to match too loosely.

Getting a selector from the browser

In the Elements panel, right-click an element, choose Copy > Copy selector. You will get something like #shop > ul > li:nth-child(2) > a. It works, but it is fragile because it encodes the exact position. Shorten it to the most meaningful, stable parts: li.product a. Rules for good scraper selectors:

  1. Prefer ids, meaningful classes and data-* attributes to positions.
  2. Avoid auto-generated class names (.css-1x9z3kq, .sc-bdVaJa); they change on every deploy.
  3. Keep it short. Every extra level is another way to break.
  4. Anchor on semantics (article, h1, [itemprop="price"], [aria-label="Add to cart"]) rather than layout.
  5. Test on several pages, including edge cases like sale items and missing images.

select versus find_all

select find_all
Syntax CSS strings Python arguments
Nested structure One expression (ul > li a) Loops or chains
Several classes .a.b Awkward
Text conditions :-soup-contains string=
Custom logic No Function filters
Speed Similar Similar

Use select for structure and find_all with a function when you need real logic. Mixing both in one scraper is normal.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does select_one('.price') return if nothing matches?

    select_one returns None when there is no match, while select returns an empty list.
  2. 2. Which selector matches links whose href ends with .pdf?

    The $= operator means ends with, ^= means starts with and *= means contains.
  3. 3. How do you select the p that immediately follows an h2?

    The + combinator is the adjacent sibling combinator; > is direct child and a space is any descendant.
  4. 4. Why avoid selectors like .css-1x9z3kq?

    Build tools generate these names, and they change with new builds, which breaks scrapers that depend on them.

Summary

  • select() returns a list and select_one() returns the first match or None, using full CSS selector syntax.
  • Master descendant, >, +, ~, attribute operators (^=, $=, *=), :nth-child, :not, :has and :is.
  • :-soup-contains("text") selects by text but works only in Soup Sieve.
  • Copy selectors from browser tools, then shorten them to stable parts.
  • Prefer meaningful classes, ids and data-* attributes over positions and generated names.

Next: XPath with lxml