Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 39 ยท Parsing with BeautifulSoup

find() and find_all() in BeautifulSoup

Master BeautifulSoup searching: find and find_all by tag, class, id, attributes, text, regular expressions, lists and functions, with limit, recursive and string filters.

find() and find_all() are the workhorses of BeautifulSoup. Almost every scraper uses them hundreds of times. find_all returns a list of every matching tag; find returns the first match (or None). Both accept the same filters: tag names, attributes, classes, text, regular expressions, lists and even your own functions.

This lesson walks through all of them using one product page, so you can compare what each filter matches.

The page

Python Setup: a product list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The next examples reuse that HTML. Each TryIt repeats it so it runs independently.

Searching by tag name

Python
soup.find("h3")              # first h3
soup.find_all("h3")          # list of all h3
soup.find_all(["h3", "p"])   # any of these tags
soup.find_all(True)          # every tag in the document

find_all returns a ResultSet, which behaves like a list; an empty list means nothing matched, so you can test if not results:.

Searching by class and id

class is a reserved word in Python, so BeautifulSoup uses class_:

Python Class and id filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice two behaviors. class_="sale" matches any element whose class list contains sale, even though it also has product. And class_="product sale" matches only elements whose full class attribute string is exactly product sale, in that order. To require two classes regardless of order, use a CSS selector: soup.select("div.product.sale").

Searching by other attributes

Pass any attribute as a keyword argument, or use the attrs dictionary for names that are not valid Python identifiers, such as data-sku:

Python Attribute filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The special values are worth memorizing:

  • attr=True matches elements that have the attribute at all.
  • attr=False matches elements that lack it.
  • attr=re.compile(...) matches by pattern.
  • attr=["a", "b"] matches any listed value.

Searching by text

The string= argument (called text= in old tutorials) filters by a tag's own text content. When used with find_all alone it returns the strings, not tags; combine it with a tag name to get tags.

Python Searching by text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

string="Phone Stand" requires an exact match of the whole string. For "contains", use a regular expression or a function. Note that string= only matches tags whose only content is that string; a tag with child tags has .string equal to None and will not match.

Searching with a function

For conditions that no keyword can express, pass a function that receives each tag and returns True or False:

Python Custom filter functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the most flexible tool, and it can look at a tag's children, attributes and position all at once.

limit, recursive and find versus find_all

  • limit=3 stops after three matches: find_all("a", limit=3). In fact find(...) is just find_all(..., limit=1) returning the first element or None.
  • recursive=False only inspects direct children, not deeper descendants.
  • find returns None when nothing matches; find_all returns an empty list. Treat them differently in code.
Python
first_link = soup.find("a")                 # may be None
if first_link is not None:
    print(first_link["href"])

for link in soup.find_all("a"):             # safe: loops zero times if none
    print(link["href"])

A shortcut: calling the soup

soup("a") is shorthand for soup.find_all("a"), and tag("a", class_="x") works on tags too. You will see it in short scripts.

Chaining searches: scope matters

Find the repeating container first, then search inside each one. That keeps each record's fields together.

Python
for card in soup.find_all("div", class_="product"):
    name = card.find("h3").get_text(strip=True)
    price = card.find("span", class_="price").get_text(strip=True)

Calling soup.find("span", class_="price") inside the loop would repeat the first price for every card.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does find_all('p') return when there are no p tags?

    find_all always returns a list-like ResultSet; it is empty when nothing matches. find returns None.
  2. 2. Which filter finds a tags that have an href attribute, whatever its value?

    Passing True means the attribute must be present.
  3. 3. How do you search for class 'sale' with BeautifulSoup keyword arguments?

    class is a Python keyword, so BeautifulSoup accepts class_ (with a trailing underscore).
  4. 4. find_all('li', string='desk') does not match <li>Desk Lamp</li>. Why?

    An exact string must equal the tag's text. Use re.compile('desk', re.I) for a case-insensitive contains match.

Summary

  • find returns the first match or None; find_all returns a list.
  • Filter by tag name, class_, id, any attribute (True, False, regex, list) and by string=.
  • Use a function filter for conditions that combine several tests.
  • limit and recursive=False bound the search.
  • Find the repeating container first, then search inside each one.

Next: Extracting Text and Attributes