Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 39 ยท Parsing with BeautifulSoup

BeautifulSoup Basics

Start with BeautifulSoup: create a soup object, choose a parser, access tags by name, work with Tag and NavigableString objects, read attributes and print the tree.

BeautifulSoup (package beautifulsoup4, import bs4) is the most popular Python library for pulling data out of HTML. It takes messy markup, builds a tree of Python objects and gives you friendly methods to search and walk it. It tolerates the broken HTML that real sites produce, which is why it is the usual first choice for scrapers.

This lesson introduces the core objects. The next lessons build on them: navigating, searching, extracting and modifying.

Creating a soup

Give BeautifulSoup your markup and the name of a parser. The markup can be a string or bytes.

Python Create a soup and print it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

prettify() returns the tree re-indented, one tag per line. It is the best way to see what the parser actually built, which can differ from the original markup when the HTML was broken. Print it whenever a selector surprises you.

To use lxml or html5lib you must install them (pip install lxml html5lib); "html.parser" is built in. The parser lesson in the toolkit chapter compares them.

The four kinds of object

Everything in the tree is one of four types.

Type What it is Example
BeautifulSoup The whole document soup
Tag An element soup.h1
NavigableString Text inside a tag soup.h1.string
Comment An HTML comment (a special string) <!-- ads -->
Python Meet the objects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Observe three things. A Tag has a name and an attrs dictionary. The class attribute is a list (['a', 'b']) because HTML allows several classes; id is a plain string. And the comment is a string subclass, which is why get_text() skips it.

Accessing tags by name

The shortcut soup.tagname returns the first tag with that name, searching the whole tree. You can chain it to walk down.

Python Dot access

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dot access is convenient for unique tags like title, h1 or body. For anything repeated, use find_all (next lessons). A missing tag returns None, and calling a method on None raises AttributeError: 'NoneType' object has no attribute 'text'. This is by far the most common BeautifulSoup error, and Handling Messy HTML shows how to guard against it.

Attributes

Read them like dictionary keys. Use .get() when the attribute might be absent, which returns None (or a default) instead of raising KeyError.

Python Reading and changing attributes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Text: .string, .text and get_text()

Three ways to read text, with different behavior:

  • .string: the single string inside a tag, or None if the tag has several children.
  • .text and .get_text(): all text inside the tag and its descendants, concatenated.
  • .strings and .stripped_strings: generators over each text piece.
Python Choosing the right text accessor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

get_text(strip=True) trims each piece, but joins them with an empty separator, which can glue words together (Hand-pickedclassicsfor the week.). Use get_text(" ", strip=True) when pieces should be separated by a space.

Printing and converting

  • str(tag) gives the tag's HTML, including children.
  • tag.decode_contents() gives the inner HTML without the tag itself.
  • tag.prettify() gives indented HTML.
  • soup.encode("utf-8") gives bytes.
Python
div = soup.find("div")
print(str(div))
print(div.decode_contents())

Working with files and bytes

During development, parse saved HTML rather than hitting the site repeatedly:

Python
with open("page.html", encoding="utf-8") as f:
    soup = BeautifulSoup(f, "lxml")

Passing an open file object works. Passing the bytes from response.content lets BeautifulSoup detect the encoding from the page's <meta charset>. Handing it the already-decoded response.text is also fine when the encoding was correct.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does soup.p return when the document has three p tags?

    Dot access returns only the first matching tag. Use find_all to get every one.
  2. 2. Why is tag['class'] a list?

    class (and a few other attributes like rel) can hold several space-separated values, so BeautifulSoup returns them as a list.
  3. 3. Which is safest when an attribute may be missing?

    .get() returns None (or a default) instead of raising KeyError.
  4. 4. What is the best first step when your selector finds nothing and you cannot tell why?

    The parser may have restructured the HTML, or the content may not be present at all; prettify shows exactly what you have.

Summary

  • BeautifulSoup(markup, parser) builds a tree of Tag, NavigableString and Comment objects.
  • soup.tag returns the first match; a missing tag returns None.
  • Attributes behave like a dictionary; class is a list; use .get() for optional attributes.
  • .string needs exactly one child, while get_text(sep, strip=True) collects all text.
  • prettify() shows the tree the parser really built.

Next: Navigating the Tree