Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 39 ยท Parsing with BeautifulSoup

Modifying the Tree and Choosing a Parser

Clean HTML before extracting: remove scripts and ads with decompose and extract, replace and wrap tags, find text and replace it, and compare html.parser, lxml and html5lib.

A BeautifulSoup tree is not read-only. You can delete the noisy parts of a page (scripts, ads, navigation), replace text, add attributes and re-serialize the result. Cleaning the tree before extraction often makes the extraction code far simpler: instead of working around junk, you remove it. This lesson covers the modification methods, then compares the three parsers, because the choice of parser decides what tree you get from broken HTML.

Removing tags

Three methods remove content, with different effects:

Method Effect
tag.decompose() Destroys the tag and everything inside it. Frees memory.
tag.extract() Removes the tag from the tree and returns it, so you can reuse it.
tag.unwrap() Removes the tag but keeps its children in place.
tag.clear() Empties the tag's contents but keeps the tag.
Python Strip noise before extracting text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice soup([...]), the shorthand for find_all with a list of names, and that comments are strings, so you search for them with string= and a type test. After the clean-up, the main article text is the only text left, and a single get_text call returns it. This is a quick recipe for extracting the "main text" of an article-like page.

Be careful when removing tags while looping. decompose inside a loop over find_all is safe because find_all already returned a list, but removing a parent and then touching its already-destroyed children raises errors.

Changing text and attributes

Assign to .string to replace a tag's contents, edit attributes like dictionary entries, and rename tags:

Python Edit text, attributes and names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Replacing text inside a larger string is a common need: you find the text node, create a replacement string, and call replace_with:

Python Find text and replace it in the soup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

node.replace("...", "...") is the normal str.replace because a text node is a string subclass, and replace_with swaps the old node for the new string inside the tree. This is how you redact or rewrite text while keeping the surrounding markup intact.

Building and inserting tags

You can create new tags with soup.new_tag and place them with append, insert, insert_before, insert_after and wrap:

Python Create, insert and wrap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You will rarely build pages in a scraper, but these tools are useful for normalizing content (turning relative links into absolute ones before saving a page copy, adding missing attributes) and for generating reports.

A concrete cleaning task: save an article's HTML but with working links.

Python Absolute URLs everywhere

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a parser

BeautifulSoup delegates parsing to one of three engines. For valid HTML they produce the same tree. They differ most on broken HTML, so a surprising result almost always comes from the parser.

Python Parsers disagree on broken HTML

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output carefully. You can see that:

  • html5lib always creates html, head and body, and adds tbody inside tables, exactly as a browser DOM would. That means a CSS selector copied from the browser's Inspector (table > tbody > tr) works with html5lib but may not with html.parser.
  • lxml wraps loose content in html and body, and drops stray end tags.
  • html.parser is the most literal; it does not invent structure, and it can leave nested links nested.
html.parser lxml html5lib
Speed Medium Fastest Slowest
Install None pip install lxml pip install html5lib
Matches the browser DOM No Mostly Yes
Best for Simple jobs, no dependencies Large crawls Badly broken pages

A practical rule: use lxml by default, and switch to html5lib if a selector that works in the browser fails in your script. Use the same parser in development and production, since results can differ.

There is also an XML mode: BeautifulSoup(xml, "xml") (requires lxml) keeps tag case and handles sitemaps, RSS feeds and namespaces.

Performance: parse less

Large pages cost time and memory. SoupStrainer tells BeautifulSoup to build a tree for matching tags only:

Python
from bs4 import BeautifulSoup, SoupStrainer

only_links = SoupStrainer("a", href=True)
soup = BeautifulSoup(big_html, "lxml", parse_only=only_links)

This is an optimization for big pages. Do not reach for it until you have a speed problem. Also remember to call decompose() on large subtrees you no longer need, and to avoid keeping every soup object of a long crawl in memory: extract your records, then let the soup go out of scope.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which method removes a tag but keeps its children in the document?

    unwrap() replaces a tag with its contents. decompose destroys it, extract returns it, clear empties it.
  2. 2. How do you replace the text node 'abc' with 'xyz' in place?

    replace_with swaps a node (tag or string) for another inside the tree.
  3. 3. A selector 'table > tbody > tr' works in Chrome but finds nothing with html.parser. What is the likely reason?

    Browsers insert an implied tbody. html.parser keeps the markup literal; html5lib mimics the browser.
  4. 4. Which parser is generally the fastest?

    lxml is written in C and is typically the fastest of the three.

Summary

  • decompose destroys, extract removes and returns, unwrap keeps the children, clear empties.
  • Remove scripts, styles, navigation and ads first, and extraction gets simpler.
  • Edit text with .string and replace_with, attributes like dictionary entries, and build tags with new_tag.
  • Parsers differ on broken HTML: html5lib mirrors the browser, lxml is fastest, html.parser is built in.
  • Use SoupStrainer and release big soups when performance matters.

Next: CSS Selectors with select()