Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 39 ยท Parsing with BeautifulSoup

Extracting Text and Attributes

Turn tags into clean values: get_text options, strip and separator, attribute extraction, converting prices and dates, handling missing fields and building records.

Finding the right tag is only half the work. The value you want is usually buried in text with currency symbols, units, labels or odd whitespace, or stored in an attribute instead of visible text. This lesson focuses on the extraction step: reading text and attributes and converting them into clean, typed values you can store and analyze.

The running example is a small list of listings with messy values, which is a good picture of real pages.

Reading text

You already met the options. Here they are side by side so you can choose deliberately.

Python Text accessors compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two cleaning idioms deserve to become reflexes:

  • get_text(strip=True) trims the ends of each text piece.
  • " ".join(text.split()) collapses any run of spaces, tabs and newlines inside the text into a single space. It also removes the non-breaking space \xa0, because str.split() treats it as whitespace.

Reading attributes

Text is for humans; attributes are often cleaner for machines. Look for data-*, datetime, content, value, href, src and alt.

Python Attributes hold the clean values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The visible text said "3 days ago", which is useless for analysis, but the datetime attribute holds an exact timestamp. Always check the markup for a machine-readable twin of the visible value.

A list comprehension over find_all extracts a whole column:

Python
links = [a["href"] for a in soup.find_all("a", href=True)]
images = [img["src"] for img in soup.find_all("img", src=True)]

Using href=True in the search guarantees the attribute exists, so a["href"] cannot raise KeyError. Remember to resolve relative URLs with urljoin, as shown in URLs and Query Strings.

Converting text to numbers

Prices, ratings, counts and areas arrive as strings like "$1,450.00", "4.5 out of 5", "1.2K" or "2 reviews". Write small converter functions and test them on awkward inputs.

Python Robust number parsers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The price parser looks at the right-most separator to decide whether a comma or a dot is the decimal mark, which handles US (1,450.00) and European (1.299,50) styles. No parser is perfect for every site: inspect what the target site actually produces and write the simplest code that handles it, plus a test list of real examples.

Ratings and dates follow the same idea:

Python
rating = float(re.search(r"[\d.]+", "4.5 out of 5").group())      # 4.5

For dates, prefer the datetime attribute and datetime.fromisoformat. For text dates use datetime.strptime(text, "%d %B %Y"), or the dateutil package for varied formats.

Handling missing fields

Real pages leave things out: no price on a "call us" listing, no image on a new product. The easiest way to stay safe is a helper that returns None instead of crashing.

Python A safe text helper

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second item has no price or image, and the script still finishes, with n/a and None recorded. Never let one odd record kill a run of ten thousand pages. Handling Messy HTML extends this idea with validation and logging.

Building a record

Combine the pieces into one function that turns a block into a typed dictionary. Keep extraction (getting raw strings) and conversion (parsing numbers) separate and easy to test:

Python
def parse_listing(block) -> dict:
    return {
        "id": int(block["data-id"]),
        "title": text_of(block, "h3"),
        "price": parse_price(text_of(block, class_="price")),
        "url": attr_of(block, "href", "a", class_="more"),
    }

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which idiom collapses runs of whitespace (including newlines and non-breaking spaces) inside text into single spaces?

    str.split() with no arguments splits on any run of whitespace, so joining the pieces with one space normalizes everything.
  2. 2. A page shows '3 days ago' in a time tag. Where might the exact timestamp be?

    The time element usually carries a machine-readable datetime attribute.
  3. 3. Why write a helper that returns None for a missing tag?

    Calling .get_text() on None raises AttributeError. A safe helper lets one odd record pass without killing the job.
  4. 4. Why does a price parser need to look at which separator comes last?

    US style writes 1,450.00 and European style writes 1.450,00; the right-most separator is the decimal point.

Summary

  • Choose among get_text(), get_text(strip=True) and " ".join(text.split()) on purpose.
  • Prefer machine-readable attributes (data-*, datetime, content) over visible text.
  • Use href=True and similar filters so attribute access cannot fail.
  • Write small, tested converters for prices, counts and dates.
  • Use safe helpers so missing fields become None instead of crashes.

Next: Modifying the Tree and Choosing a Parser