Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 39 ยท Selectors and Data Extraction

Handling Messy HTML and Layout Changes

Write scrapers that survive real pages: defensive extraction, fallback selectors, validation, logging, tests with saved HTML and detecting when a site layout changes.

A scraper that works on your sample page is a prototype. A scraper that keeps working for months on thousands of real pages is a product. Real pages have missing fields, odd formats, A/B-tested layouts, broken markup and the occasional error page that returns HTTP 200. This lesson turns the techniques you have learned into a defensive style: expect surprises, handle them gracefully and find out quickly when the site changes.

The five usual breakages

  1. A field is missing on some records: no price, no image, no rating.
  2. The format varies: $5, 5 USD, Free, Call for price.
  3. The layout changes: a class is renamed, a wrapper added, the table replaced by cards.
  4. The page is not the page you wanted: a CAPTCHA, a consent wall, a "no results" message or a login redirect.
  5. The HTML is malformed, so different parsers build different trees.

Defensive extraction

The classic crash is AttributeError: 'NoneType' object has no attribute 'text': find returned None and you called a method on it. Make the safe path the easy path with small helpers, and give every field a declared default.

Python Records that survive missing fields

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four principles are on display. Helpers return None instead of raising. Conversion errors become None rather than aborting the run. Unusable records are skipped but recorded in problems. And partial records are kept (the Stand with no numeric price) so you can decide later what to do. Silent data loss is the enemy: always collect a list of what went wrong.

Fallback selectors

Sites often have two versions of the same element (old and new layout, mobile and desktop). Try selectors in order until one works:

Python First selector that matches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Order the list from the most stable (semantic attributes such as data-test, itemprop, aria-label) to the least. Keep the selectors in one named constant at the top of the file, so a layout change means editing a single line.

Detect the wrong page early

A 200 response is not proof you got what you wanted. Add a page check before extraction: a test for something that must be present on a good page, and signs of the bad ones.

Python Validate the page before parsing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the classification is not ok, stop or slow down instead of storing garbage. Never try to defeat a CAPTCHA with your scraper; treat it as a signal to stop (the legal lesson explains why). Log the URL and save the HTML so you can look at it later.

Validate what you extracted

After extraction, check that the values make sense. Simple rules catch most layout-change bugs:

  • Required fields are present and non-empty.
  • Numbers are within a plausible range (a price of 0 or 9,999,999 is suspect).
  • Dates parse and are not in the far future.
  • The count of records matches expectations: if a category page normally lists 20 products and today you got 0, something changed.
  • The share of None values per field has not jumped.
Python Validation and a layout-change alarm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If validation fails on a run, do not overwrite yesterday's good data with today's broken results. Write new output to a separate file and alert yourself (email, chat message or a nonzero exit code).

Logging instead of printing

In a long run, use the logging module. It records levels, timestamps and URLs without cluttering output:

Python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    filename="scraper.log",
)
log = logging.getLogger("shop")

try:
    record = parse_listing(block)
except Exception:
    log.exception("failed to parse %s", url)        # logs the traceback too

Log which URL failed and why. A log line like no price on https://shop.example.com/p/881 is actionable; error is not.

Test with saved HTML

Make your parser a pure function (HTML in, records out) and test it against a folder of saved pages: the normal case, a sale item, an out-of-stock item, a page with no results, and every bug you have fixed. Because no network is involved, the tests are fast and stable.

Python
from pathlib import Path

def test_parse_sale_item():
    html = Path("fixtures/sale-item.html").read_text(encoding="utf-8")
    record = parse_listing(html)
    assert record["price"] == 19.99
    assert record["was_price"] == 24.99

When the site changes, you save the new page as a fixture, watch the test fail, adjust the selectors and watch it pass. This is also why saving raw HTML during scraping (see your toolkit) pays off.

Malformed HTML

When a page is badly broken, two things help. First, try another parser; html5lib rebuilds the tree like a browser, so a selector that works in DevTools will work (Modifying the Tree and Choosing a Parser). Second, work around the damage by selecting from stable anchors, or pre-clean the text (for example, remove a stray </div> with a regular expression) before parsing. Print soup.prettify() to see exactly what the parser made of the page.

Python Same selector, different parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

lxml and html5lib close the unclosed cells and rows and find two clean rows. html.parser never closes them, so each td nests inside the previous one and the "rows" come out as garbled, overlapping text. Same selectors, three different answers: keep your selectors loose, and test with the parser you ship.

A robustness checklist

  • Every find result is checked, or wrapped in a helper that returns a default.
  • Every conversion (int, float, date) is inside try/except.
  • Selectors live in one place, with fallbacks.
  • A page check detects blocks, login walls and empty results.
  • Records are validated and counts compared with the last run.
  • Failures are logged with the URL; raw HTML of failures is saved.
  • The parser has tests that run on saved pages.

You have now covered the whole path from raw HTML to clean, trustworthy records. The next chapters of this track move from single pages to real requests and APIs, browser automation, crawling at scale, and storing the data. Continue at the Web Scraping track page.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a 200 status not enough to trust a response?

    Many sites serve block pages and walls with a success status, so you must inspect the content.
  2. 2. Which approach best avoids losing data silently?

    Logging each skipped record and the reason lets you spot patterns and fix the parser.
  3. 3. Today's run found 0 products where yesterday found 40. What should happen?

    A sudden drop in record count is a strong signal that the layout or access changed. Protect the good data and alert yourself.
  4. 4. Why keep selectors in one named constant with fallbacks?

    Centralizing selectors and listing fallbacks makes maintenance quick and handles layout variants.

Summary

  • Expect missing fields, varied formats, layout changes, wrong pages and malformed HTML.
  • Use safe helpers that return None, convert with try/except, and record every skipped record.
  • Keep selectors in one place with fallbacks ordered from stable to fragile.
  • Classify pages before parsing, validate results, and compare counts with previous runs.
  • Log URLs and reasons, save failing HTML and test the parser on saved fixtures.

Next: The requests Library