Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 39 ยท HTTP and HTML Basics

HTML Structure for Scrapers

Learn to read HTML the way a scraper must: elements, attributes, the DOM tree, ids and classes, tables and lists, data attributes, and how to inspect a page.

To scrape a page, you must know where the thing you want lives inside the HTML. You do not need to be a web designer, but you do need to read markup fluently: what an element is, how elements nest, and which clues (ids, classes, attributes) identify the right one. This lesson covers the HTML a scraper needs, with a focus on spotting patterns.

If you want the full language, see the HTML tutorial. Here we concentrate on reading structure.

Elements, tags and attributes

HTML is made of elements. Most have a start tag, content and an end tag:

HTML
<a href="/books/dune" class="title" id="first-link">Dune</a>
  • a is the tag name.
  • href, class and id are attributes, written name="value".
  • Dune is the text content.
  • <a ...> and </a> are the start and end tags.

Some elements are void: they have no content and no end tag, such as <img src="x.jpg" alt="...">, <br> and <input>.

The attributes scrapers use most:

Attribute Why it matters
href Link targets: where to crawl next
src Image, script and media URLs
class Marks groups of similar elements (every product card)
id Unique element on the page
data-* Custom data: data-price="9.99", often cleaner than visible text
alt, title Extra text for images and tooltips
content Value in <meta> tags (descriptions, Open Graph data)
datetime Machine-readable dates in <time>

The DOM tree

The browser turns the text into a tree called the Document Object Model (DOM). Every element is a node; nested elements are children. Parsers like BeautifulSoup build the same kind of tree, and your code navigates it.

HTML
<ul id="menu">
  <li class="item">Home</li>
  <li class="item">Books</li>
  <li class="item active">About</li>
</ul>
flowchart TD
    UL["ul #menu"] --> L1["li .item<br/>Home"]
    UL --> L2["li .item<br/>Books"]
    UL --> L3["li .item .active<br/>About"]

Parent, children and siblings: ul is the parent of three li children, which are siblings of each other.

Family words matter because scraping selectors use them: parent, child, descendant (any depth below), ancestor, and sibling (same parent).

ids and classes

  • An id should appear once per page. <div id="price-box"> is the most reliable handle when it exists.
  • A class can be shared by many elements and an element can have several: class="item active". Classes are how you find "every product card".
  • Class names generated by build tools, such as css-1x9z3kq, change whenever the site is redeployed. Prefer meaningful classes, ids, data-* attributes and structure to those random names.

Common structural patterns

Real pages are built from repeating patterns. Recognize them and you recognize what to loop over.

Lists of items (ul/ol with li), where each li is one record.

Cards (div or article with a class), each containing a title, price and link.

Tables (table, tr rows, th header cells, td data cells), which map directly to rows and columns.

Definition lists (dl, dt, dd), used for label/value pairs such as specs.

Navigation and pagination (nav, a rel="next"), the way to find the next page to crawl.

Metadata in <head>: <title>, <meta name="description">, <link rel="canonical">, Open Graph <meta property="og:...">.

Structured data in <script type="application/ld+json">: JSON describing products, recipes, articles, events. It is a gift for scrapers because it is already clean data. Extracting Links, Images and Metadata shows how to read it.

Inspect before you code

The best workflow: open the page, right-click the thing you want, choose Inspect, and read the surrounding HTML. Ask:

  1. What element holds the value? What is its tag, class or id?
  2. Is there a repeating parent (a card) that holds all the fields of one record?
  3. Is the same pattern used for every item, including odd ones (sale prices, missing images)?
  4. Does the value appear in View Source (what your scraper gets), or only in the Elements panel (the DOM after JavaScript ran)?

That last check is crucial. View Source shows the raw response. Inspect shows the live DOM, which JavaScript may have changed. If the value is missing from View Source, a plain request will not find it.

Reading a page by eye

Study this snippet as a scraper would. The goal is to name the repeating unit and each field's handle.

Python Map a page, then list its structure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The outline function prints the structure so you can see the tree. Notice that the clean values (data-min, datetime) live in attributes, not in the visible text $52k - $64k or Mar 14. Prefer such attributes when they exist: they are meant for machines.

Whitespace, entities and nested text

Three details cause surprises:

  • Whitespace. The newlines and indentation between tags are text nodes. Strip them with get_text(strip=True).
  • Entities. &amp; means &, &lt; means <, &nbsp; is a non-breaking space. Parsers decode them for you: AT&amp;T becomes AT&T.
  • Nested text. <p>Price: <b>$5</b> only</p> has text in the p and in the b. Decide whether you want one or both.
Python Entities, whitespace and nesting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the first output: it contains the literal text with extra spaces and a \xa0 (the non-breaking space). get_text(" ", strip=True) joins the pieces with a space and trims them. Replacing \xa0 is a typical clean-up step you will repeat.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which attribute should be unique within a page?

    An id identifies a single element; classes are meant to be shared.
  2. 2. A value shows in the browser's Inspect panel but not in View Source. What does that mean?

    View Source is the raw response; Inspect shows the DOM after scripts ran. A plain HTTP request only sees the raw response.
  3. 3. Why prefer data-price='9.99' over the visible text '$9.99 (save 10%)'?

    Attributes designed for machines avoid symbols, labels and formatting that you would otherwise have to clean up.
  4. 4. What is the relationship between two li elements inside the same ul?

    Elements sharing the same parent are siblings.

Summary

  • Elements have tags, attributes and content, and form a DOM tree of parents, children and siblings.
  • id is unique, class marks groups, and data-*, datetime and content attributes hold clean values.
  • Pages are made of repeating patterns: lists, cards, tables, pagination and structured data.
  • Use Inspect to find the handle, and View Source to confirm the data is in the raw response.
  • Watch for whitespace, entities (&nbsp;) and nested text when reading values.

Next: URLs and Query Strings