Skip to content
Tutorials Center
My learning

Topics

Lesson 86 of 147 ยท PHP and the Web

Parsing HTML and Scraping with DOMDocument and XPath

Parse HTML in PHP with DOMDocument, DOMXPath and PHP 8.4's Dom\HTMLDocument - extract links, images and prices, resolve URLs, fetch pages with cURL and scrape politely.

Sooner or later you need data that only exists inside a web page: the price list of a supplier, the headings of your own pages for a site audit, the links in an HTML e-mail. Regular expressions are the tempting tool, and they are the wrong one. HTML is not a regular language: attributes can appear in any order, tags can be unclosed, and a single > inside an attribute breaks most patterns. PHP ships a real parser instead: the DOM extension. You load the HTML into a document tree, then find what you want with the tree methods or with XPath, a small query language for trees.

This lesson uses DOMDocument and DOMXPath (available everywhere), then shows the modern Dom\HTMLDocument from PHP 8.4, which understands HTML5 and CSS selectors. It finishes with fetching pages over HTTP and the rules of polite scraping. Related reading: XML (the same DOM classes work on XML), HTTP Headers and Regular Expressions.

Only scrape pages you are allowed to read. Check the site's terms and its robots.txt, prefer an official API (REST APIs), identify your program and keep the request rate low. Scraping content and republishing it can also violate copyright.

Loading HTML into a DOMDocument

DOMDocument::loadHTML() accepts a string. Real-world HTML is rarely valid, so the parser prints warnings for every problem it repairs. Turn those into a collectable list with libxml_use_internal_errors(true). Another quirk: the classic parser assumes ISO-8859-1 unless told otherwise, so UTF-8 text turns into garbage. The simple fix is to prefix the string with an XML encoding declaration.

PHP Load HTML and read the title

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

getElementsByTagName() returns a live list you can loop over, getAttribute() returns an empty string for missing attributes (use hasAttribute() to tell the difference) and textContent is all the text inside a node with the tags stripped. The unclosed <b> and <i> were repaired silently, which is the whole point of using a parser.

XPath: query the tree

XPath describes a path through the document. //li means "every li anywhere", //ul[@id="list"]/li means "li children of the ul with that id", @href selects an attribute and text() the text. DOMXPath::query() returns a node list, and evaluate() returns a scalar when the expression produces one (string(...), count(...)).

Goal XPath
all links //a
link addresses //a/@href
items with class sale //li[contains(concat(' ', normalize-space(@class), ' '), ' sale ')]
second list item //ul/li[2]
text of the price inside each item .//span[@class="price"] (note the leading dot for a relative path)
number of images count(//img)
meta description //meta[@name="description"]/@content

The class test looks clumsy because XPath 1.0 has no "has class" operator: the concat trick makes sale match "item sale" but not "onsale".

PHP Extract products with XPath

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

A scraper that audits your own pages can flag images without alt text as shown. Always select values from a specific node using the second argument of evaluate() and a path starting with a dot, otherwise a query like //span will search the whole document again for each item.

The modern way: Dom\HTMLDocument (PHP 8.4)

PHP 8.4 added a second DOM API in the Dom namespace. Dom\HTMLDocument::createFromString() follows the HTML5 parsing rules like a browser does, handles UTF-8 properly, and has querySelector() and querySelectorAll() with CSS selectors, so a lot of XPath becomes the same selectors you know from CSS. Use it when you can require PHP 8.4 or later.

PHP CSS selectors with Dom\HTMLDocument

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

Scraped href values can be absolute (https://example.com/p/2), root-relative (/p/1) or relative (p/3). Before you follow them you need absolute URLs. A small resolver handles the three cases; for full RFC 3986 behaviour (dot segments, queries) PHP 8.5's Uri\Rfc3986\Uri::resolve() or a library does it for you.

PHP Resolve relative URLs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

Fetching a page: cURL

The runner behind this site has no outgoing network, so the fetching code is shown read-only. file_get_contents($url) works for simple cases when allow_url_fopen is on; cURL gives you timeouts, redirects, headers and error codes, which is what you want in a real scraper.

PHP
function fetch(string $url): ?string
{
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_MAXREDIRS      => 3,
        CURLOPT_CONNECTTIMEOUT => 5,
        CURLOPT_TIMEOUT        => 15,
        CURLOPT_USERAGENT      => 'MyCompanyBot/1.0 (+https://example.com/bot; [email protected])',
        CURLOPT_HTTPHEADER     => ['Accept: text/html'],
        CURLOPT_PROTOCOLS      => CURLPROTO_HTTP | CURLPROTO_HTTPS,   // no file://, ftp://
    ]);
    $body = curl_exec($ch);
    $status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);

    if ($body === false || $status !== 200) {
        return null;
    }
    return $body;
}

$html = fetch('https://example.com/shop/list.html');

The two security lines matter. Never fetch a URL that came from a user without restricting the protocols and checking that the host is not on your private network (a "server-side request forgery" attack). See Secure Coding.

Scraping politely

  • Read robots.txt and respect Disallow rules and Crawl-delay.
  • Identify yourself with a descriptive User-Agent and a contact address, as in the code above.
  • Go slowly: sleep between requests (one per second is a sensible default), and stop on 429 or 503 responses.
  • Cache pages you have fetched (see Caching Strategies) so that while you develop the extraction code you do not hit the site again.
  • Expect change. Class names and layouts change without notice. Keep selectors in one place, validate that you got the fields you expected, and fail loudly instead of saving empty data.
  • Prefer an API or a feed whenever one exists; it is more stable and more polite.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a DOM parser better than regular expressions for HTML?

    Patterns break on valid variations of markup. A parser builds the same tree a browser would and lets you query it reliably.
  2. 2. What does libxml_use_internal_errors(true) do?

    Real-world HTML is rarely valid. The parser repairs it and reports problems through libxml_get_errors() when this option is on.
  3. 3. In XPath, why does the path for the price inside each item start with a dot (.//span)?

    Without the dot, //span searches from the document root, so every item would return the first price on the page.
  4. 4. Which is an important rule for polite scraping?

    Check what the site allows, identify yourself and go slowly. Stop when the server asks you to slow down.

Summary

  • Parse HTML with DOMDocument::loadHTML(); collect warnings with libxml_use_internal_errors(true) and declare UTF-8 explicitly.
  • Query the tree with getElementsByTagName() or, more powerfully, with DOMXPath::query() and evaluate().
  • Use relative XPath (.//x) inside loops; select classes with the concat/normalize-space pattern.
  • PHP 8.4's Dom\HTMLDocument offers HTML5 parsing and CSS selectors.
  • Resolve relative links against the page URL before following them.
  • Fetch with cURL using timeouts, a clear User-Agent and restricted protocols; scrape slowly and cache.

Next: Templating Engines