Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 39 ยท Selectors and Data Extraction

XPath with lxml

Learn XPath for scraping: lxml.html parsing, axes, predicates, text() and contains(), attribute extraction, relative paths, and when XPath beats CSS selectors.

XPath is a query language for tree documents. Where CSS selectors describe which elements to match, XPath describes a path through the tree and can also return text, attributes and computed values. It is more powerful than CSS in a few important ways: it can move up to parents, select by text content, and use functions like contains() and normalize-space(). In Python you use XPath through lxml, the fast C-based parser that also underlies the "lxml" parser option of BeautifulSoup.

Parsing with lxml.html

lxml.html.fromstring() parses a string into an element tree. .xpath() runs a query.

Python Your first XPath queries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Important: xpath() always returns a list. Depending on the expression, items are elements, strings (for text() and @attr) or numbers/booleans (for functions like count()). Empty list means no match; there is no None.

Path syntax

XPath Meaning
//div every div anywhere in the document
/html/body/div absolute path from the root
.//div every div below the current element (relative)
div/p p children of div
div//p p descendants of div
* any element
. and .. current node and parent
@href the href attribute
text() the node's own text pieces
a | b union of two paths

The most common beginner trap is // inside a loop. //span always searches from the document root, even when called on a sub-element. Use .//span to search only inside the current element:

Python // versus .//

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Predicates: filtering with [ ]

Square brackets filter the nodes selected so far.

Predicate Meaning
[1] the first (XPath counts from 1)
[last()] the last
[position() < 3] the first two
[@id] has an id attribute
[@class='card'] attribute equals (whole string)
[contains(@class, 'card')] attribute contains
[starts-with(@href, '/p/')] attribute starts with
[contains(text(), 'Sale')] text contains
[normalize-space()='Sale'] text equals after trimming
[price > 10] numeric comparison on a child
[a and span] has both children
[not(@disabled)] negation
Python Predicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The class trap: @class='sale' only matches when the entire class string is sale. For an element with class="product sale", use the word-safe pattern contains(concat(' ', normalize-space(@class), ' '), ' sale '), which avoids false matches such as unsale. For quick scripts contains(@class, 'sale') is usually enough; for fragile pages use the safe form.

Axes: moving in any direction

An axis says which relatives of the current node to look at. This is where XPath beats CSS selectors.

Axis Selects
parent:: or .. the parent
ancestor::div all ancestor divs
following-sibling:: later siblings
preceding-sibling:: earlier siblings
following:: everything after, in document order
descendant:: all descendants
Python Label and value with axes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two lines express "the value next to this label" in one query. The third goes up from a badge to its card and then down to the title, which a CSS selector cannot do (CSS has :has() for going the other way, but cannot return the parent).

Text, string() and normalize-space()

text() returns each text node separately, which breaks when a tag mixes text and children. Use string(...) to get all text of an element joined, and normalize-space(...) to also collapse whitespace:

Python Getting clean text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Using XPath with BeautifulSoup

BeautifulSoup does not support XPath itself. Two common solutions: use lxml.html for the whole job, or hand the HTML to lxml when you need an XPath query:

Python
from bs4 import BeautifulSoup
from lxml import etree

soup = BeautifulSoup(page, "lxml")
dom = etree.HTML(str(soup))
dom.xpath("//span[@class='price']/text()")

Another route is the parsel library (made for Scrapy), which offers Selector(text=html).xpath(...) and .css(...) on one object, plus .get() and .getall(). In Scrapy itself you will use response.xpath(...) and response.css(...).

XPath or CSS?

Need Use
Simple class, id, tag, attribute CSS: shorter and familiar
Match by text content XPath
Go up to a parent or ancestor XPath
Complex label/value patterns XPath axes
Compute counts or numeric comparisons XPath
Easy copy from the browser console Both: DevTools can copy XPath too

Many scrapers use both. A good habit is to start with CSS and reach for XPath only for the cases CSS cannot express.

In Chrome DevTools, press Ctrl+F in the Elements panel and type an XPath or CSS selector. It highlights the matching nodes and shows the count, so you can try expressions live before putting them in code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Inside a loop over cards, which expression searches only within the current card?

    .//span starts from the current element. //span always starts at the document root, even when called on a sub-element.
  2. 2. What does //li[1] select?

    XPath positions start at 1, and li[1] means the first li among each parent's li children.
  3. 3. Which axis lets you select the label's value that follows it at the same level?

    following-sibling selects later nodes that share the same parent.
  4. 4. What does xpath('//a/@href') return?

    Attribute and text() queries return lists of strings; an empty list means no match.

Summary

  • lxml.html.fromstring(page).xpath(...) always returns a list of elements, strings or numbers.
  • Use .// for relative searches and remember XPath positions start at 1.
  • Predicates filter by attributes, text, position and children; contains, starts-with and normalize-space do most of the work.
  • Axes (following-sibling, ancestor, preceding) go places CSS cannot, including up to a parent.
  • Use string() or normalize-space() for clean text; combine XPath with BeautifulSoup through lxml or parsel.

Next: Regular Expressions in Scraping