Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 39 ยท Selectors and Data Extraction

Regular Expressions in Scraping

Use Python regular expressions alongside parsers: search, findall, groups, named groups, extracting prices, phone numbers, dates and IDs from text, and when not to use regex.

Selectors find elements. Regular expressions find patterns inside text. Scrapers need both: a selector grabs the paragraph, and a regex pulls the price, date, phone number or product ID out of it. Regex also works on text that has no useful structure, such as an embedded JSON-like blob inside a script tag.

There is a famous warning against parsing HTML itself with regular expressions, because HTML is nested and irregular. The rule of thumb for scrapers: use a parser for structure and regex for the text you extract from it.

The re module in five functions

Function Purpose
re.search(p, s) first match anywhere, or None
re.match(p, s) match only at the start of the string
re.fullmatch(p, s) the whole string must match
re.findall(p, s) list of all matches (or of groups)
re.finditer(p, s) iterator of match objects (with positions)
re.sub(p, repl, s) replace matches
re.split(p, s) split on a pattern

Always write patterns as raw strings (r"\d+") so backslashes reach the regex engine untouched.

Pattern cheat sheet

Pattern Matches
\d, \w, \s digit, word character, whitespace
\D, \W, \S the opposites
. any character except newline
[abc], [a-z], [^0-9] a set, a range, a negated set
*, +, ? 0 or more, 1 or more, 0 or 1
, {2,4} exactly 3, between 2 and 4
*?, +? lazy (shortest) versions
^, $ start and end of string
\b word boundary
( ) capture group
(?P<name> ) named group
(?: ) group without capturing
a|b alternation

Extracting prices, emails and phone numbers

Python Pull values out of text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note how findall returns the groups when the pattern has one capture group (the order IDs without #) and the whole match when it has none. A non-capturing group (?:...) groups without affecting what findall returns, which is why the email and phone patterns return whole matches.

Groups and named groups

Parentheses capture parts of a match so you can break text into fields. Named groups make the code self-documenting.

Python Named groups and match objects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

re.compile pre-builds the pattern, which is faster inside loops and gives the pattern a name. Always check if m: before using the result, because search returns None when nothing matches, and None["rent"] is an error.

Greedy versus lazy

Quantifiers are greedy by default: they match as much as possible. When text contains repeated delimiters, that grabs too much.

Python Greedy versus lazy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last form, [^<]*, is clearer and faster: "any characters that are not an opening angle bracket". Prefer negated sets to .*? whenever the delimiter is a single character.

Cleaning text with sub and split

re.sub replaces patterns, which makes it perfect for normalizing messy scraped text:

Python Normalize messy text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Backreferences (\1, \2) in the replacement reuse captured groups, and (?i) turns on case-insensitive matching for the rest of the pattern (or pass re.IGNORECASE as a flag).

Using regex with selectors

You can pass a compiled regex straight to BeautifulSoup filters:

Python Regex inside find_all

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A two-step process (selector first, regex second) is the usual best practice: the selector narrows the search to a small piece of text, and the regex parses it.

Regex on embedded scripts

Some pages hide data in a <script> block as a JavaScript assignment. Regex can capture the object, and json.loads turns it into data:

Python
import re, json

script_text = "window.__DATA__ = {\"price\": 24.5, \"sku\": \"A1\"};"
m = re.search(r"window\.__DATA__\s*=\s*(\{.*?\});", script_text, re.S)
data = json.loads(m.group(1))
print(data["price"])

This works when the data is valid JSON. For nested braces, prefer a proper JSON parse of the whole blob, as in Extracting Links, Images and Metadata.

When not to use regex

  • Parsing nested HTML. Use BeautifulSoup or lxml.
  • Overly clever patterns. A 200-character regex nobody can read is a maintenance burden. Split the job into steps, use re.VERBOSE and add comments.
  • Dates in many formats. Use datetime.strptime or the dateutil package.
  • Catastrophic backtracking. Nested quantifiers such as (a+)+b can take exponential time on adversarial input. Keep patterns simple and anchored.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does re.findall(r'#(\w+)', 'a #x b #y') return?

    With exactly one capture group, findall returns the contents of that group for each match, without the # that sits outside it.
  2. 2. Why use a raw string such as r'\d+' for patterns?

    In a normal string, backslash sequences are interpreted by Python first, which can change or break the pattern.
  3. 3. Which pattern is lazy (shortest match)?

    Adding ? after a quantifier makes it lazy, so it stops at the first possible end.
  4. 4. What is the recommended division of labor between a parser and regex?

    A parser understands nesting; regex excels at pulling patterns such as prices and IDs out of the text you extracted.

Summary

  • Use re.search, findall, finditer, sub and split with raw-string patterns.
  • Capture groups (( ), named (?P<name>)) split text into fields; check for None before using a match.
  • Prefer negated sets ([^<]*) or lazy quantifiers to avoid over-matching.
  • Use re.sub to normalize whitespace and strip symbols.
  • Combine a selector (to find the text) with regex (to parse it); never parse nested HTML with regex alone.

Next: Extracting Tables and Lists