Skip to content
Tutorials Center
My learning

Topics

Lesson 117 of 149 ยท Data and the Web

Web Scraping with requests and BeautifulSoup

Learn web scraping in Python. Fetch pages with requests, parse HTML with BeautifulSoup, select elements with CSS selectors, and scrape politely with robots.txt.

Many useful facts live on web pages that have no API: prices, schedules, headlines, tables of results. Web scraping means writing a program that downloads a page, picks out the pieces you care about and turns them into data you can use. Python is a favorite for this job because two small libraries cover almost everything: requests to fetch the page and BeautifulSoup to read it.

In this lesson you will fetch pages, parse HTML into a tree, find elements with tags and CSS selectors, pull out text, attributes and tables, and follow links. You will also learn the rules of polite scraping, because a scraper that hammers a server or ignores a site's wishes causes real problems. If you have not met HTTP yet, read Networking first, and have Packages and pip handy for installing libraries.

The two-step recipe

Every scraper does the same two things:

  1. Fetch: send an HTTP request and receive the page's HTML as text.
  2. Parse: turn that text into a tree of objects you can search.
flowchart LR
    A["Your script"] -->|"GET /page"| B["Web server"]
    B -->|"HTML text"| A
    A --> C["BeautifulSoup tree"]
    C --> D["find / select"]
    D --> E["Data: lists, dicts, CSV"]

A scraper downloads HTML, parses it into a tree, then searches the tree for the data it needs.

On your own computer, install both libraries into a virtual environment:

TEXT
python -m pip install requests beautifulsoup4

The package is named beautifulsoup4, but you import it as bs4. Fetching a page looks like this:

Python
import requests

response = requests.get(
    "https://example.com/products",
    headers={"User-Agent": "my-learning-scraper/1.0 ([email protected])"},
    timeout=10,
)
response.raise_for_status()   # raise an error for 404, 500, ...
html = response.text          # the page as a string

Always pass a timeout (without one, requests can wait forever), call raise_for_status() so a failed request is not mistaken for a real page, and send an honest User-Agent that says who you are. The examples below parse HTML held in a string so they run anywhere, including in the editors on this site, which cannot make real network requests to other sites. The parsing code is identical for a downloaded page.

Parsing HTML

Hand the HTML text to BeautifulSoup along with the name of a parser. "html.parser" is built into Python and is a good default. Messy real-world HTML is no problem: the parser repairs unclosed tags for you.

Python Parse a page and read its parts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

soup.title is shorthand for "the first <title> tag". Each tag object has .string (its text when it holds exactly one string), .get_text() (all the text inside, tags stripped) and dictionary-style access to attributes. Note that class comes back as a list (['intro']), because an element can have several classes.

Finding elements

Dot access only gets you the first match. To search properly, use these methods:

Method Returns
soup.find("li") the first <li>, or None
soup.find_all("li") a list of every <li>
soup.find_all("a", class_="buy") links with the class buy
soup.select("div.card > h2") a list of matches for a CSS selector
soup.select_one("#price") the first match for a CSS selector, or None

class_ has a trailing underscore because class is a Python keyword. If you know CSS (see the CSS selectors lessons), select() and select_one() are the most expressive tools: the same selector you would write in a stylesheet works here.

Python Scrape a product list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pattern is always the same: loop over the repeating element (here each li.item), pull out each field with a nested search, clean it up (strip=True removes surrounding whitespace, lstrip("$") removes the currency sign, float() converts to a number) and collect plain dictionaries. Once you have a list of dicts you can write them to a CSV file, a JSON file or a pandas DataFrame.

Test selectors in the browser first. Open the page, press F12, and in the console run document.querySelectorAll("li.item"). If it finds the right elements there, the same selector will work in soup.select().

Data hides in attributes as often as in text. Links are in href, images in src, product IDs in data-* attributes. Use tag["name"] when the attribute must exist, and tag.get("name") when it might not (it returns None instead of raising KeyError).

Relative links such as /books/1 need the page's address to become a full URL. urllib.parse.urljoin does that correctly:

Python Links and a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

href=True means "has this attribute at all". Turning an HTML table into a list of dictionaries by zipping the header cells with each row is a pattern you will reuse often. (For simple tables, pandas.read_html() can even do it in one call.)

Sometimes the data you want has no handy class, but it sits next to something that does. BeautifulSoup lets you walk the tree from a tag you have already found:

  • tag.parent is the enclosing tag; tag.children yields direct children.
  • tag.find_next_sibling("dd") finds the next matching element at the same level.
  • tag.find_parent("div") climbs to the nearest matching ancestor.
Python
from bs4 import BeautifulSoup

html = "<dl><dt>Author</dt><dd>Ada</dd><dt>Year</dt><dd>1843</dd></dl>"
soup = BeautifulSoup(html, "html.parser")

for dt in soup.find_all("dt"):
    print(dt.get_text(), "->", dt.find_next_sibling("dd").get_text())

This prints Author -> Ada and Year -> 1843. Label-then-value layouts like definition lists are exactly where sibling navigation shines.

Scraping politely

Scraping is only acceptable when you behave. The checklist:

  • Read robots.txt. Sites publish rules at /robots.txt saying which paths bots may visit. Python can check them for you with urllib.robotparser.
  • Read the terms of service. Some sites forbid scraping, and some data is copyrighted or personal. If the site offers an official API or a downloadable dataset, use that instead.
  • Slow down. Pause between requests (time.sleep(1) or more), never fire hundreds of parallel requests, and cache pages you have already fetched so you do not download them twice.
  • Identify yourself with a descriptive User-Agent and a contact address.
  • Stop when asked. If you get 429 Too Many Requests or a block, back off instead of trying to sneak around it.
flowchart TD
    A["Want data from a site"] --> B{"Official API<br/>or dataset?"}
    B -->|"Yes"| C["Use it"]
    B -->|"No"| D{"robots.txt and<br/>terms allow it?"}
    D -->|"No"| E["Do not scrape"]
    D -->|"Yes"| F["Scrape slowly, identify yourself, cache"]

Check for an official source first, then permission, and only then scrape, slowly.

The standard library's robotparser answers "may I fetch this URL?". You can feed it the text of a robots.txt file directly, which makes it easy to try:

Python Check robots.txt rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a real site, call parser.set_url("https://site.example/robots.txt") and parser.read() once, then ask can_fetch() before every request. Putting it all together, a polite scraper looks like this (read-only, because it needs a network):

Python
import time
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "my-learning-scraper/1.0 ([email protected])"}

def scrape_titles(urls):
    titles = []
    with requests.Session() as session:       # reuses the connection
        session.headers.update(HEADERS)
        for url in urls:
            response = session.get(url, timeout=10)
            if response.status_code != 200:
                print("Skipping", url, response.status_code)
                continue
            soup = BeautifulSoup(response.text, "html.parser")
            titles.append(soup.title.get_text(strip=True))
            time.sleep(2)                     # be gentle
    return titles

What scraping cannot do easily

Some pages build their content with JavaScript after loading, so the HTML you download is nearly empty. Signs of this: your find() returns None for something you can plainly see in the browser. Before reaching for a headless browser such as Playwright, open the browser's Network tab and look for a JSON request that supplies the data. Calling that endpoint directly with requests is simpler, faster and gives you clean JSON. Also expect pages to change: write scrapers defensively, check for None before calling methods, and handle exceptions so one odd page does not end a long run.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the import name for the package installed with pip install beautifulsoup4?

    The distribution is called beautifulsoup4 but you write from bs4 import BeautifulSoup.
  2. 2. What does soup.select_one(\"div.card h2\") return when nothing matches?

    select_one() and find() return None when nothing matches, while select() and find_all() return an empty list. Check for None before calling methods on the result.
  3. 3. Why should every requests.get() call include a timeout?

    By default requests does not time out, so a stalled server could freeze your scraper indefinitely.
  4. 4. Which of these is NOT part of scraping politely?

    Flooding a server with parallel requests can slow it down for everyone. Throttle yourself, cache pages and respect the site's rules.
  5. 5. A relative link /about was found on https://shop.example/catalog/. How do you get the full URL?

    urljoin applies the URL resolution rules, so /about becomes https://shop.example/about and page2.html becomes https://shop.example/catalog/page2.html.

Summary

  • Scraping is two steps: fetch with requests (with a timeout, raise_for_status() and a clear User-Agent), then parse with BeautifulSoup(html, "html.parser").
  • Search with find(), find_all(), select() and select_one(); read text with get_text(strip=True) and attributes with tag["href"] or tag.get().
  • Turn repeating elements and tables into lists of dictionaries, then save them as CSV, JSON or a DataFrame.
  • Use urljoin for relative links and navigate with parent, find_next_sibling() and friends.
  • Prefer official APIs, respect robots.txt and terms of service, slow down and cache.
  • If the data is loaded by JavaScript, look for the underlying JSON request first.

Next: Image Processing with Pillow.