Skip to content
Tutorials Center
My learning

Topics

Lesson 65 of 72 ยท Projects

Project: Web Scraping API

Build a Flask API that extracts data from web pages. Parse HTML with BeautifulSoup, respect robots.txt, block SSRF with a host allowlist, cache results and return clean JSON.

Sometimes the data you need is on a web page and there is no API for it: product names on a catalog, headlines on a news index, entries in a directory. Web scraping means downloading the page and extracting the pieces you need from its HTML. In this project you will build a small Flask service that does it on request and returns clean JSON, so other programs can use the data without ever touching HTML.

Scraping has rules, and the project is built around them: check the site's terms, obey robots.txt, identify yourself, go slowly, cache, and protect your own server from being turned into a tool for attackers. If the site offers an official API or a data export, use that instead. It is more stable and clearly allowed.

Only scrape pages you are allowed to scrape. Read the site's terms of service and robots.txt, never bypass logins or technical protections, do not collect personal data, and do not overload a server. The examples here parse small HTML strings that stand in for pages, so nothing in this lesson touches a real website.

What a scraper does

flowchart LR
    A["Client:<br/>GET /api/books?page=1"] --> B["Flask API"]
    B --> C{"allowed host?<br/>allowed by robots.txt?"}
    C -- no --> X["403 / 400"]
    C -- yes --> D["requests.get<br/>with timeout"]
    D --> E["BeautifulSoup<br/>parse HTML"]
    E --> F["extract fields<br/>to dicts"]
    F --> G["cache + JSON response"]

Download, parse, extract, return JSON. Everything before the download is about being allowed and safe.

Step 1: parse HTML with BeautifulSoup

BeautifulSoup (pip install beautifulsoup4) turns HTML into a tree you can search with CSS selectors. The most useful methods are select() (all matches), select_one() (first match or None), get_text() and element["attribute"].

The page below imitates a small bookshop's listing. Look at how the code deals with the things that go wrong on real pages: missing elements, extra whitespace and relative links.

Python Extract books from HTML

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Habits that make scrapers survive:

  • Use select_one() and check for None before using an element. A single missing price must not crash the whole page.
  • Prefer attributes and stable class names over text position; avoid selectors like div > div > div:nth-child(3).
  • Make relative links absolute with urljoin.
  • Convert text to real types (float, bool, int) in one place.
  • Pages change. Keep the parsing in one function with a test that runs on a saved sample page, so you notice when the structure changes.

Step 2: be polite: robots.txt

robots.txt at the root of a site tells crawlers which paths they may fetch. Python's standard library can read it: urllib.robotparser. Fetch it once, cache it, and check every URL before you request it. The same file may declare a Crawl-delay.

Python Check robots.txt rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note that /cartoons is blocked too: Disallow: /cart is a path prefix, so it also matches /cartoons. Never rely on your own reading of the file; call can_fetch and honor the answer. Also wait at least the crawl delay between requests to the same host, and send a User-Agent that names your project and a way to contact you.

Step 3: protect your own server (SSRF)

An endpoint that takes a URL from the caller and fetches it is dangerous. An attacker can ask it for http://127.0.0.1:8080/admin, http://169.254.169.254/latest/meta-data/ (cloud credentials) or other machines on your private network, and your server will happily fetch them and hand back the response. This is server-side request forgery (SSRF).

The safest design is not to accept arbitrary URLs at all: the API takes a site name and parameters, and your code builds the URL from an allowlist of hosts you have chosen. If you must accept URLs, check the scheme (https only), the host against the allowlist, and block private addresses, including after redirects.

Step 4: put it together

The service below exposes GET /api/books?page=1. The API never sees a full URL from the client: the page number selects a path on the one allowed host. A fake network returns the HTML above, so you can run it, test it and see every behavior: robots check, parsing, caching and error handling.

Python The scraping API, tested offline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the last line: a caller who tries to smuggle in a url= parameter is simply ignored, because no code path reads it. That is what "design out" a vulnerability means.

Where this can grow

  • Scheduled scraping. Scrape on a timer in a background job (Async Views and Background Tasks), store results in your database (Flask-SQLAlchemy) and let the API read your own data. Then your response time and the target site's load are independent of your traffic.
  • JavaScript-rendered pages. requests only gets the HTML the server sends. If the data is loaded by JavaScript, look in the browser's network tab for the JSON endpoint it calls (often simpler and more stable than HTML), or use a headless browser such as Playwright in a worker.
  • Rate limiting your API with Flask-Limiter, and API-key authentication (API Authentication).
  • Change detection. Compare the parsed result with the previous one and alert when the structure suddenly yields zero items.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does soup.select_one('p.price') return when no such element exists?

    select_one returns None for no match. Calling methods on None raises AttributeError, so guard against it.
  2. 2. What is SSRF in the context of a scraping endpoint?

    If the API fetches any URL it is given, attackers can reach localhost and cloud metadata services through your server.
  3. 3. What is the safest way to avoid SSRF here?

    Designing the API so clients cannot choose the host removes the whole class of problem.
  4. 4. Which standard library module checks whether a path may be fetched according to robots.txt?

    RobotFileParser reads the rules, and can_fetch(agent, url) answers whether the URL is allowed.

Summary

  • Prefer an official API; scrape only where allowed, politely, with a clear User-Agent, timeouts, caching and robots.txt checks.
  • Parse with BeautifulSoup selectors, guard against missing elements, resolve relative links and convert types in one place.
  • Never fetch caller-supplied URLs: build URLs from fixed hosts and validated parameters to prevent SSRF.
  • Return clean JSON with sensible statuses (400 bad input, 403 disallowed, 404 missing, 502 upstream failure) and cache results.
  • Keep parsing testable with saved sample pages, because sites change their HTML without warning.

Next: build a content site with a contact form in Project: Single-Page Portfolio Site.