Skip to content
Tutorials Center
My learning

Topics

Lesson 69 of 76 ยท Projects

Project: Scrape and Analyse Product Prices

Build a small scraping and analysis pipeline: fetch paginated product pages politely, parse HTML with BeautifulSoup, clean messy prices, track price changes and report the findings.

Price data rarely arrives as a tidy file. It lives on web pages, spread over many pages, written for humans ("$1,299.00", "Out of stock", "from 15.50"), and it changes every day. A common first real-world data project is therefore a pipeline: collect pages, parse them into rows, clean the values and analyse what changed.

In this project you will build that pipeline end to end. To keep it safe and repeatable we will not hit a real website. Instead the code fetches pages from a small simulated shop, a Python function that returns HTML exactly the way a server would. Everything after the fetch (parsing, cleaning, analysis) is identical to what you would do on a real site. Read Collecting Data from APIs and Web Pages first for the ground rules.

Before scraping any real site, read its terms of service and robots.txt, identify your program with a User-Agent, go slowly and collect only what you need. Prefer an official API when one exists, and never collect personal data without a lawful reason.

The plan

flowchart LR
    A["Fetch pages<br/>polite, with delay"] --> B["Save raw HTML<br/>so you can re-parse"]
    B --> C["Parse<br/>BeautifulSoup to rows"]
    C --> D["Clean<br/>prices, stock, types"]
    D --> E["Analyse<br/>distributions, changes"]
    E --> F["Report<br/>findings and caveats"]

Keep each stage separate, so a change to the parser never forces a new download.

The questions we want to answer:

  1. How many products are there, and how are prices spread within each category?
  2. Are there suspicious prices (typos, placeholders)?
  3. Between two scrape dates, which products got cheaper or more expensive, and by how much?

Step 1: Fetch and parse

The simulated shop has three pages of a laptop and accessory catalogue. Each page lists products in a div.product block with a name, a price string and a stock label, and a "next" link leads to the following page. The loop follows that link until it ends, waits between requests, and keeps the raw HTML.

Python Fetching paginated pages and parsing them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two design choices matter. The scraper stores the raw HTML, so you can fix a parsing bug without downloading again, and it follows the next link instead of guessing page numbers, so it stops by itself when the site stops.

Step 2: Clean the messy values

The text columns hold human formatting. Some problems we can see at once: thousands separators, a "from" prefix that means a minimum price, a $0.00 placeholder, a typo ($1,2990.00) and stock labels that mix a status and a number. Cleaning rules go in code, are documented, and flag rather than silently fix what they cannot know.

Python Cleaning prices and stock labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "from" price is kept but annotated, the $0.00 placeholder is flagged as suspect, the typo with an extra digit does not match the price pattern and becomes a missing value with the note "unparseable", and no row is deleted. Deciding what to do with a flagged row (drop it, fix it from another source, ask the shop) is an analysis decision, not a parsing accident. That is the habit from Outliers and Errors.

Step 3: Analyse prices and changes

With clean rows from two scrape dates we can answer the questions. We join the days on the product name, compute the change, and summarise per category. Rows flagged as suspect are excluded from the statistics but listed in the report.

Python Price spread and day-to-day changes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what the analysis does not say. The Glide pad went from $0.00 to $12.00, a "+infinity percent" change that is really a placeholder being replaced, so it is excluded rather than reported. The Iris monitor's first price was unparseable, so no change can be computed. Good data work states these exclusions instead of hiding them.

Step 4: Turn it into a repeatable job

A one-off script becomes useful when it can run every day and keep history. The structure is small:

Python
# Read-only sketch of the daily job
def run_daily(site_fetch, today):
    df_raw, html_pages = scrape(site_fetch)          # step 1
    save_raw(html_pages, f"raw/{today}/")             # keep the HTML
    df_clean = clean(df_raw)                          # step 2
    df_clean["scraped_on"] = today
    append_to_history(df_clean, "prices.parquet")     # one table, one row per product per day
    return df_clean

Run it from a scheduler (cron, a cloud scheduler, a GitHub Action) and add checks: the number of products should not drop by half overnight, parse failures should stay below a threshold, and a human should be notified when they do not. See Reproducible Projects and Testing Data Code. When the site changes its HTML, the parser breaks first; because you kept the raw pages, you can fix it and re-parse the history.

The written conclusion

Finding: In the catalogue of nine products, laptops span roughly 900 to 1,550 dollars, while accessories cost under 80. Between the two scrapes, 4 of the 7 comparable products (57%) changed price: one laptop, the dock and a monitor became cheaper, while another laptop rose by about 6.5%. Evidence: Two scrapes of the full catalogue, parsed and cleaned with documented rules. Two products were excluded from the comparison: one with a placeholder price of zero and one with an unparseable price. Caveats: Two days show a snapshot, not a trend; prices are list prices without discounts or shipping; one "from" price is a minimum, not a fixed price. Next steps: Scrape daily for a month, add the stock level to the history and look for price drops that follow stock-outs.

State the exclusions and the data limits in the conclusion itself. A reader who knows what was left out can trust what remains.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the scraper save the raw HTML of every page?

    Separating collection from parsing means a change in your parser never needs new requests, and you keep the evidence of what the site showed.
  2. 2. A scraped price is the text 'from $79'. What is the best handling?

    The number is useful, but it means something different from a fixed price, so record that difference.
  3. 3. Why follow the 'next' link instead of requesting /page/1, /page/2, ... until an error?

    The page itself says whether there is a next page, which avoids missing pages and avoids pointless extra requests.
  4. 4. Which is the main reason for flagging suspicious prices instead of deleting them?

    A silent deletion hides a data problem. A flag keeps the evidence and lets you decide to fix, exclude or ask the source.

Summary

  • A scraping project is a pipeline: fetch politely, keep the raw HTML, parse into rows, clean with documented rules, analyse, report.
  • Follow pagination links, wait between requests, and respect robots.txt and terms of service.
  • Clean prices and stock labels with explicit patterns; flag implausible or unparseable values rather than guessing or deleting.
  • Compare scrapes by joining on a stable key, and exclude rows whose changes would be meaningless, saying so in the report.
  • Make the job repeatable with a scheduler, history storage and sanity checks, and write conclusions with evidence and caveats.

Next: Practice Exercises.