Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 39 ยท Projects and Practice

Project: Book Catalog Scraper

Build a complete scraper from crawl to report: walk paginated category pages, parse book cards, clean prices and ratings, store in SQLite and analyze the results.

Time to put the whole course together. In this project you build a complete scraper for a small online bookshop: it walks the category pages, follows the pagination, opens each book's detail page, cleans the values, saves everything to a SQLite database and finishes with a short report.

The bookshop is fictional and lives inside the lesson as a Python dictionary of HTML pages, so the project runs anywhere and gives identical results every time. When you copy it to your own machine and aim it at a real, permitted site, you change exactly one function (fetch). That is the point of the structure: fetching, crawling, parsing, cleaning and storing are separate pieces.

flowchart LR
  A["fetch(url)"] --> B["crawl listing pages"]
  B --> C["parse_card()"]
  C --> D["parse_detail()"]
  D --> E["clean()"]
  E --> F["SQLite upsert"]
  F --> G["report"]

The plan

  1. Fake site and fetch.
  2. Parse a listing page into cards and find the next page.
  3. Parse a detail page for the extra fields.
  4. Clean the values.
  5. Store with an upsert and run twice to prove it is repeatable.
  6. Report.

Each step is its own TryIt that builds on the previous one. Run them in order; if one fails, read the error and compare with the earlier code. (Each editor is separate, so every block repeats the definitions it needs.)

Step 1 and 2: the site and the listing parser

The site has two listing pages and four books. Notice the messy details we will have to deal with: ยฃ price symbols, a missing rating, different spellings of stock status, and a relative "next" link.

Python Listing pages: cards and next link

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two things worth noticing. We read the full title from the title attribute, because the visible text can be truncated (Ulyssesโ€ฆ). And the rating is not text at all: it is a CSS class, Four, so we take the class that is not rating. Selectors like these are typical of real sites.

Step 3 and 4: detail pages and cleaning

Detail pages hold the description and a product table. We add a parser for them and the cleaning functions from Cleaning scraped data.

Python Detail parser and cleaning

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table parser builds a dictionary from the th/td pairs, which is a robust way to read "label: value" tables because the order of rows can change without breaking it.

Step 5: store with an upsert

Now the complete pipeline in one runnable block, with the site, parsers, cleaning and the SQLite step. We run the scrape twice, changing a price in between, to prove the database updates instead of duplicating.

Python The complete pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three books, not six: the second run updated Dune's price instead of adding a duplicate.

Step 6: the report

A scraper is not finished until it tells you something. Query the database for the answers the project was meant to give:

Python Report from the stored data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is quality control. A rising count of missing values is the first sign that the website changed.

Taking it to a real site

To use this on a real website you are allowed to scrape:

  1. Replace the dictionary lookup with a fetch built on a requests.Session, with timeout, raise_for_status and an honest User-Agent (The requests library).
  2. Check robots.txt and the terms first (robots.txt and sitemaps, Legal and ethical scraping).
  3. Add a delay between pages and retries with backoff (Rate limiting and retries).
  4. Cache the HTML while you develop.
  5. Run the scrape on a schedule (cron, Task Scheduler) and watch the missing-value counts.

Good practice sites exist for this: sandbox sites built specifically for scraping practice (look for "scraping sandbox" or "toscrape") are intended to be scraped.

Ideas to extend it

  • Fetch each detail page and store upc, genre and description.
  • Add a history table so you can chart price changes.
  • Export the report to CSV or a small HTML page.
  • Port the loop to Scrapy (The Scrapy framework).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the project separate fetch, parse, clean and store?

    Separate stages are easy to test on saved or fake data and easy to swap out.
  2. 2. Why read the book title from the title attribute instead of the link text?

    Listing pages often shorten visible titles but keep the full title in an attribute.
  3. 3. What did running the scrape twice demonstrate?

    The URL primary key and ON CONFLICT clause made the run repeatable.
  4. 4. Why count rows with missing ratings in the report?

    Monitoring missing values is a cheap early warning for broken selectors.

Summary

  • A good scraper is a pipeline: fetch, crawl, parse, clean, store, report.
  • Real sites hide data in classes, attributes and tables; read them with the right selector.
  • Primary keys plus upserts make scrapers repeatable.
  • Reports and missing-value counts turn a scraper into something you can trust.
  • Before pointing it at a real site: permission, robots.txt, delays and an honest User-Agent.

Next: Project: Price Tracker