Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 39 ยท Projects and Practice

Practice Exercises

Eight scraping exercises with runnable solutions: tables, nested lists, JSON, pagination, robots rules, cleaning, deduplication and a mini crawler. Try each before reading the answer.

You learn scraping by doing it, and there is no substitute for facing a messy page and making it yield. These exercises use HTML and data held in strings, so they run anywhere and have exact right answers. Each one has a task, a hint, and a runnable solution.

Work in this order for each: read the task, try it in the editor on your own (replace the starter code), and only then compare with the solution. The starter blocks below contain the data and an empty function; the solution block is a complete version. If you get stuck, read the hint first and the matching lesson linked from it.

Exercise 1: a table to dictionaries

Task. Turn the table into a list of dictionaries with typed values (int for stock, float for price). Hint: use the header cells as keys (Extracting tables and lists in the BeautifulSoup chapter).

Python Exercise 1 solution: table to dicts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 2: nested categories

Task. The menu is a nested list. Produce the path of every leaf, such as Books > Fiction > Sci-Fi. Hint: recursion over li elements and their child ul.

Python Exercise 2 solution: tree paths

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 3: JSON pagination with a cursor

Task. The fake API returns pages with a next cursor. Collect all item names and stop when there is no cursor. Hint: Pagination patterns.

Python Exercise 3 solution: cursor loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 4: respect robots.txt

Task. Given a robots.txt and a list of candidate URLs, return only the ones your bot (MyBot) may fetch, and the delay to use between requests. Hint: robots.txt and sitemaps.

Python Exercise 4 solution: filter by robots.txt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note how the specific MyBot group wins over * for your bot, while everyone else is blocked entirely.

Exercise 5: clean a messy product list

Task. Clean the raw rows: trim and collapse whitespace in names, parse prices like 1.299,00 โ‚ฌ and $7.50, convert "12 in stock" to an integer, and drop rows with no price. Hint: Cleaning scraped data.

Python Exercise 5 solution: cleaning pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 6: deduplicate by ID, keep the cheapest

Task. The same product appears several times with different prices. Keep one row per id, the cheapest, and report how many duplicates were removed. Hint: Deduplicating and updating.

Python Exercise 6 solution: keep the cheapest per id

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 7: a mini crawler with a depth limit

Task. Crawl the fake site from /, staying on the same host, up to depth 2, and list every page with its depth and title. Ignore mailto: links. Hint: Building a crawler.

Python Exercise 7 solution: depth-limited crawl

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The page /c is at depth 3 and is correctly never fetched.

Exercise 8: find what changed

Task. Compare yesterday's and today's snapshots and print which products are new, which were removed, and which changed price with the old and new values. Hint: set operations on dictionary keys.

Python Exercise 8 solution: snapshot diff report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where to practice for real

  • Use sites that exist to be scraped: search for "scraping sandbox" or "toscrape" practice sites.
  • Public data portals and open APIs (government statistics, weather, public libraries) are ideal and legal-friendly.
  • Pick a site you care about, read its terms and robots.txt first (Legal and ethical scraping), and build something small and gentle.
  • Save HTML to disk, write the parser against the saved files, then add the fetch layer last.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. What is the best workflow when writing a parser for a real site?

    Working on saved copies is faster, repeatable and kinder to the site.
  2. 2. In exercise 4, why can MyBot fetch /about while OtherBot cannot?

    A bot follows the most specific matching group; unmatched bots fall back to the * group.
  3. 3. What does the depth check in the crawler exercise guarantee?

    Depth limits (with a seen set) bound the crawl even on large or looping sites.

Summary

  • Practice on data you can reproduce: saved HTML, fake sites and string fixtures.
  • The exercises covered tables, nested lists, pagination, robots rules, cleaning, deduplication, crawling and change detection.
  • Develop parsers on saved pages and add the network last.
  • Choose practice targets that welcome scrapers, and keep every real run polite.

Next: Back to the Web Scraping track