Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 39 ยท Crawling, Politeness and Ethics

Building a Crawler

Build a breadth-first web crawler in Python: a frontier queue, URL normalization, a visited set, same-domain rules, depth limits and resumable state. Runs offline.

Scraping one page gives you data. Crawling gives you a whole site: you start at one URL, extract its links, visit those, extract their links, and keep going. Search engines are giant crawlers. Yours will be smaller, usually aimed at one site or a section of it, but the engine inside is the same few ideas: a frontier of URLs still to visit, a visited set so you never fetch twice, a set of rules about what to follow, and limits so it always stops.

In this lesson we write a small crawler and run it on an in-memory fake website, so you can run and change it without touching any real server. The only line that changes for a real site is the function that fetches a page.

The crawl loop

flowchart TD
  S["Seed URL(s)"] --> F["Frontier queue"]
  F --> P["Pop next URL"]
  P --> C{"Seen before?<br/>Allowed? Within limits?"}
  C -- "no" --> F
  C -- "yes" --> G["Fetch and parse"]
  G --> D["Save the data"]
  G --> L["Extract links"]
  L --> N["Normalize and filter"]
  N --> F

A queue (first in, first out) gives breadth-first order: all pages one click from the start, then two clicks, and so on. A stack (last in, first out) gives depth-first order. Breadth-first is the usual choice because it finds important pages early and makes the depth limit meaningful.

Step 1: normalize URLs

The same page can be written many ways: /a, /a/, /a#top, /a?utm_source=x, HTTPS://Example.com/a. If you treat those as different pages, you fetch the same content again and again. Normalization maps them to one canonical form:

Python URL normalization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Whether /a/ and /a are really the same depends on the site, so check one before you apply that rule. Always remove fragments (#...), because the server never sees them. Strip well-known tracking parameters, and sort the remaining ones so that order does not matter.

  • Same host only (or a list of allowed hosts). Following every external link makes a crawl unbounded.
  • Schemes: only http and https (skip mailto:, tel:, javascript:).
  • Path rules: for example only /products/ and /category/, never /login or /cart.
  • File types: skip .jpg, .zip, .pdf unless you want them.
  • robots.txt: check can_fetch for each URL (robots.txt and sitemaps).
  • Limits: a maximum depth and a maximum number of pages.

Step 3: the crawler

Our fake site is a dictionary from URL to HTML. fetch looks the page up; on a real site it would call session.get(...).

Python A breadth-first crawler on a fake site

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice three details. The seen set is updated when a URL is queued, not when it is fetched, so the same link found on ten pages is queued only once. The cart page and the external site are filtered by allowed. And the trailing-slash and #reviews variants all collapse into one URL through normalize.

Try changing max_depth to 1 and see which pages remain.

Making it fit for real use

The toy version needs more before it is ready for a real site:

  1. A real fetch with a shared requests.Session, timeout and error handling (The requests library).
  2. Delay between requests, plus backoff when the server pushes back (Rate limiting and retries).
  3. Content-Type check: only parse responses that are text/html.
  4. Redirects: normalize the final response.url too, and add it to seen.
  5. Spider traps: calendars that link to the next month forever, or session IDs in URLs that create infinite variants. The depth limit, page limit and parameter stripping protect you.
  6. Persistence: store the frontier and seen so a crash does not make you start over.

Resumable state

Because the crawl state is just a queue and a set, saving it is a few lines:

Python Save and resume the crawl state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For bigger crawls keep the state in SQLite instead (Saving to CSV, JSON and SQLite): a table of URLs with a status column (queued, done, failed) is a persistent frontier that also tells you what failed.

Crawler or scraper?

The words overlap, but the split is useful. The crawler decides which pages to visit (frontier, rules, limits). The scraper extracts data from one page you give it. Keep them in separate functions: you can then test extraction on a saved file and the crawl logic on a fake site, exactly as we did here. Frameworks such as Scrapy (Scrapy framework) provide this structure ready-made.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a crawler keep a visited (seen) set?

    Pages link to each other in cycles; without a seen set the crawler would loop and waste requests.
  2. 2. What does breadth-first crawling use to hold the frontier?

    A FIFO queue visits pages level by level, so depth limits behave predictably.
  3. 3. Which of these should URL normalization remove?

    Fragments never reach the server, and tracking parameters create duplicate URLs for the same page.
  4. 4. Which limits protect a crawler from spider traps?

    Calendars and session-ID URLs can generate endless links; hard limits guarantee the crawl ends.

Summary

  • A crawler is a frontier queue, a seen set, link rules and limits.
  • Normalize URLs (absolute, lowercase host, no fragment, no tracking parameters) before comparing them.
  • Mark URLs as seen when you queue them; filter by host, path, scheme and robots.txt.
  • Always set depth and page limits, and save state so crashes are not fatal.
  • Keep crawling logic separate from extraction logic so each is testable.

Next: Rate Limiting and Retries