Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 40 ยท Projects and Practice

Project: A Concurrent Web Crawler with asyncio

Build a concurrent crawler with asyncio: a queue and worker pool, a visited set, depth limits, retries, timeouts and a summary report, tested against a fake in-memory web.

A web crawler is the classic asyncio project: it spends nearly all of its time waiting for the network, it discovers new work as it goes, and it must not hammer any server. In this project you will build one step by step and use almost everything from this chapter group: a queue and worker pool, a visited set, concurrency limits, timeouts, retries and clean shutdown.

The web is fake here. The browser editor cannot open network connections, so the crawler runs against FakeWeb, an in-memory website with the same shape as a real HTTP client: await web.get(url) takes a little while and returns a page with a status and a list of links, or fails. The crawler code is real asyncio and works unchanged against a real client such as aiohttp or httpx; the end of the lesson shows the only function you would replace.

The plan

flowchart LR
    S["Start URL"] --> Q["Queue of (url, depth)"]
    Q --> W["N workers"]
    W -->|"fetch with timeout + retry"| F["Page"]
    F -->|"new links not seen before"| Q
    F --> R["Results"]

Workers take URLs from the queue, fetch them, record the result and put any new links back on the queue.

Design decisions:

  1. Workers, not one task per URL. A fixed pool of workers with a shared queue keeps memory and open connections bounded (worker pools).
  2. A seen set, updated before queueing a link, so no page is fetched twice. Check-and-add happens with no await in between, so it is race-free.
  3. A depth limit so the crawl ends.
  4. queue.join() to know when everything is done: it returns when every queued URL has been marked task_done().
  5. Failures are data. A page that errors is recorded and the crawl continues.

Step 1: the fake web

The site has a home page linking to sections; one page is slow, one is flaky (fails the first time), one is missing and one is a loop back to the home page.

Python The in-memory website

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: fetch one page safely

fetch() adds a timeout and a few retries around a single request, and returns a result that always has a status, so the caller never has to handle exceptions. Real transient errors are retried; a 404 is a valid answer and is not.

Python
async def fetch(web, url, *, retries=2, timeout=1.0):
    for attempt in range(retries + 1):
        try:
            async with asyncio.timeout(timeout):
                page = await web.get(url)
            return {"url": url, "status": page["status"], "links": page["links"], "attempts": attempt + 1}
        except (ConnectionError, TimeoutError) as exc:
            if attempt == retries:
                return {"url": url, "status": "error", "links": [], "attempts": attempt + 1, "error": repr(exc)}
            await asyncio.sleep(0)       # a real crawler would back off here

(See retries and backoff for the full version with jitter.)

Step 3: the crawler

Now the heart of the project: the queue, the workers, the seen set and the shutdown. Read the comments, then run it.

Python The complete crawler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see every page within two links of the home page exactly once (including the 404 for /old and the page that succeeded on its second attempt; the two /docs/api/v1 and v2 pages are three links deep, so the depth limit skips them), the loop back to / ignored thanks to the seen set, and never more than three requests at a time.

Why it works

  • queue.join() completes only when the count of unfinished items reaches zero. Workers add new links before calling task_done() for the current URL, so the count cannot hit zero while there is still work to discover. If you called task_done() first, the crawl could end early.
  • try/finally around the body guarantees task_done() even if something unexpected raises, otherwise join() would hang forever.
  • Workers loop forever and are cancelled at the end (cancel, then gather).

Step 4: swap in a real HTTP client

Only web.get needs replacing. With aiohttp, the adapter is small; the links are extracted from the HTML (a real crawler would use an HTML parser such as html.parser or BeautifulSoup in a thread, since parsing is CPU work).

Python
class RealWeb:
    def __init__(self, session):
        self.session = session

    async def get(self, url):
        async with self.session.get(url) as response:
            if response.status != 200:
                return {"status": response.status, "links": []}
            html = await response.text()
            return {"status": 200, "links": await asyncio.to_thread(extract_links, html, url)}

(Read-only: it needs network access and the aiohttp package. Also add a per-host rate limit as in rate limiting, obey robots.txt, and set an honest User-Agent.)

Extensions to try

  1. Depth reporting: store depth in each result and print the pages grouped by depth.
  2. Per-host limits: keep a defaultdict(lambda: asyncio.Semaphore(2)) keyed by hostname so no site gets more than two connections.
  3. Politeness delay: add a token bucket per host.
  4. Bounded queue: give the queue a maxsize and see what deadlock risk a full queue creates when workers also put into it (hint: put with put_nowait and handle QueueFull, or keep a separate pending list).
  5. Time budget: wrap crawl() in asyncio.timeout(30) and return partial results on timeout.
  6. Streaming results: turn crawl into an async generator that yields pages as they finish, using async iteration patterns.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do workers add new links to the queue before calling task_done() for the current URL?

    queue.join() returns when the unfinished count is zero. Queueing discoveries first keeps the count above zero until all work is accounted for.
  2. 2. What is the purpose of the "seen" set?

    Pages often link to each other in loops. Marking a URL as seen before queueing it prevents duplicate work and endless crawling.
  3. 3. How does the crawler limit concurrency?

    Only as many fetches can be in progress as there are workers.
  4. 4. A worker's body raises an unexpected exception and there is no try/finally. What happens to queue.join()?

    Each get() must be matched by task_done(). A missing call leaves the unfinished count above zero.

Summary

  • A crawler is a queue, a pool of workers, a seen set and a way to know when to stop (queue.join()).
  • Wrap each fetch in a timeout and bounded retries, and return failures as data.
  • Keep the real HTTP client behind one small get() function so the logic is testable with a fake.
  • The same structure scales to downloaders, link checkers, scrapers and any job that discovers more jobs.

Next: Project: a chat room server