Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 39 ยท Requests and APIs

Asynchronous Scraping with asyncio

Speed up scraping with asyncio: concurrent fetching, semaphores to limit load, timeouts, error handling and threads, with offline runnable examples.

A scraper spends most of its life waiting. After sending a request it does nothing until the server answers, often for half a second or more. Fetching 200 pages one after another at 0.5 seconds each takes over a minute and a half, and your CPU is idle nearly the whole time.

Concurrency fixes that by starting the next request while the previous one is still waiting. In Python the cleanest way is asyncio. This lesson explains the model, shows the real network code (read-only), and lets you run the exact same structure against a simulated server so you can see the speed-up and the safety limits yourself.

Why concurrency, and why a limit

Speed is the easy part. The hard part is restraint. Starting 200 requests at once is an attack on the server, will get your IP blocked, and usually makes things slower. The fix is a concurrency limit: allow, say, five requests in flight at a time. Combine that with a delay between requests and you stay polite. Read Rate limiting and retries before you point an async scraper at a real site.

flowchart LR
  Q["200 URLs"] --> L["Semaphore: max 5 at once"]
  L --> W1["worker"]
  L --> W2["worker"]
  L --> W3["worker"]
  W1 --> R["results"]
  W2 --> R
  W3 --> R

async and await in thirty seconds

An async def function is a coroutine. Inside it, await marks the places where it may pause and let others run (typically waiting for the network). asyncio.gather runs many coroutines at the same time on one thread. Nothing runs in parallel on several CPU cores; they just take turns whenever one is waiting.

Python Sequential vs concurrent (simulated network)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ten fetches of 0.2 s take at least 2 s one by one but only a little over 0.2 s together. Exact timings vary from run to run and machine to machine.

Limiting concurrency with a semaphore

asyncio.Semaphore(n) lets at most n coroutines into a protected block at the same time. Wrap the fetch in it:

Python At most three requests in flight

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The peak never exceeds three, no matter how many URLs you queue.

Errors, retries and timeouts

With gather, one exception cancels your view of everything else unless you ask for exceptions to be returned. Use return_exceptions=True (or handle errors inside each task) and add timeouts so one slow server cannot hold a slot forever.

Python Errors and timeouts in concurrent tasks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Collecting failures as data rather than letting them crash the run lets you retry just the failed URLs afterwards.

The real thing with httpx

httpx has an async client with the same feel as requests. This is read-only code for your own machine (pip install httpx):

Python
import asyncio, httpx
from bs4 import BeautifulSoup

SEM = asyncio.Semaphore(5)

async def fetch(client, url):
    async with SEM:
        resp = await client.get(url)
        resp.raise_for_status()
        await asyncio.sleep(0.2)             # courtesy delay per slot
        return resp.text

async def main(urls):
    headers = {"User-Agent": "DemoBot/1.0 (+https://example.org/bot)"}
    async with httpx.AsyncClient(headers=headers, timeout=10,
                                 follow_redirects=True) as client:
        pages = await asyncio.gather(*(fetch(client, u) for u in urls),
                                     return_exceptions=True)
    for url, page in zip(urls, pages):
        if isinstance(page, Exception):
            print("failed", url, page)
            continue
        title = BeautifulSoup(page, "lxml").title
        print(url, title.get_text(strip=True) if title else None)

asyncio.run(main([f"https://example.com/?page={i}" for i in range(1, 6)]))

aiohttp is the other popular choice; its ClientSession plays the role of httpx.AsyncClient. Both give you one shared client for all requests, which keeps connections alive and cookies consistent, just like a requests.Session.

Parsing HTML is CPU work, not waiting. Doing it inside the event loop is fine for small pages, but for heavy parsing hand the work to asyncio.to_thread(...) or a process pool so downloads keep flowing.

Threads as a simpler alternative

If you already have working requests code, a thread pool gives a similar speed-up with almost no rewrite:

Python
from concurrent.futures import ThreadPoolExecutor

def get(url):
    return session.get(url, timeout=10).text

with ThreadPoolExecutor(max_workers=5) as pool:
    pages = list(pool.map(get, urls))

max_workers is your concurrency limit. Threads are perfectly good for a few dozen simultaneous requests; asyncio scales better to thousands of idle connections.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does asyncio.Semaphore(5) do in a scraper?

    A semaphore caps simultaneous access, which is how you limit requests in flight.
  2. 2. Why does async scraping speed things up?

    Network waits dominate; concurrency overlaps them.
  3. 3. What does return_exceptions=True do in asyncio.gather?

    Failed tasks appear in the results list as exception objects, so one failure does not lose the others.
  4. 4. Which is the right attitude to concurrency limits when scraping someone else's site?

    A polite scraper keeps concurrency low and respects the site's capacity and rate limits.

Summary

  • Scrapers wait on the network; asyncio (or threads) overlaps those waits.
  • asyncio.gather runs many coroutines; a Semaphore limits how many run at once.
  • Always add timeouts and capture errors per task so failures can be retried.
  • httpx.AsyncClient and aiohttp.ClientSession are the async counterparts of a requests.Session.
  • Speed is not the goal on its own: keep concurrency modest and delays polite.

Next: JavaScript-Rendered Pages