Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 39 ยท Crawling, Politeness and Ethics

Rate Limiting and Retries

Be a good citizen on the web: delays with jitter, token buckets, 429 and Retry-After handling, exponential backoff and caching so you never re-download a page.

Every request you send costs the site's owner something: CPU, bandwidth, and sometimes real money. A human visitor loads a page every few seconds. A careless scraper can load a hundred pages a second, which looks like an attack and can slow down or crash small sites. Being polite is not only the right thing to do, it is also practical: scrapers that behave are blocked less, and they get cleaner data because they are not constantly hitting error pages.

This lesson builds the three habits of a polite scraper: slow down (rate limiting), back off when the server pushes back (retries), and do not ask twice (caching). All the examples use a fake clock and a fake server, so they run instantly and give the same output every time.

Slow down: delays and jitter

The simplest rule is a pause between requests. How long? Use, in this order: the Crawl-delay in robots.txt if there is one, a limit stated in the site's API docs or terms, or a conservative default such as one request per second per host, slower for small sites.

Add jitter, a small random variation, so your requests do not arrive on a perfectly regular beat (also easier on the server when many scrapers start at the same moment):

Python
import random, time

def polite_sleep(base=1.0, jitter=0.5):
    time.sleep(base + random.uniform(0, jitter))

A rate limiter you can test

Calling time.sleep makes tests slow. Instead, write the limiter so that it asks a clock for the time and a sleep function to wait, then pass in fakes for testing. The next limiter allows at most one request every min_interval seconds per host:

Python A per-host rate limiter with a fake clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The limiter tracks each host separately, so while you wait for a.test you can use that time to fetch from b.test. The real version uses time.monotonic and time.sleep in place of the fake clock.

Token bucket: allow bursts, cap the average

Another common design is a token bucket. The bucket refills at a steady rate (say 1 token per second) up to a maximum. Each request spends a token. You can burst a few requests at once, but the average rate can never exceed the refill rate.

Python Token bucket

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first three requests at once are allowed (the burst), the fourth must wait, and after a pause the bucket has refilled enough for one more.

Back off when the server pushes back

Servers talk back in status codes. The ones you will meet:

Status Meaning What to do
429 Too Many Requests You are going too fast Wait (obey Retry-After), then slow down for good
503 Service Unavailable Overloaded or maintenance Wait and retry later; may include Retry-After
500, 502, 504 Server trouble or gateway timeout Retry a few times with backoff
403 Forbidden You are not allowed, or you were blocked Do not hammer; investigate and likely stop
404, 410 Not there Do not retry

Retry-After is either a number of seconds or an HTTP date. Exponential backoff doubles the wait after each failed attempt (1 s, 2 s, 4 s, 8 s), usually with jitter, and gives up after a few tries. Always respect the header if it is present:

Python Exponential backoff that honors Retry-After

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real code does the same with requests. A complete, read-only fetch function that combines timeouts, limits, retry and backoff:

Python
import time, requests

RETRY_STATUS = {429, 500, 502, 503, 504}

def polite_get(session, url, limiter, max_attempts=5):
    for attempt in range(1, max_attempts + 1):
        limiter.wait(requests.utils.urlparse(url).netloc)
        try:
            resp = session.get(url, timeout=(3.05, 15))
        except (requests.ConnectionError, requests.Timeout):
            resp = None
        if resp is not None and resp.status_code not in RETRY_STATUS:
            return resp                                   # includes 404: caller decides
        retry_after = resp.headers.get("Retry-After") if resp is not None else None
        time.sleep(backoff_delay(attempt, retry_after))
    raise RuntimeError(f"gave up on {url}")

If you keep getting 429 even after backing off, your base rate is too high. Cut it in half. If you get 403 or a CAPTCHA, stop and think; do not rotate IPs or fake browsers to push past a block. That is the site saying no.

Do not ask twice: caching

The politest request is the one you do not send. While you develop a scraper you will fetch the same page dozens of times. Cache responses on disk keyed by URL and reuse them:

Python A disk-style cache keyed by URL hash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In production, add freshness: store a timestamp and re-fetch only when older than a day or when a conditional request says the page changed. Conditional requests send If-None-Match: <etag> or If-Modified-Since: <date>; if nothing changed, the server replies 304 Not Modified with no body, which is cheap for both sides. Libraries like requests-cache do all of this for you.

Other good habits

  • Identify yourself: a descriptive User-Agent with a contact URL or email.
  • Scrape during off-peak hours for the site's time zone when you can.
  • Fetch only what you need: no images unless you want them, no pages outside your target section.
  • Prefer an official API or data export when one exists.
  • Stop if the owner asks you to.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the Retry-After header tell you?

    The server states a wait in seconds or as a date, and a polite client obeys it.
  2. 2. What is the purpose of jitter in a delay?

    Random variation keeps many clients from hitting the server in lockstep.
  3. 3. How does a token bucket differ from a fixed delay?

    The bucket's capacity permits a burst, while the refill rate bounds the long-run average.
  4. 4. Which status should you NOT retry?

    A 404 means the page does not exist; retrying will not change that.
  5. 5. Why cache responses during development?

    The politest request is the one never sent, and caching makes development faster too.

Summary

  • Add a per-host delay (with jitter) and honor Crawl-delay; token buckets allow bursts within an average cap.
  • Retry only transient errors (429, 5xx, timeouts) with exponential backoff and obey Retry-After.
  • Never retry 404; treat 403 and CAPTCHAs as stop signs, not obstacles.
  • Cache responses and use conditional requests so you do not download unchanged pages.
  • Write rate and retry logic against an injectable clock so it is testable.

Next: The Scrapy Framework