Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 40 ยท Async Libraries and Patterns

Rate Limiting and Concurrency Limits in asyncio

Limit how much async work runs at once: Semaphore, bounded worker pools with queues, and a token-bucket rate limiter, with deterministic runnable examples.

Creating a task is so cheap that it is easy to start 100,000 of them. The problem is what they do next: open 100,000 sockets, hit an API that allows 10 requests per second, or load 100,000 files into memory. Concurrency without limits crashes programs and gets you blocked by the services you call.

There are two different things to limit, and people often confuse them:

  • Concurrency: how many operations are in progress at the same time ("no more than 10 open connections").
  • Rate: how many operations start per unit of time ("no more than 5 requests per second").

This lesson gives you a tool for each: the semaphore and the worker pool for concurrency, and a token bucket for rate.

Semaphore: at most N at a time

An asyncio.Semaphore(n) holds a counter that starts at n. async with semaphore: takes one slot, waiting if none is free, and gives it back on exit. Wrap the part of your code that must be limited, usually the network call.

Python At most three jobs at once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All ten tasks exist immediately, but only three are inside the async with at any moment; the rest wait their turn in first-in, first-out order. Because the async with releases the slot even when the body raises or is cancelled, slots do not leak.

Related classes:

  • BoundedSemaphore(n) raises ValueError if you release more times than you acquired, which catches bugs.
  • Lock is a semaphore of one: only one task at a time inside the section.
  • Event, Condition and Barrier coordinate tasks rather than limit them; see the synchronization chapter of this course.

Worker pool: a fixed number of tasks and a queue

The semaphore approach creates one task per item up front. For very large inputs (millions of URLs) that is wasteful: all those task objects sit in memory. A worker pool starts a fixed number of worker tasks that pull items from a queue, so you hold only the queue's contents.

Python A pool of three workers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

queue.join() returns once task_done() has been called for every item that was put in. The workers loop forever, so we cancel them at the end. In production, feed the queue from a producer that respects a bounded maxsize for backpressure, and prefer a TaskGroup to own the workers.

Choosing between the two:

Pattern Best when
gather + Semaphore a few hundred to a few thousand items; simplest code
Queue + worker pool huge or endless streams; workers need set-up cost (a connection each); you want fixed memory

Rate limiting: a token bucket

A semaphore limits simultaneous calls, but ten calls that each take 10 ms still allow 1,000 requests per second. To cap the rate, the classic algorithm is the token bucket: a bucket holds up to capacity tokens and refills at rate tokens per second. Each request removes one token; with an empty bucket you wait. A full bucket allows a short burst, but the long-term average cannot exceed rate.

flowchart LR
    R["Refill: rate tokens per second"] --> B["Bucket (max capacity)"]
    B -->|"take 1 token"| C["Request runs"]
    D["Request arrives"] --> B
    B -->|"empty: wait"| D

Tokens refill steadily; a request needs a token to start, so bursts are bounded and the average rate is controlled.

To make the demonstration repeatable, the limiter below takes its clock as a function and its sleeper as a function. Here we feed it a fake clock that advances only when the limiter "sleeps", so no real time passes and the output is always the same. In a real program you would use time.monotonic and asyncio.sleep.

Python A token-bucket rate limiter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first three requests start at once (the burst), and then one starts every half second, which is exactly the configured rate of two per second. With the real clock, replace FakeTime with time.monotonic and asyncio.sleep.

Combining both limits

A realistic client limits concurrency and rate, and keeps both limits at the single choke point where requests are made:

Python
class LimitedClient:
    def __init__(self, session, max_concurrent=10, per_second=20):
        self.session = session
        self.gate = asyncio.Semaphore(max_concurrent)
        self.bucket = TokenBucket(per_second, per_second, time.monotonic, asyncio.sleep)

    async def get(self, url):
        await self.bucket.acquire()          # obey the rate first
        async with self.gate:                # then the concurrency ceiling
            return await self.session.get(url)

Callers just use await client.get(url) from as many tasks as they like, and the policy is enforced in one place. This works with the fake API and real clients alike.

Practical advice

  • Put the limit inside the function that does the I/O, not in every caller, so no one can forget it.
  • Read the provider's documented limits and stay safely below them. Honour 429 responses and Retry-After, then slow down with backoff.
  • A semaphore queue is fair (first come, first served); a burst of new tasks cannot starve old ones.
  • With gather over a giant list, memory still holds every coroutine. Switch to a queue and workers when the list is huge.
  • Each limiter is per process. If you run several processes or servers, a shared limit needs shared state (Redis, for instance).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does asyncio.Semaphore(5) limit?

    A semaphore limits concurrency, the number of holders at once, and says nothing about how fast they start.
  2. 2. Ten tasks wait on a Semaphore(3). What happens when a holder leaves the "async with" block?

    Leaving the block releases one slot, and exactly one waiter takes it.
  3. 3. Which tool controls a requests-per-second limit?

    A semaphore bounds simultaneous calls. A token bucket spaces out the start times so the average rate stays below a limit while allowing a small burst.
  4. 4. When is a queue plus worker pool better than gather with a semaphore?

    Workers pull items on demand, so memory stays flat, and each worker can hold its own connection or resource.

Summary

  • Concurrency (how many at once) and rate (how many per second) are different limits.
  • Semaphore caps simultaneous operations; a queue with N workers does the same for huge inputs.
  • A token bucket caps the rate and allows small bursts; inject the clock to make it testable.
  • Enforce limits inside the I/O function so every caller is covered.

Next: Debug mode and finding slow callbacks