Skip to content
Tutorials Center
My learning

Topics

Lesson 75 of 147 ยท Concurrency

Rate Limiting and Backpressure

Build rate limiters in Go from scratch - a token bucket with an injectable clock, per-client HTTP limits with 429 and Retry-After, bounded queues, load shedding and context-aware semaphores.

Every system has a limit. A database accepts only so many connections, a partner API allows a fixed number of calls per second, and your own server has a finite number of CPU cores and a finite amount of memory. When work arrives faster than it can be done, something must give. If you do nothing, the queue grows without bound, latency explodes, memory runs out and the process dies, taking its other users with it.

Two ideas keep a system healthy. Rate limiting controls how fast work is allowed in (or how fast you send it out). Backpressure pushes the slowness back to whoever is producing the work, so that a slow consumer slows the producer instead of drowning in a pile of pending jobs. The lesson on concurrency patterns showed a ticker limiter and a channel semaphore. Here you will build the real thing: a token bucket that is easy to test, a per-client limiter for HTTP with proper 429 responses, bounded queues with explicit overflow policies, and a semaphore that respects contexts.

Limit, queue or shed?

When a request arrives and there is no capacity, a program has only a handful of choices:

flowchart TD
    A["Work arrives"] --> B{"Capacity free?"}
    B -- yes --> C["Do it now"]
    B -- no --> D{"Policy"}
    D --> E["Wait<br/>(blocks the caller)"]
    D --> F["Queue<br/>(bounded buffer)"]
    D --> G["Reject<br/>(429 or 503 + Retry-After)"]
    D --> H["Drop<br/>(old or low priority)"]

The four things a system can do when it is full. Never choose "queue without a limit".

Waiting is natural inside a program (a blocked channel send is backpressure). Rejecting quickly is the right answer at a network edge, because a fast 429 Too Many Requests costs almost nothing, while a request that sits in a queue for 30 seconds ties up resources and is probably abandoned by the client anyway.

The token bucket

The most common rate-limiting algorithm is the token bucket. Picture a bucket that holds up to burst tokens. Tokens drip in at rate per second, and each operation takes one token. If the bucket has a token, the operation proceeds immediately. If it is empty, the operation must wait or be refused. The bucket gives you two numbers to tune: the long-run average (rate) and how big a spike you tolerate (burst).

You do not need a goroutine to add tokens. Instead, whenever someone asks, compute how much time has passed since the last visit and add that many tokens (capped at burst). That is cheap and exact.

To keep the limiter testable, do not call time.Now() inside it. Take the clock as a function. In tests (and in the example below) you pass a fake clock that you move forward by hand, so the output is the same on every run and you never need time.Sleep.

Go A token bucket driven by a fake clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first five calls give true true true false false: three tokens from the full bucket, then nothing. After half a second at 2 tokens per second exactly one token has dripped in. After an idle hour the bucket is full, but never more than full, so there are again exactly three successes. That cap is what makes burst a real limit and not just an initial value.

Reserve shows the second way to use a bucket: instead of refusing, it hands out a place in line. Each caller takes a token even if none is left (the balance goes negative) and is told how long to wait, so three callers get waits of 500 ms, 1 s and 1.5 s and the long-run rate is respected without anyone being rejected. A Wait method is just Reserve plus a sleep.

Waiting politely: Wait with a context

A blocking Wait(ctx) is what you want when you call someone else's API from a worker. It must stop early if the context is cancelled, and in that case give the token back so the cancelled caller does not use up capacity. A time.Timer plus a select does it:

Go Wait(ctx): pacing real calls, and giving up

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first loop prints six lines and shows that pacing happened, using a comparison with a safety margin instead of the exact time (timers may fire a little late, never early). The second part gives up after 30 ms, because the next token would not arrive for a whole second. In production, use the ready-made golang.org/x/time/rate package, which offers the same ideas (Allow, Reserve, Wait, and SetLimit to change the rate while running) with years of testing behind it. Writing your own once is the best way to understand it.

Limiting each client: HTTP middleware

A single bucket protects a resource globally. For a public API you also want one bucket per client, so one noisy caller cannot use up everyone's allowance. Keep a map from a key (the client IP address, an API key or a user ID) to a bucket, guarded by a mutex. When a request is refused, respond with 429 Too Many Requests and a Retry-After header that tells well-behaved clients how many seconds to wait.

Two details are easy to forget. First, the map grows with every new client, so a sweep must remove buckets that have been idle long enough to be full again (a fresh bucket is identical, so nothing is lost). Second, the key must be trustworthy: r.RemoteAddr is the real peer, but behind a proxy it is the proxy's address, and the X-Forwarded-For header can be forged by clients unless your own proxy sets it.

Go Per-client limits with 429 and Retry-After

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The third request from the first client is rejected with Retry-After: 1s, because it needs one more token and the rate is one per second. The second client is untouched. When the clock moves forward the first client gets in again, and a minute later the sweep removes both entries. In a real server you would call Sweep from a goroutine on a ticker every few minutes, and stop it when the server shuts down.

Rate limits protect against abuse, but they are not a security feature on their own. For login endpoints (see authentication) combine a per-IP limit with a per-account limit, because an attacker can rotate IP addresses. In clusters of servers, per-process limiters only limit each process; a shared limit needs a shared store such as Redis.

Bounded queues: backpressure inside a program

Channels are Go's natural backpressure tool. A buffered channel is a bounded queue: when it is full, a sender can do one of three things, and each is a policy decision:

  • Block until there is room. The producer slows to the consumer's speed. Simple and safe, as long as nobody holds a lock while blocking. Combine it with a context so a blocked sender can give up.
  • Reject with a non-blocking send (select with default) and tell the caller the system is busy.
  • Drop something: the new item, or the oldest one in the queue, for data where only the latest value matters.
Go Block, reject and give up on a full queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

With no consumer running, the first two jobs fit in the buffer and the other three are rejected. A sender that is willing to wait gives up after 20 ms. Once a consumer starts, the blocking send for job 7 succeeds, and the consumer sees jobs in first-in, first-out order. The buffer size is a real design choice: it absorbs short bursts, but a big buffer only hides the problem and adds delay. Start small, measure, and make the number configurable.

Limiting concurrency: the semaphore

Rate limiting says how often; a semaphore says how many at the same time. This protects connection pools, memory-hungry tasks and downstream services. A buffered channel makes a fine semaphore, and wrapping it in a small type lets you offer all three behaviors: Acquire(ctx) waits (but respects the context), TryAcquire returns immediately and Release frees the slot.

On an HTTP server, a semaphore turns into load shedding: when too many requests are already in flight, answer new ones with 503 Service Unavailable right away instead of letting them pile up and make every request slow.

Go A semaphore: Acquire with context, and load shedding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The two long requests occupy both slots, so the third gets 503 immediately. When they finish, capacity returns and the next request succeeds. Note the order: Release runs in a defer, so the slot is freed even if the handler panics (see defer). Always pair every successful acquire with exactly one release.

Choosing and tuning limits

  • Limit at the edge, shed early. Reject before you do expensive work such as parsing large bodies or opening database transactions.
  • Always return a hint. 429 with Retry-After, or 503 with Retry-After, lets good clients back off instead of hammering you.
  • Clients should back off too. Retry with exponential delay and jitter, not in a tight loop. The next lessons on resilience build that.
  • Measure. Count rejected requests, queue depth and time spent waiting. A queue that is always full means you need more capacity or a lower input rate.
  • Don't hold locks while waiting. A mutex held across a blocking send or Wait is a classic deadlock; see the sync package.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A token bucket has rate 5 per second and burst 10. After a long idle period, how many calls can succeed instantly?

    Tokens accumulate while idle but are capped at the burst size. So a spike of up to 10 passes immediately, then calls are limited to the 5 per second average.
  2. 2. Why does the bucket in this lesson take its clock as a function instead of calling time.Now() directly?

    Injecting the clock makes time-dependent logic deterministic to test. The real program passes time.Now, and tests pass a clock they advance by hand.
  3. 3. A client exceeds its limit. What should your HTTP middleware respond with?

    429 is the status for too many requests from this client. Retry-After tells polite clients when to try again. 503 is for overload of the server as a whole.
  4. 4. What happens with select { case q <- job: default: } when the buffered channel q is full?

    A select with a default case never blocks. When the send cannot proceed, the default branch runs, which is how you implement reject-when-full.
  5. 5. In Semaphore.Acquire(ctx), why is there a case <-ctx.Done() in the select?

    Without it, a goroutine blocked on a full semaphore could wait forever even though its request was cancelled. Acquiring with a context bounds the wait.

Summary

  • Rate limiting controls how fast work is admitted; backpressure makes slow consumers slow down producers; a semaphore controls how much runs at once. When full you can wait, queue (bounded), reject or drop, but never queue without limit.
  • A token bucket has a rate and a burst. Compute refills lazily from elapsed time and cap tokens at burst. Allow refuses, Reserve returns how long to wait, Wait(ctx) sleeps with a timer and gives the token back when cancelled.
  • Take the clock as a function so tests can use a fake clock.
  • Per-client HTTP limiting needs a map of buckets, a periodic sweep, a trustworthy key, and 429 with Retry-After.
  • Buffered channels are bounded queues: a plain send blocks, select with default rejects, select with ctx.Done() gives up.
  • Use a channel-based semaphore with Acquire(ctx), TryAcquire and Release to cap concurrency, and shed load with 503 when it is full. In production, golang.org/x/time/rate provides a tested token bucket.

Next: fmt in Depth: Formatting and Scanning