Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Resilience

Rate Limiting, Load Shedding and Backpressure

Protect a service from too much traffic: token bucket and leaky bucket limiters, 429 responses, load shedding and backpressure with bounded queues.

Resilience is not only about callers coping with a failing service. The service must also protect itself from callers. Every service has a capacity, and when requests arrive faster than it can process them, queues grow, latency climbs, memory fills up, and eventually the service answers nobody. Because a few well-behaved requests are better than none, a healthy system chooses what to refuse instead of letting overload choose for it.

This lesson covers four related tools. Rate limiting caps how much each client may send. Load shedding drops work when the whole service is overloaded. Backpressure pushes the pressure back to the producer so it slows down. Bounded queues keep waiting lines finite. They complement the caller-side tools from Timeouts, Retries and Exponential Backoff and Circuit Breakers, Bulkheads and Fallbacks.

Four ways to say "enough"

Rate limit A quota per client.
Load shedding Drop work when overloaded.
Backpressure Tell the producer to slow down.
Bounded queue A waiting line with a maximum.
1 / 2

Why unlimited acceptance is a bug

A server that accepts every request is not generous, it is fragile. Suppose it can finish 100 requests per second and receives 150. Fifty requests per second pile up in an unbounded queue. After ten seconds there are 500 waiting requests, each one now ten seconds old, and by the time they are served the callers have long since timed out. The server spends its capacity on answers nobody will read. This is why a limit with a quick rejection often increases the number of successful responses.

Token bucket

The token bucket is the most common rate-limiting algorithm. Picture a bucket that holds up to capacity tokens. Tokens are added at a fixed refill rate, for example 5 per second, until the bucket is full. Each request removes one token; if the bucket is empty, the request is rejected.

This allows bursts (a full bucket of 10 tokens allows ten requests at once) while enforcing a steady average (5 per second in the long run). Its sibling, the leaky bucket, works like a queue with a hole: requests enter a bucket and are released at a fixed rate, which smooths traffic into an even stream. Token bucket permits bursts; leaky bucket shapes them out. Two simpler options exist as well: a fixed window counter ("100 requests per minute") is easy but allows a double burst at a window boundary, and a sliding window fixes that at the cost of more bookkeeping.

The stand-in below implements both buckets with a virtual clock, so you can see the effect of time without waiting.

Node.js Token bucket versus leaky bucket

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

In the burst at time zero, both buckets accepted five requests and rejected three. After waiting, both started accepting again as capacity drained or refilled, at 2 per second. The difference in real systems is what happens to accepted requests: the token bucket lets them through immediately, the leaky bucket (used as a queue) releases them at the steady rate.

Typical limits are applied per client: per API key, per user id or per IP address, because a global limit lets one noisy client starve the rest. At scale, the counters live in a shared store such as Redis, or the limit is enforced in the API gateway. Often there are several tiers: a free plan, a paid plan and internal callers get different quotas.

429 responses and Retry-After

When a limit is exceeded, tell the client clearly. HTTP has a status code for it: 429 Too Many Requests. Add a Retry-After header saying how many seconds to wait, and many APIs also publish limit headers such as RateLimit-Remaining so well-behaved clients can slow down before they are rejected. Compare with 503 Service Unavailable, which says "the whole service is overloaded", not "you in particular sent too much".

A good client reads Retry-After, waits at least that long, and adds jitter (see the previous lesson). The stand-in below starts a real HTTP server on the loopback address, rate-limits it with a token bucket on a virtual clock, and has a client that respects Retry-After.

Node.js A rate-limited HTTP service that answers 429

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first three requests used the bucket's tokens. The fourth got a 429 with Retry-After: 1, waited the advised time and then succeeded, and so on. A client that ignored the header and hammered the endpoint would simply be rejected over and over, using server capacity just to say "no".

Load shedding

Rate limits are per client and decided in advance. Load shedding is a reaction to the server's own state: when it is overloaded (CPU very high, queue long, latency above a target), it drops some work early and cheaply so the rest can finish. Good shedding is selective:

  • Drop low-priority traffic first: analytics, prefetches, background sync. Keep checkout and login.
  • Prefer shedding new work over work that is nearly done.
  • Reject before doing expensive work such as parsing a large body or opening database connections.
  • Return 503 with Retry-After, or a cached or reduced response, rather than timing out.

Measure the overload signal at the edge of the service (queue length or in-flight request count) and compare it with a threshold. Shedding at the gateway protects every service behind it.

Backpressure and bounded queues

Where a producer feeds a consumer through a queue, the consumer may be slower. If the queue is unbounded, memory grows until the process dies, and every item waits longer than the one before. Backpressure means the consumer's limits are communicated back to the producer, so it slows down or stops. Mechanisms include a bounded queue (when full, the producer blocks or gets an error), HTTP 429 and 503, TCP's own flow control, and Reactive Streams demand signals where the consumer asks for N more items. In Node.js streams, write() returning false means "wait for the drain event".

When a bounded queue is full you have three honest choices: block the producer (pressure moves upstream), reject the new item (fast failure, caller decides), or drop the oldest or least important item. Which is right depends on the data: a payment must not be dropped silently, a sensor reading often can be.

Node.js Bounded queue with backpressure versus an unbounded queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Both consumers handled 80 jobs. The unbounded queue accepted all 200, ended with 120 still waiting, and its longest wait had already reached 12 ticks and was growing. The bounded queue said "no" to 116 jobs right away, so the 84 it accepted were served within 1 tick. In both cases the consumer's throughput is the same; the difference is whether the overload is visible and controlled or hidden and growing.

Capacity limits should come from measurements (load tests), not guesses. Set limits slightly below the point where latency starts to climb, and review them whenever you change the service.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a token bucket allow that a strict fixed-rate limiter does not?

    Saved-up tokens allow a burst. Over time the refill rate caps the average.
  2. 2. Which status code and header tell a client it sent too many requests and when to retry?

    429 Too Many Requests, with Retry-After giving the number of seconds to wait.
  3. 3. What is load shedding?

    Shedding keeps the service working for important requests instead of failing for everyone.
  4. 4. Why can an unbounded queue make an overloaded service worse?

    Items served after their caller gave up are wasted work, and memory eventually runs out.
  5. 5. What is backpressure?

    Backpressure pushes overload back toward the source instead of buffering it endlessly.

Summary

  • Services must protect themselves: unlimited acceptance turns overload into slow failure for everyone.
  • A token bucket allows bursts and enforces an average rate; a leaky bucket smooths traffic into a steady stream.
  • Reject with 429 and Retry-After for per-client limits, and 503 when the whole service is overloaded.
  • Load shedding drops low-priority work early; backpressure and bounded queues push overload back to producers.
  • Base limits on load-test measurements, and pair them with client-side backoff.

Next: Fault Tolerance and Chaos Engineering.