Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Reliability and Operations

Resilience Patterns

Learn timeouts, circuit breakers, bulkheads, load shedding, backpressure and hedged requests, and run simulations that show why each one helps.

In a distributed system, some part is always slow, overloaded or down. The question is not how to prevent that, but how to stop one sick component from taking everything else down with it. A slow database makes web servers wait, waiting servers pile up requests, and soon a problem in one corner has become an outage everywhere. This is a cascading failure.

Resilience patterns are small, well-known techniques that contain failure. They fail fast, limit damage, shed work you cannot do and give the broken part room to recover. This lesson covers the most useful ones and simulates three of them. Idempotency, Retries and Backoff is a good companion, because retries are the most common way to make an outage worse.

The resilience toolbox

Timeouts Never wait forever.
Circuit breaker Stop calling a broken service.
Bulkheads Separate pools so one leak cannot sink the ship.
Load shedding Refuse extra work early.
1 / 2

Timeouts: the first line of defense

Every network call needs a timeout. Without one, a thread or connection can wait forever for a reply that never comes, and since waiting resources are finite, a few hung calls can use them all up. A timeout turns "unknown" into "failed", which your code can handle.

Good timeouts follow a few rules:

  • Set them on everything: connect, read, and the whole request. Defaults are often infinite or very long.
  • Base them on measured latency. A value a bit above the 99th percentile of normal latency is a typical starting point. See Observability and Tail Latency for why percentiles matter.
  • Use a deadline across the whole call chain. If the user's request has 2 seconds in total, pass the remaining time down so deeper services do not keep working for a caller who has already given up.
  • Remember that a timeout is not a failure of the remote work. The operation may still have happened, which is why writes need idempotency.

Circuit breakers

A circuit breaker wraps calls to a dependency and watches the results. It has three states:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: too many failures
    Open --> HalfOpen: cooldown elapsed
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

Closed passes calls through, open rejects them instantly, half-open lets a single trial call test recovery.

While open, calls fail immediately without touching the sick service. That saves your callers from waiting for timeouts and gives the service breathing room. After a cooldown, one trial call checks whether it has recovered.

The simulation below uses a backend that is down from time 10 to 29 and makes one call per tick. It uses a counter instead of the real clock, so every run is the same.

Python A circuit breaker during an outage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without the breaker the backend receives 20 doomed calls during the outage. With it, only 6 reach the backend: the first three that open the breaker, plus three half-open trial calls that failed. The price is visible too: the breaker kept rejecting for two more ticks after the backend recovered at time 30, so two calls that would have worked (ticks 30 and 31) were skipped. Cooldown length balances protecting the service against reacting slowly to recovery.

In practice breakers usually count failures over a sliding window or a failure rate, not a plain streak, and they pair with a fallback: a cached value, a default, or a degraded feature, so a user sees a reduced page instead of an error.

Bulkheads

Ships have watertight compartments so one hull breach does not sink the whole vessel. A bulkhead isolates resources in the same way: give each dependency its own limited pool of threads, connections or concurrency slots. If the recommendation service hangs, it can use up only its own 10 slots, and checkout still has its slots free.

Bulkheads also apply at larger scales: separate clusters for different customers or tiers, or a "cell" architecture where each cell serves a slice of users and an incident hits one cell only.

Load shedding and backpressure

Every service has a capacity. When arrivals exceed it, work queues up, latency grows, and eventually every request is so late that the client has timed out. The server burns effort on answers nobody wants anymore.

Two ideas fix that:

  • Load shedding: when overloaded, reject some requests immediately (HTTP 503 or 429) so the ones you accept finish quickly. Cheap early rejection beats slow late failure. Prefer to shed low-priority or duplicate work first.
  • Backpressure: push the pressure back to producers. A bounded queue that blocks or rejects when full, a reader that stops reading from a socket, or a consumer that pulls at its own pace all slow the source instead of letting memory grow without limit.
Python Unbounded versus bounded queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both versions serve exactly 800 requests, since capacity is the same. The unbounded queue accepts everything, so the backlog keeps growing and the median accepted request waits 20 ticks. The bounded queue rejects 196 requests but those it accepts wait about one tick. When the offered load is permanently above capacity, someone is going to be disappointed. Shedding picks who, quickly, instead of making everyone wait.

Return a clear "try later" signal such as HTTP 429 or 503 with a Retry-After header. Well-behaved clients then back off, which is backpressure across the network.

Hedged requests

Even healthy systems have slow outliers. A hedged request sends the request to one replica, and if no answer arrives within a short delay (often about the 95th percentile of latency), sends a second copy to another replica and uses whichever answer comes first. It cuts tail latency at the price of a little extra load. Hedging is safe only for idempotent reads or operations.

In this simulation 5% of calls are slow (100 ms) and the rest take 8 to 14 ms. The hedge fires after 20 ms.

Python Hedging cuts the tail

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The median is unchanged and the 99th percentile drops from 100 ms to about 33 ms, at about 5% extra requests. The 99.9th percentile still shows 100 ms, because occasionally both copies are slow. Hedging helps the tail, not the extreme tail.

Combining the patterns

These patterns work as layers around a call: a deadline bounds the total time, a bulkhead limits concurrency, a circuit breaker stops calls to a failing dependency, retries with backoff handle transient errors (never inside an already-overloaded dependency without a budget), and a fallback supplies a degraded answer. Each layer needs metrics, so you can see when a breaker opens or requests are shed. Validate the whole stack by injecting faults, as shown in Testing Distributed Systems.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an open circuit breaker do?

    Failing fast saves callers from waiting and gives the sick service time to recover.
  2. 2. Why is a bounded queue often better than an unbounded one under sustained overload?

    Capacity is the same. Bounding the queue sheds excess early, so accepted requests stay fast instead of waiting behind a growing backlog.
  3. 3. What problem does a bulkhead solve?

    Separate resource pools per dependency keep a failing one from starving everything else.
  4. 4. When is hedging a request appropriate?

    The duplicate may execute too, so the operation must be safe to repeat; the extra load is small when the hedge delay is near a high percentile.

Summary

  • Cascading failures start when waiting callers exhaust resources; resilience patterns contain them.
  • Always set timeouts, and prefer an overall deadline passed down the call chain.
  • A circuit breaker (closed, open, half-open) fails fast while a dependency is down and probes for recovery.
  • Bulkheads isolate resource pools; load shedding and backpressure keep overloaded services responsive.
  • Hedged requests reduce tail latency for idempotent calls at the cost of a few percent extra load.

Next: Observability and Tail Latency.