Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 65 ยท Reliability and Operations

Circuit Breakers and Bulkheads

Stop cascading failures with circuit breakers, bulkheads, timeouts, fallbacks and load shedding. Learn the states and thresholds, with runnable Python simulations.

A single slow dependency can bring down an entire system. Service A calls service B. B becomes slow. Threads in A pile up waiting for B. A runs out of threads, so A's other endpoints, which have nothing to do with B, stop responding. Callers of A start piling up and the failure spreads backward through the architecture. This is a cascading failure, and it is the typical way large systems collapse.

Fault tolerance patterns exist to contain failures. This lesson covers the two most important: the circuit breaker, which stops calling something that is failing, and the bulkhead, which limits how much damage any one dependency can do. Together with timeouts and fallbacks they make a resilient system: one that degrades instead of dying.

The circuit breaker

An electrical breaker trips when current is too high, so the wiring does not burn. A software circuit breaker wraps calls to a dependency and trips when failures are frequent. While tripped, calls fail immediately without even trying, which protects the caller (no threads blocked) and gives the struggling dependency time to recover instead of drowning in retries.

It has three states:

Three states

Closed Calls pass through. Failures are counted.
Open Calls fail fast. No load on the dependency.
Half-open A few trial calls test whether it recovered.

Too many failures open it; a successful trial closes it.

1 / 2
stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failures exceed threshold
  Open --> HalfOpen: wait time elapsed
  HalfOpen --> Closed: trial call succeeds
  HalfOpen --> Open: trial call fails

The breaker opens on repeated failure, then probes carefully before closing.

Python A circuit breaker from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

While the breaker was open, 14 attempts produced far fewer real calls. The caller got instant errors rather than slow timeouts, and the backend got breathing room.

Tuning a breaker

  • Failure signal: count errors and timeouts; many implementations use an error rate over a sliding window (such as 50% of at least 20 calls in 10 seconds) rather than raw counts, so low traffic does not trip it by accident.
  • Reset timeout: long enough for the dependency to recover, short enough to resume quickly. Many breakers use exponential backoff on repeated reopening.
  • Half-open probes: allow only one or a few trial calls so recovery is not flooded.
  • Per dependency (and often per endpoint): one breaker for "payments" does not need to affect "recommendations".
  • Observability: expose the state as a metric and alert when breakers open; it is an early outage signal.

Fallbacks: what to do when it is open

A tripped breaker needs an answer for users. Options, from best to worst:

  1. Cached or stale data: show the last known product price.
  2. Default value: an empty recommendation list, a generic avatar.
  3. Degraded feature: disable "Customers also bought" but keep checkout.
  4. Queue for later: accept the request and process it when the dependency returns.
  5. A clear error: honest, fast failure beats a spinner that never ends.

Never fall back in a way that is wrong (a fake successful payment). A fallback must match the business meaning of the operation.

Bulkheads

Ships are divided into watertight compartments so one hull breach does not sink the ship. A bulkhead isolates resources per dependency or per client so that one failing part cannot consume everything.

Practical forms:

  • Separate thread or connection pools per downstream service: if the payment pool is exhausted, the search pool still works.
  • Concurrency limits (semaphores): at most N in-flight calls to a dependency; extra calls fail immediately.
  • Separate deployments or queues for critical and non-critical traffic: the nightly report job cannot starve checkout.
  • Per-tenant quotas so one noisy customer does not harm others (see rate limiting).
Python Bulkhead: one slow dependency, two designs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one shared pool, the hung dependency takes every thread and healthy traffic gets nothing. With bulkheads, healthy features keep half the capacity.

Load shedding and timeouts

Load shedding means an overloaded service deliberately rejects excess work (fast 503s) to protect its core. Prioritize: drop low-value requests (analytics pings) before high-value ones (payments). Combined with timeouts (see Retries, Timeouts and Dead Letter Queues), rate limiting and backpressure (telling producers to slow down), these form a layered defense:

Pattern Protects against
Timeout Waiting forever on a hung call
Retry with backoff Transient errors
Circuit breaker Hammering a dependency that is down
Bulkhead One failure exhausting shared resources
Fallback Users seeing errors
Load shedding Total overload

Libraries and platforms implement these for you: Resilience4j and Polly in code, Envoy and Istio in a service mesh. Read about similar ideas in Resilience Patterns.

Do not apply breakers blindly everywhere. Calls to a single local database that your service cannot work without are better protected by timeouts, pooling and good capacity than by tripping and returning errors for everything.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A circuit breaker is open. What happens to a new call?

    An open breaker fails fast, protecting the caller from blocked threads and the dependency from extra load.
  2. 2. What is the purpose of the half-open state?

    Half-open probes carefully: a successful trial closes the breaker, a failure reopens it.
  3. 3. Which is a bulkhead?

    Bulkheads isolate resources so that one failing dependency cannot exhaust everything the service needs.
  4. 4. Which fallback is inappropriate for a payment call?

    A fallback must not misrepresent the outcome. Faking success would ship goods without payment.
  5. 5. What typically starts a cascading failure?

    Waiting callers hold resources, so slowness spreads upstream until unrelated features fail.

Summary

  • Cascading failures spread from one slow or failing dependency to everything that calls it.
  • A circuit breaker (closed, open, half-open) fails fast when a dependency is unhealthy and probes for recovery.
  • Fallbacks keep users served with cached, default or degraded results, never with wrong ones.
  • Bulkheads isolate pools, limits and deployments so one failure cannot consume all resources.
  • Combine timeouts, retries, breakers, bulkheads and load shedding as layers.

Next: Monitoring, Logging and Tracing