Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Resilience

Circuit Breakers, Bulkheads and Fallbacks

Stop one failing dependency from taking down your whole system: closed, open and half-open circuit breakers, bulkhead isolation and graceful fallbacks.

Retries help with brief hiccups. But when a dependency is truly down or hopelessly slow, retrying keeps sending it traffic and keeps your own threads waiting on calls that will fail. The damage then travels upward: callers pile up, their resources run out, and a failure in the recommendations service makes the whole shop page time out. This is a cascading failure.

Three patterns stop the cascade. A circuit breaker stops calling a dependency that is known to be failing. A bulkhead limits how much of your capacity any one dependency can use. A fallback decides what the user sees when the call cannot be made. Together they let a system lose a feature without losing the page.

Three guards against cascades

Circuit breaker Stop calling what is failing.
Bulkhead Cap what one dependency can use.
Fallback Return something useful instead.
1 / 2

The circuit breaker

The name comes from electrical circuit breakers: when there is too much current, the breaker trips and cuts the circuit, protecting the wiring. A software breaker wraps calls to a dependency and watches the results. It has three states.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures reach the threshold
    Open --> HalfOpen: cool-down time passed
    HalfOpen --> Closed: trial calls succeed
    HalfOpen --> Open: a trial call fails

The breaker closes when healthy, opens when failing, and probes carefully before closing again.

  • Closed. Normal operation: calls go through, and failures are counted. If failures reach a threshold (for example 5 in a row, or more than 50 percent of the last 20 calls), the breaker trips.
  • Open. Calls are rejected immediately, without touching the dependency ("fail fast"). This protects the dependency (it gets quiet time to recover) and the caller (no waiting). After a cool-down period, the breaker moves to half-open.
  • Half-open. A limited number of trial calls are let through. If they succeed, the breaker closes. If one fails, it opens again and the cool-down restarts.

Failing fast is the key benefit: an open breaker returns an error in microseconds instead of holding a thread for a ten-second timeout.

The stand-in below is a complete breaker class. It takes the clock as a parameter so the demo can move time forward instantly instead of waiting.

JS A circuit breaker class and its three states

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Calls 4 and 5 never reached the dependency: they were rejected instantly, so the failing service saw only three requests instead of five. After the cool-down, call 6 was the trial; it succeeded and the breaker closed.

Practical settings matter. Use a failure rate over a sliding window rather than "N in a row" for busy services, and a minimum number of calls before the rate is judged, so two failures out of two do not trip the breaker. Count timeouts as failures; ignore client errors (4xx) because they say nothing about the dependency's health. Keep one breaker per dependency (or per endpoint), never one global breaker. Libraries such as Resilience4j (Java), Polly (.NET), Opossum (Node.js) and open-source service meshes (Envoy, Istio's outlier detection) implement all of this, so you rarely write your own.

Bulkheads

On a ship, bulkheads divide the hull into watertight compartments, so a hole floods one compartment instead of sinking the ship. In software, a bulkhead gives each dependency its own limited share of resources: its own thread pool, connection pool or concurrency limit.

Without one, a single slow dependency can use up all of your worker threads. Imagine 20 workers shared by the recommendations call and the payment call. If recommendations hangs, every worker ends up waiting on it, and checkout stops working too even though payments are fine. With a bulkhead of, say, 5 concurrent calls to recommendations, only those 5 slots can be stuck. The other 15 workers stay free, and calls beyond the limit are rejected immediately (or queued briefly).

The stand-in uses a semaphore, a counter of free slots, as the concurrency limit. A real thread-pool bulkhead (Java) isolates threads; in event-loop languages like Node.js a concurrency limit plays the same role.

JS A bulkhead limits concurrent calls per dependency

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Only two shoppers got a slot in the recommendations compartment; the other three were rejected immediately instead of waiting in line behind a hung service, and payment was never affected. If both dependencies had shared one pool of two slots, the hung calls would have blocked the payment as well.

Fallbacks and graceful degradation

When a call fails, times out, is rejected by an open breaker or by a full bulkhead, the caller must still do something. A fallback is that something. Which one is right is a product decision, not just a technical one:

  • Return cached data, even slightly old (product details, exchange rates).
  • Return a default, such as "most popular items" instead of personalised recommendations.
  • Omit the feature: show the page without the recommendations box.
  • Queue the work and confirm later ("we will email your receipt").
  • Fail clearly when no honest fallback exists: you should not guess a payment result or an account balance.

The goal is graceful degradation: a partial but working experience rather than a blank error page. Mark essential and optional dependencies, so you know which failures may be hidden and which must be reported. Test your fallbacks, because code that runs only during incidents tends to be broken when it finally runs.

A fallback must not hide a failure from you. Log it, count it in your metrics and alert on it (see Observability). A system that quietly serves defaults for days is down, it just does not know it.

Putting the pieces together

The usual order around one remote call, from the outside in, is: fallback โ†’ bulkhead โ†’ circuit breaker โ†’ retry (with backoff) โ†’ timeout โ†’ the call. The timeout bounds each attempt, retries handle brief glitches (see Timeouts, Retries and Exponential Backoff), the breaker watches the combined outcome and trips if retries keep failing, the bulkhead limits concurrency, and the fallback shapes what the user sees. Many teams put these in a service mesh or client library once, rather than coding them by hand in every service. The callee has a matching tool: Rate Limiting, Load Shedding and Backpressure lets it protect itself when callers send too much.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does an open circuit breaker do with a new call?

    Failing fast protects the failing dependency from more load and saves the caller from waiting on a doomed call.
  2. 2. What is the purpose of the half-open state?

    After the cool-down the breaker probes carefully. Success closes it, a failure opens it again.
  3. 3. Why use a separate bulkhead per dependency?

    Isolated compartments keep a failure contained, so the rest of the service keeps working.
  4. 4. Which is a reasonable fallback for a failed personalised-recommendations call?

    Recommendations are optional, so degrading to a default or omitting them keeps the page usable.
  5. 5. Which responses should usually NOT count as failures for tripping a breaker?

    Client errors say the request was wrong, not that the dependency is unhealthy.

Summary

  • A circuit breaker has three states (closed, open, half-open) and fails fast while a dependency is unhealthy.
  • A bulkhead caps the resources one dependency can use, so its failure stays in its own compartment.
  • A fallback (cache, default, omission, queued work) keeps a degraded but working experience.
  • Combine timeout, retry, breaker, bulkhead and fallback, and make failures visible in logs and metrics.
  • Prefer a tested library or a service mesh over hand-written resilience code in production.

Next: Rate Limiting, Load Shedding and Backpressure.