Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 57 ยท Architectural and Modern Patterns

Circuit Breaker, Retry and Bulkhead Patterns

Learn resilience patterns for calling unreliable services: retry with backoff, circuit breaker states, fallbacks and bulkheads, with runnable stand-ins you can step through.

Every time your program calls something it does not control, such as a payment API, another service or a database, it can fail. Failures come in two kinds. Transient failures fix themselves in a moment (a dropped packet, a restarting server). Persistent failures last until someone repairs something. A good design treats the two differently: it retries transient ones and stops hammering a service that is truly down.

Without protection, one sick dependency can take down the healthy parts of your system. Threads pile up waiting on a slow service, memory fills, callers time out and retry, and the load makes the sick service sicker. This is a cascading failure. The patterns in this lesson, Retry, Circuit Breaker, Fallback and Bulkhead, are small wrappers that stop it. They are usually combined, and libraries such as Resilience4j (Java), Polly (.NET) and Failsafe or opossum (Node.js) implement them. Here you will build tiny versions to understand how they work.

Retry with backoff

The simplest wrapper: if a call fails, try again. Two details make it safe.

  • Backoff: wait longer between attempts (for example 100 ms, 200 ms, 400 ms). Immediate retries just add load to a struggling service.
  • Jitter: add a random offset to the wait so thousands of clients do not all retry at the same instant.

Also set a maximum number of attempts, and retry only idempotent operations or errors you know are safe to repeat. Retrying "charge the card" after a timeout can charge twice unless the call carries an idempotency key.

Node.js Retry with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Retry alone is not enough. If the service stays down, every caller keeps waiting through its whole retry schedule. That is where the circuit breaker comes in.

Circuit Breaker

The name comes from the electrical fuse. A circuit breaker wraps a call and watches the failures. When too many happen, it "opens" and rejects calls immediately without even trying, protecting both the caller (no waiting) and the struggling service (no extra load). After a cool-down it lets a few calls through to test recovery.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures reach the threshold
    Open --> HalfOpen: cool-down time passes
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

Closed lets calls through, Open rejects them, Half-open allows a trial call to decide which way to go.

  • Closed: normal operation. Calls pass through and failures are counted.
  • Open: calls fail instantly with a "circuit open" error. No traffic reaches the dependency.
  • Half-open: after the cool-down, one (or a few) trial calls are allowed. Success closes the circuit; failure reopens it and restarts the timer.

The example below uses an injected clock instead of real waiting, so you can step through the states deterministically. Passing in the clock is also how you would test a real breaker.

Node.js A circuit breaker you can step through

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Calls 4 and 5 never reached the service: they failed instantly, which is the whole point. Real breakers add refinements: a failure rate over a sliding window instead of a plain count, a minimum number of calls before judging, a limit of concurrent half-open trials, and metrics so you can see when a breaker trips.

Fallback

When the call fails or the circuit is open, a fallback supplies a degraded but useful answer instead of an error: cached data from the last good response, a default value, a simplified result, or a message such as "recommendations unavailable". Decide this per feature. Showing slightly old product reviews is fine; showing a stale account balance is not.

Node.js Breaker plus fallback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The page still renders in both cases, only the quality changes.

Bulkhead

Ships have watertight compartments so that a leak floods one section, not the whole hull. A bulkhead does the same with resources: give each dependency its own limited pool of threads, connections or concurrent slots. If the slow reporting service uses up its ten slots, calls to the payment service still have theirs.

Node.js A bulkhead limiting concurrent calls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three reports were rejected quickly instead of piling up, and the payment still went through. In a real service a rejected call would use a fallback or return "try again later" with a 503.

Timeouts, the missing partner

None of these patterns works without a timeout on every remote call. A call that never returns holds a thread and a connection forever, and a breaker cannot count a failure that never happens. Set an explicit timeout slightly above the normal worst case, and make sure the total of retries stays within what the user is willing to wait.

Putting them together

A common order, from outside in, is: fallback wraps circuit breaker wraps retry wraps bulkhead wraps the timed call. Retries happen inside the breaker so that a burst of retries counts as failures; the fallback catches whatever still fails. Tune each number with real measurements, and watch the metrics: a breaker that never opens or always stays open is telling you something.

Related ideas: the Proxy pattern is the structural form most of these wrappers take (same interface, extra control), and the Decorator lets you stack them, as in the combined order above.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should retries use backoff and jitter?

    Waiting longer between attempts and randomizing the wait spreads the load, so retries do not form a stampede.
  2. 2. In which state does a circuit breaker reject calls without trying the dependency?

    Open means the failure threshold was reached; calls fail instantly until the cool-down ends.
  3. 3. A half-open breaker's trial call fails. What happens?

    A failed trial shows the dependency is still unhealthy, so the circuit returns to open.
  4. 4. What problem does a bulkhead solve?

    Each dependency gets its own limit on concurrent calls, so a failure stays contained, like a flooded ship compartment.
  5. 5. Which operation is the riskiest to retry blindly after a timeout?

    The first charge may have succeeded even though the response was lost, so a blind retry can charge twice. Reads are safe to repeat.

Summary

  • Remote calls fail; transient failures deserve retries, persistent ones deserve fast rejection.
  • Retry uses a maximum count, exponential backoff and jitter, and only for safe, idempotent operations.
  • A circuit breaker moves between closed, open and half-open to stop calling a failing dependency and test recovery after a cool-down.
  • A fallback returns a degraded answer such as cached data when the call fails.
  • A bulkhead gives each dependency its own limited resources so failures stay contained.
  • Always set timeouts, and combine the patterns as fallback, breaker, retry, bulkhead, timed call.

Next lesson: Anti-Patterns.