Skip to content
Tutorials Center
My learning

Topics

Lesson 107 of 119 ยท Spring Cloud

Circuit Breakers and Resilience with Resilience4j

Build resilient Java services with Resilience4j - sliding-window circuit breakers, retries with jitter, bulkheads, rate limiters, time limiters and fallbacks in Spring.

In a microservice system, every remote call is a bet. The other service may be slow, overloaded, restarting, or gone. Without protection, one sick dependency ties up your threads, then your callers' threads, and within minutes a problem in one corner takes down the whole system. Resilience patterns are the guard rails that keep a local failure local.

The Microservices lesson introduced the basic toolkit: timeouts, retries with backoff, a simple circuit breaker that counts consecutive failures, bulkheads and fallbacks. This lesson goes deeper, to the level you need to configure Resilience4j in production with confidence: sliding windows and slow-call detection, the half-open state, retry strategies and their traps, the two kinds of bulkheads, rate limiters, and the order in which these decorators wrap each other.

Resilience4j and Spring Cloud Circuit Breaker are not available in our code runner, so each runnable example builds the mechanism in plain Java (often against a real HTTP server on 127.0.0.1 inside the program), and the real annotations and YAML are shown read-only.

How remote calls fail

Before choosing a pattern, name the failure you are defending against:

Failure What you see Main defense
Service down, port closed Connection refused, quickly Circuit breaker, fallback
Network partition, host gone Connect timeout Connect timeout, circuit breaker
Service overloaded Slow responses, then 503 Timeout, bulkhead, breaker counting slow calls
Brief blip (restart, dropped connection) One failure, then success Retry with backoff
Too many requests 429 Too Many Requests Rate limiter on your side, respect Retry-After
Bug or bad input 400, 404, 500 with the same input every time Nothing to retry; fix or report

Slow is worse than down. A dead service fails in a millisecond; a slow one holds a thread and a connection for the full timeout. That is why every pattern below starts with a timeout, and why good circuit breakers count slow calls, not only errors.

The circuit breaker in depth

A circuit breaker has three main states. CLOSED lets calls through and records their outcomes. OPEN rejects calls immediately (Resilience4j throws CallNotPermittedException) for a wait duration. HALF_OPEN lets a limited number of trial calls through and decides, from their results, whether to close or open again.

Resilience4j decides from a sliding window of recent calls rather than consecutive failures:

  • sliding-window-type: COUNT_BASED (the last N calls) or TIME_BASED (calls in the last N seconds).
  • sliding-window-size: N (default 100).
  • minimum-number-of-calls: no decision before this many calls are recorded (default 100), so two failures out of two calls after a deploy don't trip the breaker.
  • failure-rate-threshold: open when this percentage of calls failed (default 50).
  • slow-call-duration-threshold and slow-call-rate-threshold: a call slower than the duration counts as slow, and the breaker also opens when this percentage of calls were slow (defaults 60 s and 100%, so you should lower both).
  • wait-duration-in-open-state (default 60 s) and permitted-number-of-calls-in-half-open-state (default 10).

The program below implements these rules and runs a scripted day of a catalog service on a real local HTTP server: healthy, then failing, then slow, then recovered. Time spent in the open state uses a manual clock so the output is the same on every run.

Java A sliding-window circuit breaker with slow-call detection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Read the output from top to bottom. No decision is made until four calls are in the window. The breaker opens when half of the recent calls failed, and the following call is rejected without touching the catalog. After the wait it goes half-open, two good trial calls close it, and the window starts fresh. Then the catalog becomes slow: every call succeeds, yet once the new window holds four calls (three of them slow, 75%), the slow-call rate trips the breaker after call 13. A breaker that only counted errors would have kept sending traffic to a service that is drowning.

minimum-number-of-calls is easy to forget. With Resilience4j's default of 100, a service that receives ten calls a minute needs ten minutes of data before its breaker can ever open. Set it to something your real traffic reaches within one wait duration.

Resilience4j also has the states DISABLED (always allow, record nothing), FORCED_OPEN (always reject) and METRICS_ONLY (record and emit metrics, never reject). You can switch to them at runtime, which helps during incidents and when rolling out a breaker carefully.

Retries without causing outages

Retries turn transient failures into successes, and turn overload into collapse when done carelessly. The rules from Microservices still hold: retry only idempotent operations (or those protected by an idempotency key), only transient errors, with backoff, jitter and a cap. What changes in practice is the schedule. This program prints the waits produced by four common strategies, with a seeded random generator so the numbers repeat:

Java Comparing backoff strategies with and without jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Fixed waits are simple but keep hammering a struggling service at the same rate. Exponential backoff backs off quickly, but every client that failed at the same moment retries at the same moments, in synchronized waves. Full jitter (a random wait between zero and the exponential value) spreads those waves out, and decorrelated jitter (each wait random between the base and three times the previous wait) does the same with a smoother curve. Both random strategies are widely used; the important part is that some randomness exists and that the wait is capped.

Three more traps that bite in production:

  • Retries multiply across layers. Retry in one place, usually the client closest to the failing dependency.
  • Retrying a rejected call is pointless. When the circuit breaker is open, an immediate retry gets the same CallNotPermittedException. In Resilience4j, add it to the retry's ignore-exceptions, because by default a retry instance retries every exception.
  • Budget your retries. Some teams cap retries at a percentage of traffic (for example 10%), so during a real outage the extra load stays bounded. Envoy and gRPC implement such retry budgets; with Resilience4j you combine retries with a breaker and a rate limiter.

Bulkheads: isolating dependencies

Your service has a limited number of threads and connections. If 200 requests are all stuck waiting for a hanging recommendation service, there's nothing left for the checkout endpoint. A bulkhead caps how much of your capacity one dependency may use. Resilience4j offers two kinds:

  • Semaphore bulkhead (Bulkhead): at most max-concurrent-calls callers may be inside at once (default 25). Others wait up to max-wait-duration (default 0, reject immediately) and then get BulkheadFullException. The call runs on the caller's thread, which suits virtual threads well.
  • Thread-pool bulkhead (ThreadPoolBulkhead): calls run on a dedicated, bounded pool with a bounded queue, and the method returns a CompletionStage. It isolates more strongly but costs threads and context switches.
Java Semaphore and thread-pool bulkheads under load

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Rejected calls fail in microseconds instead of piling up. That sounds harsh, but it's what keeps the rest of the service alive, and a fallback can turn the rejection into a degraded but useful answer. Size bulkheads from measurements: roughly the dependency's normal throughput times its normal latency, plus headroom. Spring Framework 7's @ConcurrencyLimit is a built-in semaphore bulkhead for simple cases.

Rate limiters and time limiters

A rate limiter caps how often you call something: a partner API that allows 50 requests per second, or an expensive report endpoint. Resilience4j's default RateLimiter divides time into cycles of limit-refresh-period and hands out limit-for-period permissions per cycle. A caller that finds no permission left may wait up to timeout-duration for the next cycle, otherwise it gets RequestNotPermitted.

Java A cycle-based rate limiter with a wait timeout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The first three calls in the first second pass; the next two are rejected because the next cycle is more than 150 ms away; the call at 900 ms waits 100 ms and uses a permission from the next cycle, which is why that cycle has room for only two more calls. Limiting your own outgoing calls like this is different from an API gateway limiting incoming traffic per client, which the API Gateway lesson covers.

A time limiter puts a deadline on an asynchronous call (CompletableFuture), with timeout-duration (default 1 s) and cancel-running-future (default true). For synchronous HTTP calls, set the client's own connect and read timeouts instead; they free the connection properly.

Putting the decorators together

In real code you combine patterns. The order matters, and Resilience4j's Spring integration applies them like this, outermost first:

TEXT
Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( your method ) ) ) ) )

The retry is outside the circuit breaker, so every retry attempt is recorded by the breaker, and once it opens the retry sees CallNotPermittedException. The fallback runs last, after all attempts. The final program wires a retry, a breaker and a time limit around calls to a scripted local service, and prints what happens to each request.

Java Retry around a circuit breaker with a time limit and a fallback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Request A succeeds at once. Request B's first attempt hits the slow call, the 300 ms time limit cuts it off, and the retry succeeds. Request C meets a failing catalog: its first attempt fails, the breaker now has four calls with two failures in its window and opens, so the second attempt is rejected and C gets the fallback. Request D never reaches the catalog at all. Only four calls reached the server for four user requests, and no user waited longer than about half a second.

Resilience4j in Spring Boot

Add the Resilience4j Spring Boot starter that matches your Boot version, plus AOP support for the annotations (spring-boot-starter-aspectj in Boot 4, spring-boot-starter-aop before). Then annotate the methods that make remote calls:

Java
@Service
class CatalogClient {

    private final RestClient rest;
    private final Map<Long, String> titleCache = new ConcurrentHashMap<>();   // filled on successful calls

    CatalogClient(RestClient.Builder builder) {
        this.rest = builder.baseUrl("http://catalog-service").build();
    }

    @Retry(name = "catalog")
    @CircuitBreaker(name = "catalog", fallbackMethod = "cachedTitle")
    @Bulkhead(name = "catalog")
    public String title(long id) {
        return rest.get().uri("/books/{id}", id).retrieve().body(BookDto.class).title();
    }

    // Same parameters plus the exception, same return type.
    private String cachedTitle(long id, CallNotPermittedException e) {
        return titleCache.getOrDefault(id, "(title unavailable)");
    }

    private String cachedTitle(long id, Exception e) {
        return "(title unavailable)";
    }
}

Fallback methods are matched by exception type, most specific first, so you can answer "breaker open" differently from "unexpected error". Because the annotations work through Spring proxies, calling title() from inside the same class bypasses them, the same self-invocation trap as @Transactional (Spring AOP).

YAML
resilience4j:
  circuitbreaker:
    configs:
      default:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        slow-call-duration-threshold: 2s
        slow-call-rate-threshold: 80
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 3
        register-health-indicator: true
    instances:
      catalog:
        base-config: default
        ignore-exceptions:
          - com.example.catalog.BookNotFoundException   # a 404 is not a sick service
  retry:
    instances:
      catalog:
        max-attempts: 3
        wait-duration: 200ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        enable-randomized-wait: true
        retry-exceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException
        ignore-exceptions:
          - io.github.resilience4j.circuitbreaker.CallNotPermittedException
  bulkhead:
    instances:
      catalog:
        max-concurrent-calls: 20
        max-wait-duration: 0

Note ignore-exceptions on the breaker: business errors such as "book not found" must not count as failures, or a burst of bad user input would open the circuit. With Actuator, /actuator/circuitbreakers and /actuator/circuitbreakerevents show the live state and recent transitions, and Micrometer publishes metrics such as resilience4j.circuitbreaker.state and resilience4j.circuitbreaker.calls, which are worth an alert (Actuator, Metrics and Observability).

Spring Cloud Circuit Breaker

Spring Cloud Circuit Breaker is a thin abstraction over the same library. Instead of annotations, you wrap a call programmatically, which also works where proxies can't, such as lambdas inside one method:

Java
@Service
class RecommendationClient {

    private final CircuitBreakerFactory<?, ?> breakers;
    private final RestClient rest;

    RecommendationClient(CircuitBreakerFactory<?, ?> breakers, RestClient.Builder builder) {
        this.breakers = breakers;
        this.rest = builder.baseUrl("http://recommendation-service").build();
    }

    List<String> forMember(long memberId) {
        return breakers.create("recommendations").run(
            () -> rest.get().uri("/members/{id}/picks", memberId).retrieve().body(PICKS),
            throwable -> List.of());                         // fallback: no recommendations
    }
}

The starter is spring-cloud-starter-circuitbreaker-resilience4j, and defaults are set with a Customizer<Resilience4JCircuitBreakerFactory> bean or the same YAML. Spring Cloud Gateway and OpenFeign plug into this abstraction too.

Built-in alternatives in Spring Framework 7

For simple cases you may not need a library. Spring Framework 7 includes @Retryable (with delay, multiplier, jitter and exception filters) and @ConcurrencyLimit in its core, activated with @EnableResilientMethods:

Java
@Retryable(includes = IOException.class, maxRetries = 3, delay = 100, multiplier = 2, jitter = 10)
@ConcurrencyLimit(10)
public Invoice fetchInvoice(String id) throws IOException { ... }

There is no circuit breaker or rate limiter in the core, so Resilience4j remains the standard choice when you need those. Check the reference documentation of your Spring version for the exact attribute names.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A breaker uses a count-based window of 10, minimum-number-of-calls 5 and failure-rate-threshold 50. The first three calls after startup all fail. What is its state?

    The breaker evaluates the failure rate only once the minimum number of calls is in the window. This prevents tripping on tiny samples.
  2. 2. A dependency answers every request successfully but takes 8 seconds. Which setting lets the breaker protect you?

    Slow successes are not failures. Only the slow-call rate makes a breaker open for a service that is overloaded but still answering.
  3. 3. With @Retry(max-attempts 3) outside @CircuitBreaker and no ignore-exceptions, the breaker is OPEN. What happens to one call?

    By default a retry instance retries every exception, including CallNotPermittedException. Add it to the retry's ignore-exceptions so an open breaker fails fast.
  4. 4. A semaphore bulkhead has max-concurrent-calls 3 and max-wait-duration 0. Eight requests arrive while three calls hang. How many get BulkheadFullException?

    Three permits are taken by the hanging calls; with no waiting allowed, the other five are rejected immediately.
  5. 5. The catalog returns 404 for unknown books, and a scraper requests thousands of unknown ids. What should you configure so the breaker stays closed?

    The breaker should measure the health of the dependency, not the quality of the input. Ignored exceptions are neither failures nor successes.

Summary

  • Name the failure first: down, slow, overloaded, transient, rate-limited, or a bug. Slow dependencies are the most dangerous, so every remote call needs a timeout.
  • Resilience4j circuit breakers decide from a count- or time-based sliding window, wait for minimum-number-of-calls, and open on a high failure rate or a high slow-call rate. Half-open allows a few trial calls; ignore-exceptions keeps business errors out of the statistics.
  • Retry only idempotent, transient failures, with capped exponential backoff and jitter, in one layer only, and never retry CallNotPermittedException.
  • Semaphore bulkheads cap concurrent calls on the caller's thread; thread-pool bulkheads use a bounded pool and queue. Rejections are fast by design.
  • Rate limiters cap your own outgoing call rate per cycle; time limiters add deadlines to async calls.
  • Spring's default decorator order is Retry, CircuitBreaker, RateLimiter, TimeLimiter, Bulkhead. Fallbacks match by exception type; Spring Cloud Circuit Breaker offers a programmatic API; Spring Framework 7 adds @Retryable and @ConcurrencyLimit.

Next: Distributed Tracing with Micrometer and Zipkin