Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 66 ยท Production Features

Resilience: Timeouts, Retries and Circuit Breakers

Keep a Spring Boot app alive when dependencies fail: timeouts, safe retries with backoff, circuit breakers, bulkheads, fallbacks, Resilience4j and @Retryable.

Every call to another system will eventually fail: the network drops a packet, a database is restarted, a partner API gets slow. In a service that depends on five others, something is always degraded. Resilience means the failure of one dependency does not become the failure of your whole application. The real danger is rarely an error. It is a slow dependency: threads pile up waiting, pools drain, and a problem in one corner takes down everything.

This lesson covers the toolbox in the order you should reach for it: timeouts, retries with backoff, circuit breakers, bulkheads and fallbacks. The real Spring and Resilience4j code is read-only, because those libraries are not available on this site; two runnable stand-ins (a retry with backoff and a circuit breaker) let you watch the logic work. We start from Calling Other Services as the place where most of these protections are applied.

A slow dependency is worse than a dead one

Slow call Each request waits for it.
Threads pile up The pool fills with waiting work.
Everything stalls Even endpoints that never use the dependency.

Fail fast and contain the damage.

1 / 2

Timeouts first

Without a timeout, a call can wait indefinitely, and one hung connection holds a thread forever. A timeout is the foundation: retries and circuit breakers all depend on a call that ends in bounded time. There are two to set:

  • Connect timeout: how long to wait to establish the connection (seconds at most; a reachable host answers in milliseconds).
  • Read (response) timeout: how long to wait for data once connected.

Boot 3.4 and later configure them for every HTTP client it builds with properties:

PROPERTIES
spring.http.client.connect-timeout=2s
spring.http.client.read-timeout=5s

Per client, you can set them in code on the builder. Database pools and message clients have their own settings (Hikari's connection-timeout, JDBC socketTimeout/queryTimeout, broker request timeouts). Choose values from observation: a little above the dependency's normal p99 latency, and below the timeout of whoever calls you, so you fail first and can respond gracefully instead of being cut off. A timeout of 30 seconds on an endpoint that users abandon after three is a leak, not a safety net.

Retries: useful, and dangerous

Many failures are transient: a connection reset, a brief overload, a failover in progress. Trying again a moment later often works. But retries multiply load on a service that may already be struggling, so they need rules:

  1. Retry only what is safe to repeat. A GET or a PUT is idempotent; a POST that creates an order or charges a card is not, unless it carries an idempotency key the server uses to ignore repeats (see Designing a Clean API).
  2. Retry only errors that can heal: timeouts, connection errors and 503. Never retry 400, 401, 403 or 404; the answer will be the same.
  3. Back off. Wait between attempts, and increase the wait (exponential backoff: 100 ms, 200 ms, 400 ms ...).
  4. Add jitter, a random fraction of the delay. Without it, a thousand clients that failed together retry together, in waves that knock the recovering service over again.
  5. Cap attempts, usually two or three. And watch for retry amplification: if service A retries 3 times and calls B, which retries 3 times calling C, one request can become nine calls to C.

Spring's built-in retry

Spring Framework 7 (the basis of Boot 4) includes retry and concurrency limiting in the core. You enable it once and annotate methods:

Java
import org.springframework.resilience.annotation.EnableResilientMethods;
import org.springframework.resilience.annotation.Retryable;
import org.springframework.resilience.annotation.ConcurrencyLimit;

@Configuration
@EnableResilientMethods
class ResilienceConfig {}

@Service
class RateClient {

    @Retryable(includes = ResourceAccessException.class,
               maxRetries = 3, delay = 200, multiplier = 2, jitter = 50)
    BigDecimal fetchRate(String currency) {
        return restClient.get().uri("/rates/{c}", currency).retrieve().body(BigDecimal.class);
    }

    @ConcurrencyLimit(10)       // at most 10 callers inside at once
    Report buildReport() { ... }
}

Delays are in milliseconds here, multiplier = 2 doubles the delay each time, and jitter adds randomness. Like @Async and @Transactional, these annotations work through a proxy, so a call from inside the same class is not protected. Programmatic use is available too, through a RetryTemplate, when you need to retry a block of code rather than a method. On Boot 3 and Spring Framework 6, the same job is done by the separate Spring Retry library (@Retryable(maxAttempts = 3, backoff = @Backoff(delay = 200, multiplier = 2))), which is in maintenance mode; check which one your version supports.

Backoff with jitter, shown

The next TryIt calls a flaky service that fails twice and then succeeds. It prints the delays the retry would wait, computed with exponential backoff and a seeded random jitter, and uses a virtual clock so it never sleeps and the output is stable. A second call shows what a non-retryable error does, and what happens when attempts run out and a fallback value is used. What is real: the retry rules and the delay arithmetic. What is the stand-in: no real waiting, no proxy, no annotation.

Java Retry with exponential backoff and jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice that the delays 200, 400, 800 ms (plus a little jitter) grow, that the permanent error is not retried, and that when retries are exhausted the caller still gets an answer from the fallback instead of an exception.

Circuit breakers

Retries help when a failure is brief. When a dependency is down, retrying every request just adds load and makes every user wait through the delays. A circuit breaker watches the recent outcomes of calls and, when too many fail, opens: further calls fail immediately without touching the dependency. After a pause it lets a few trial calls through to see whether the service has recovered. The name comes from the electrical device that cuts the current to protect the house.

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> OPEN: failure rate above threshold
    OPEN --> HALF_OPEN: wait time elapsed
    HALF_OPEN --> CLOSED: trial calls succeed
    HALF_OPEN --> OPEN: a trial call fails

Closed passes calls through, open rejects them at once, half-open probes with a few calls.

The benefits are two: the failing dependency gets breathing room to recover, and your callers get a fast answer (an error or a fallback) instead of waiting for a timeout every time.

Resilience4j

The most used library with Spring Boot is Resilience4j. Add its Boot starter for your Boot version (and Spring AOP, which the annotations need), configure instances by name, and annotate:

YAML
resilience4j:
  circuitbreaker:
    instances:
      payments:
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 3
  retry:
    instances:
      payments:
        max-attempts: 3
        wait-duration: 200ms
        enable-exponential-backoff: true
Java
@Service
class PaymentClient {

    @CircuitBreaker(name = "payments", fallbackMethod = "pendingPayment")
    @Retry(name = "payments")
    PaymentResult charge(PaymentRequest request) {
        return restClient.post().uri("/charge").body(request).retrieve().body(PaymentResult.class);
    }

    // same parameters, plus the exception, same return type
    PaymentResult pendingPayment(PaymentRequest request, Exception cause) {
        return PaymentResult.pending(request.orderId());   // accept the order, charge later
    }
}

The configuration above means: judge the last 20 calls, but only once at least 10 have happened; open when half or more failed; stay open for 10 seconds; then allow 3 trial calls. Resilience4j also provides @Bulkhead (limits concurrent calls), @RateLimiter and @TimeLimiter (for async calls). The order in which the aspects wrap each other matters (retry typically sits outside the circuit breaker so that an open circuit short-circuits retries), and it is configurable.

When the breaker is open the call throws CallNotPermittedException. Your fallback and your error handling (Error Handling and Problem Details) should map it to something sensible, such as a 503 with a Retry-After hint.

What each setting means

Property Meaning Typical value
sliding-window-type judge the last N calls (COUNT_BASED, the default) or the last N seconds (TIME_BASED) count-based
sliding-window-size how many calls (or seconds) the window holds 20 to 100
minimum-number-of-calls calls needed before the failure rate is evaluated at all; stops one early failure from opening a cold circuit about half the window
failure-rate-threshold percentage of failed calls that opens the circuit 50
slow-call-duration-threshold, slow-call-rate-threshold count calls slower than the duration as "slow", and open when too many are a bit above p99 latency
wait-duration-in-open-state how long the circuit stays open before moving to half-open 10 to 60 s
permitted-number-of-calls-in-half-open-state trial calls allowed in half-open 3 to 10
automatic-transition-from-open-to-half-open-enabled move to half-open by timer instead of waiting for the next call true when you want fast recovery
record-exceptions, ignore-exceptions which exceptions count as failures; a validation error (HTTP 400) must not open the circuit server errors and timeouts only
register-health-indicator publish the breaker state through the Actuator health endpoint true

The breaker also publishes events and metrics: with Actuator and Micrometer you get resilience4j.circuitbreaker.state and call counters, and GET /actuator/circuitbreakers lists each breaker's state. With register-health-indicator: true, an open circuit shows in /actuator/health (set allow-health-indicator-to-fail: false if you do not want a downstream outage to mark your own service as down; see Actuator).

The same breaker can be used without annotations, which is useful for code that is not a bean or needs dynamic names:

Java
@Service
class RatesClient {

    private final CircuitBreaker breaker;

    RatesClient(CircuitBreakerRegistry registry) {
        this.breaker = registry.circuitBreaker("rates");              // configured under resilience4j.circuitbreaker.instances.rates
    }

    BigDecimal rate(String pair) {
        try {
            return breaker.executeSupplier(() -> fetchRate(pair));
        } catch (CallNotPermittedException open) {
            return cachedRate(pair);                                  // fallback when the circuit is open
        }
    }
}

Netflix Hystrix was the first widely used circuit breaker library in the Spring Cloud world (@HystrixCommand and @EnableCircuitBreaker). It has been in maintenance mode since 2018 and Spring Cloud removed its support, so new code should not use it. Resilience4j replaced it: it is lighter (built on functional decorators instead of thread pools per command), works with plain functions, offers retry, rate limiter, bulkhead and time limiter modules, and integrates with Micrometer. Spring Cloud's spring-cloud-starter-circuitbreaker-resilience4j wraps it behind a common CircuitBreakerFactory interface so you can switch implementations.

A circuit breaker you can step through

The TryIt below is a small breaker driven by a virtual clock: one call per second against a service that is broken from second 3 to second 11. It uses a count-based window of the last 4 calls, opens at 50% failures, waits 5 seconds, then lets 2 probe calls decide. What is real: the state machine and the thresholds idea. What is the stand-in: no concurrency, a tiny window and a fake clock.

Java A circuit breaker state machine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Follow the story in the output: failures are counted until the window is half failures, the circuit opens and rejections cost nothing, after five seconds a probe hits the still-broken service and reopens the circuit, and only when the service is back do two successful probes close it again.

Bulkheads and fallbacks

A bulkhead is named after the compartments of a ship: a leak floods one section, not the whole hull. In software it means giving each dependency its own limited share of threads or concurrent permits, so a slow partner can use at most that share. Options: @ConcurrencyLimit or Resilience4j's @Bulkhead (a semaphore that rejects extra callers immediately), a separate small thread pool per dependency, and separate connection pools. With virtual threads the thread cost of waiting disappears, which makes explicit concurrency limits even more important: nothing else stops ten thousand waiting calls from reaching a fragile service.

A fallback decides what the caller gets when the protected call fails or is rejected. Good fallbacks degrade gracefully:

  • Serve cached or stale data (yesterday's exchange rate, the last known stock level).
  • Return a sensible default or an empty list when the feature is optional (recommendations).
  • Accept now, finish later: queue the work and tell the user it is pending.
  • Fail clearly with 503 when no honest substitute exists. A payment must never "succeed" in a fallback unless it really did.

Make fallbacks cheap and independent of the failing system, and count their use with a metric (Observability): a fallback that quietly runs all day hides an outage.

Putting it together

For one outgoing call, the layers from outside in are usually: a bulkhead (limit concurrency) around a circuit breaker around a retry around the call, which itself has a timeout. Add the layers that your risk justifies: a timeout always, retries for idempotent calls on flaky links, a breaker for dependencies that can be down for minutes, a bulkhead for the ones that can be slow. Then test the failures: use a tool such as WireMock or a Testcontainers proxy that injects latency and errors, and verify that your timeouts, breakers and fallbacks behave as configured. Resilience code that has never seen a failure is a guess.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is a missing timeout dangerous?

    Without a bound on waiting, a slow dependency consumes threads until nothing is left to serve other requests.
  2. 2. Which request should NOT be retried automatically unless it carries an idempotency key?

    Repeating a non-idempotent POST can create two orders or charge twice. An idempotency key lets the server recognize and ignore the repeat.
  3. 3. What is the purpose of jitter in retry backoff?

    Clients that failed together would otherwise retry together in waves. Random jitter spreads the load.
  4. 4. A circuit breaker is OPEN. What happens to a new call?

    Failing fast protects both the dependency and the caller. After the wait time the breaker goes half-open and lets a few probe calls through.
  5. 5. Service A retries each call to B three times, and B retries each call to C three times. How many calls can C receive for one request to A?

    Retry amplification: 3 attempts at A, each causing 3 attempts at B-to-C, gives up to 9 calls. Retry at one layer, or use budgets.

Summary

  • Slow dependencies are more dangerous than dead ones; always set connect and read timeouts, shorter than your own callers' timeouts.
  • Retry only safe, idempotent calls and only transient errors, with exponential backoff, jitter and a small cap; beware retry amplification.
  • Spring Framework 7 offers @Retryable and @ConcurrencyLimit (enable with @EnableResilientMethods); Resilience4j provides circuit breakers, bulkheads, rate limiters and fallbacks for Boot.
  • A circuit breaker moves between closed, open and half-open, failing fast while the dependency recovers.
  • Use bulkheads to cap each dependency's share of resources, design honest fallbacks, count them, and test with injected failures.

Next: Performance Tuning.