Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Resilience

Timeouts, Retries and Exponential Backoff

Make service calls survive failure: set timeouts, retry only what is safe, add exponential backoff with jitter, avoid retry storms and use idempotency keys.

In a distributed system every remote call can be slow, can fail, or can succeed without you finding out. The lessons in The Fallacies of Distributed Computing said this; this lesson starts the Resilience chapter by turning it into habits. Three tools work together: a timeout so you never wait forever, a retry for failures that are likely to be temporary, and backoff with jitter so your retries help the struggling service instead of finishing it off.

Used carelessly, retries are dangerous: they multiply load at exactly the moment a service is weakest, and they can repeat actions that should happen only once. Everything below is about using them with care. All stand-ins run in one Node file with simulated services and a seeded random generator, so the output is the same every time.

Three habits for every remote call

Timeout Never wait without a limit.
Retry Only for temporary, safe failures.
Backoff + jitter Wait longer each time, at random.
1 / 2

Timeouts

Without a timeout, a call to a hung service holds a thread, a connection and a user's patience until something far away gives up, which may be minutes. The slow service then drags down its callers, and their callers. A timeout turns "unbounded" into "bounded".

There are usually several: a connect timeout (how long to establish the connection, short, such as a second), a request or read timeout (how long to wait for the response) and an overall deadline for the whole user request. Pick values from measurements: a timeout slightly above the 99th percentile latency of the callee is a sensible start. Too short causes false failures, too long defeats the purpose.

A timeout does not tell you the other side did nothing. The request may have arrived and been processed while the response was lost or late. Remember this for retries.

Node.js A timeout wrapper around a slow call

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first call gave up quickly instead of waiting the full 200 ms, and the second had enough patience. Note that Promise.race only stops waiting: the slow work keeps running. Real HTTP clients can cancel the request itself, for example fetch(url, { signal: AbortSignal.timeout(500) }); use that where you can.

When to retry

A retry makes sense when the failure is probably transient: a connection reset, a timeout, a 503 Service Unavailable, a 429 Too Many Requests (after the Retry-After time). It makes no sense for permanent failures: a 400 for a bad request, 401, 403, 404, 422. Retrying those only adds load.

Two more rules. Retry only idempotent operations (or give them an idempotency key, below). And retry at one layer only: if the browser, the gateway and three services each retry three times, one failure turns into 3 ร— 3 ร— 3 ร— 3 ร— 3 = 243 calls at the bottom.

Exponential backoff with jitter

If a service is failing because it is overloaded, retrying immediately is the worst thing to do. Exponential backoff waits longer after each failure: for example 100 ms, 200 ms, 400 ms, 800 ms, capped at some maximum. That gives the service time to recover.

Backoff alone has a flaw: if a thousand clients failed at the same moment, they all retry at the same moments too, in synchronized waves. Jitter adds randomness so clients spread out. The popular "full jitter" variant picks a random delay between zero and the current exponential ceiling:

TEXT
delay = random(0, min(cap, base * 2 ** attempt))
Node.js Retry with exponential backoff and full jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Each failure raised the ceiling (100, 200, 400 ms) while the actual wait was a random value below it. The call succeeded on the fourth try. Variants: "equal jitter" keeps half the delay fixed and randomizes the other half, and "decorrelated jitter" bases each delay on the previous one. All of them beat plain exponential backoff because they break synchronization.

Retries and duplicates: idempotency keys

Here is the dangerous case. You send "charge 50", the payment service charges the card, and the response is lost. You see a timeout, retry, and the customer is charged twice. A timeout never tells you whether the work happened.

The fix is an idempotency key: the client generates a unique id for the logical operation and sends it with every attempt (commonly in an Idempotency-Key header). The server remembers the result for each key and returns the stored result when it sees the key again. This is the same idea as the deduplication key in The Transactional Outbox and Idempotent Consumers, and the contracts are discussed in Synchronous Communication with HTTP APIs.

Node.js Lost response: double charge versus idempotency key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Both clients retried after the lost response; only the one with a key avoided the second charge. Generate the key once per logical action (for instance from the order id), never once per attempt, or it protects nothing.

Retry storms and retry budgets

Picture a service that was down for three seconds and then comes back with limited capacity. A hundred clients are waiting to retry. If they all retry on a fixed schedule, the recovering service is hit by the same crowd again and again, and may fall over a second time. This is a retry storm, one of the classic ways a small outage becomes a big one.

The stand-in below simulates this with ticks instead of seconds. The server is down for the first three ticks, then can handle 25 requests per tick and rejects the excess. A hundred clients start at tick 0, and each retries until it succeeds. Strategy A retries at the very next tick. Strategy B uses exponential backoff with full jitter.

Node.js A retry storm versus backoff with jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Compare the two lines. In a typical run the fixed schedule finishes by tick 6 but sends 550 requests (the whole crowd of 100 returns every tick, so the peak is 100). Backoff with jitter needs until tick 16 but sends only 319 requests, and the busiest tick has 28. The demand is spread out, at the cost of some clients waiting longer. Which is better depends on whether your constraint is latency or protecting the dependency, and during an incident protection usually wins.

The other guard is a retry budget: allow retries to be only a fraction (for example 10 percent) of normal traffic, and when the budget is used up, fail fast instead of retrying. During a real outage most retries would fail anyway, so a budget caps the extra load. Service meshes and libraries such as Envoy, gRPC and Resilience4j offer budgets and limits for exactly this reason. The next lessons add more protection: the Circuit Breaker stops calling a service known to be down, and Rate Limiting and Backpressure lets the callee say "enough".

Never retry in a tight loop without a delay and a maximum number of attempts. An unbounded immediate retry loop is a very efficient way to take down a service you depend on.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should every remote call have a timeout?

    Without a limit a slow dependency ties up resources in the caller, and the problem spreads upward.
  2. 2. Which failure is a good candidate for a retry?

    Connection resets and 503 are often temporary. The 4xx errors would fail the same way again.
  3. 3. What does jitter add to exponential backoff?

    Clients that failed together would otherwise retry together. Random delays spread the load.
  4. 4. A payment call times out and you retry it. How do you avoid charging twice?

    The server stores the result under the key and returns it again. A new key per attempt would defeat the protection.
  5. 5. Why is retrying at every layer of a call chain a problem?

    With three attempts at each of five layers, a single failing call is tried 3 to the power of 5 times.

Summary

  • Give every remote call a timeout, and remember that a timeout does not prove the work was not done.
  • Retry only transient failures, only idempotent operations, and at one layer of the call chain.
  • Use exponential backoff with jitter so retries give a struggling service room to recover.
  • Use idempotency keys, created once per logical action, to make retried writes safe.
  • Cap retries with a maximum attempt count and a retry budget to prevent retry storms.

Next: Circuit Breakers, Bulkheads and Fallbacks.