Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 65 ยท Reliability and Operations

Retries, Timeouts and Dead Letter Queues

Build fault-tolerant calls: timeouts, retries with exponential backoff and jitter, idempotency, retry storms and dead letter queues, with runnable Python simulations.

In a distributed system, calls fail constantly: a packet is dropped, a server restarts, a database is briefly overloaded. Most of these failures are transient, which means that trying again a moment later will work. That makes retries the simplest and most common reliability tool, and also one of the easiest to get wrong. A careless retry turns a small hiccup into a self-inflicted outage.

This lesson covers the three companions of safe failure handling: timeouts (do not wait forever), retries (try again, wisely) and dead letter queues (park what cannot succeed).

Timeouts: never wait forever

Every network call needs a timeout. Without one, a hung server ties up a thread, a connection and memory of its caller, and slowness spreads upstream until everything is stuck. A few rules:

  • Set a timeout on every remote call: connect timeout and read timeout.
  • Pick values from measured latency, for example a bit above the 99th percentile, not a round guess.
  • Use a deadline for the whole user request and pass the remaining time downstream, so inner calls do not outlive the request.
  • A timeout does not tell you whether the operation happened. The server may have completed it after you gave up. That is why retries need idempotency.

Retries: when and how

Retry only when it can help:

  • Retry: timeouts, connection resets, HTTP 503, 429 (after waiting), brief database failovers.
  • Do not retry: 400 Bad Request, 401/403, 404, validation errors: the same request fails the same way.
  • Careful with non-idempotent operations. Retrying "charge $20" can charge twice unless the call carries an idempotency key that lets the server detect the repeat.

Backoff and jitter

Retrying immediately is nearly useless (the problem is probably still there) and harmful (many clients hammering a struggling server). Use exponential backoff: wait 100 ms, then 200, 400, 800... up to a cap. Add jitter (randomness) so thousands of clients that failed together do not all retry at the same instant, which would create waves of load called a thundering herd.

Python Backoff with jitter spreads retries out

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without jitter, every client retries in the same instant. With jitter they spread across the whole window.

Retry budget

Retries multiply load. If each of three layers retries three times, one user request can become 27 calls to the deepest service. Limit this: retry at one layer only, cap attempts (often 2 to 4), and use a retry budget that allows retries only up to, say, 10 percent of regular traffic.

Python A retry loop with idempotency

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Retrying a call whose response was lost, without an idempotency key, charges the customer repeatedly. The same lesson from Message Queues applies to every retry.

Dead letter queues

Some work will never succeed no matter how often you retry: a message with corrupt data, a reference to a deleted record, a bug in the consumer. If it keeps cycling it wastes capacity and blocks the queue (a poison message). The fix is a dead letter queue (DLQ):

  1. A message is attempted up to a limit (for example 5 times, with backoff).
  2. After the limit, the broker or consumer moves it to the DLQ.
  3. Engineers alert on DLQ depth, inspect messages, fix the bug or data and redrive them back to the main queue.
flowchart LR
  Q[(Main queue)] --> W[Worker]
  W -->|ack on success| Done[Done]
  W -->|fail, retries left| Q
  W -->|retries exhausted| D[(Dead letter queue)]
  D --> H[Human or tool: inspect, fix, redrive]

Failed messages retry with backoff, then park in the DLQ instead of blocking the queue.

Python Retry limits and a dead letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keep the original payload, error and attempt count in the DLQ so it is useful for debugging. Note that DLQs also apply to event-driven systems: an event that can never be processed should not stop the stream behind it.

Retry storms and when to stop

When a dependency is down, thousands of clients retrying make recovery harder. Two protections:

  • Backoff with jitter and caps, as above.
  • Circuit breakers that stop calling a failing dependency for a while (next lesson).

Also consider hedged requests (send a duplicate to a second replica if the first is slow, for latency-sensitive reads only) and load shedding (a server that is overloaded returns 503 quickly instead of queueing work it cannot finish).

Make retry behavior part of the API contract: document which errors are retryable, send Retry-After on 429 and 503, and accept an Idempotency-Key header on POST requests that create things.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which failure should NOT be retried?

    A malformed request fails identically every time. Retry only transient errors that a later attempt might fix.
  2. 2. Why add jitter to exponential backoff?

    Without jitter, clients that failed together retry together, creating synchronized spikes (thundering herd).
  3. 3. What does an idempotency key let a server do?

    The server stores the result by key and returns it for repeats, so retries after a lost response are safe.
  4. 4. What is a dead letter queue for?

    After retries are exhausted, messages move to the DLQ for inspection and redrive.
  5. 5. Three layers each retry 3 times. One request can cause up to how many calls to the deepest service?

    Retries multiply across layers: 3 x 3 x 3 = 27. Retry at one layer and use budgets.

Summary

  • Put a timeout on every call and use an overall deadline; a timeout does not mean the work did not happen.
  • Retry only transient failures, with exponential backoff, jitter, attempt caps and retry budgets.
  • Use idempotency keys so retries of non-idempotent operations are safe.
  • Dead letter queues contain poison messages; alert on their depth and redrive after fixing.
  • Pair retries with circuit breakers and load shedding to avoid retry storms.

Next: Circuit Breakers and Bulkheads