Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Communication

Idempotency, Retries and Backoff

Make retries safe with idempotency keys and a dedup table, then compare fixed, exponential and jittered backoff in a retry-storm simulation.

Retries are the cheapest way to survive a flaky network, and also one of the easiest ways to knock a struggling service flat. Two ideas make retries safe: idempotency (repeating an operation does no extra harm) and backoff with jitter (spacing the retries out so clients don't stampede).

You saw in Messages, RPC and Delivery Guarantees that at-least-once delivery means duplicates. This lesson shows how to make them harmless and how to retry without causing a retry storm. Both halves come with simulations you can run.

What idempotent means

An operation is idempotent if doing it many times has the same effect as doing it once. Some operations are naturally idempotent, and some are not:

Operation Idempotent? Why
set balance = 100 Yes Same end state however often you run it
delete user 7 Yes (effect) Already gone the second time
add 10 to balance No Every run adds again
append line to file No Every run adds another line
create order (server picks the ID) No Each run creates another order

The fix for the non-idempotent ones is to turn them into idempotent ones, either by describing the target state ("set") instead of the change ("add"), or by attaching an idempotency key.

Idempotency keys

An idempotency key is a unique ID the client generates once per logical operation, and reuses on every retry of that operation. The server stores the result under that key. When the same key arrives again, the server returns the stored result instead of repeating the work.

sequenceDiagram
    participant C as Client
    participant S as Server
    C->>S: charge $20, key=k1
    S->>S: run charge, store k1 -> receipt
    S--xC: receipt (lost)
    C->>S: charge $20, key=k1 (retry)
    S-->>C: stored receipt, no second charge

The retry carries the same key, so the server replays the answer instead of charging again.

Python A payment service with a dedup table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive client needed 2 attempts and was charged twice (balance 60), while the keyed client also needed 2 attempts but was charged once (balance 80).

Rules for a good dedup table

  • Generate the key on the client, once per user intent (a UUID or an order ID), not once per attempt.
  • Store the key and the effect atomically. In a database, write both in one transaction, or put a unique constraint on the key so a concurrent duplicate fails.
  • Keep keys long enough. A key must live at least as long as retries (and any queue redelivery) can arrive. Hours to days is common, then expire them.
  • Reject a reused key with different content. Same key but a different amount means a client bug, so return an error.
  • Handle in-flight duplicates. Two copies may arrive at the same moment. A unique constraint or lock makes one wait for the other.

Which errors are worth retrying?

Not every failure should be retried. Retry transient problems: timeouts, connection resets, "service unavailable", "too many requests". Do not retry errors that will fail the same way again, such as validation errors or "not found". And only retry non-idempotent operations if you carry an idempotency key.

Also always cap the retries: a maximum number of attempts or a total time budget (a deadline). Unlimited retries turn a short outage into a long one.

Backoff: don't retry instantly

If a server is overloaded and every client retries immediately, the load doubles exactly when the server is weakest. Backoff waits longer between each attempt. Exponential backoff doubles the wait each time (for example 100 ms, 200 ms, 400 ms, 800 ms), usually with a cap.

But exponential backoff alone has a flaw. If 1,000 clients fail at the same moment, they all wait the same 100 ms, then all retry together, then all wait 200 ms and retry together. The load arrives in synchronized waves. The cure is jitter: randomness added to each wait so the clients spread out.

Three retry styles

Immediate Retry on the next tick. Fast, but a stampede.
Exponential Wait 1, 2, 4, 8... Gentler, but still synchronized.
Jittered Wait a random time up to the exponential cap.

Full jitter picks a random delay between 0 and the cap.

1 / 2

The "full jitter" formula is simple: delay = random(0, min(cap, base * 2**attempt)).

A retry-storm simulation

Let's model time in ticks. A server can handle only 10 requests per tick and rejects the rest. 200 clients all start at tick 0 and each needs one success. Clients that are rejected retry according to their strategy, up to 12 attempts. We measure the total requests the server received (wasted load), the peak requests in a single tick, and the tick when the last client finished.

Python Comparing retry strategies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table like this. Immediate retries finish quickly, but 80 of the 200 clients run out of attempts, and the server received 1,740 requests to serve 120 of them. Plain exponential backoff has the same totals: the clients stay in lock step, so every wave is exactly as big as the last one minus the 10 that got served, and the waves are just spread further apart (the run takes thousands of ticks). Full jitter breaks the lock step. Clients land on different ticks, so the server's capacity is used on almost every tick; the total drops to 779 requests, the retry peak is lower, nobody gives up, and the run ends at tick 38.

The exact numbers depend on the seed and on the model, but the pattern holds in real systems: backoff alone helps individual clients, and jitter is what helps the server.

Backoff protects the server, but only if everyone does it. For layered systems, retry at one layer only. If three layers each retry 3 times, one user click can become 27 requests downstream.

A reusable retry helper

Here is the shape of a retry loop you would write in real code. It uses a deadline, full jitter, and only retries errors marked transient. Time is fake here so the example is deterministic.

Python A retry helper with full jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which operation is naturally idempotent?

    Setting a value to a fixed target gives the same end state however many times it runs. Adding, appending and creating change the state again each time.
  2. 2. When should the client generate an idempotency key?

    The key identifies the intent. If it changed on each retry, the server could not recognise a retry as a duplicate.
  3. 3. Why add jitter to exponential backoff?

    Without jitter, clients that failed together wait identical times and return together. Randomness spreads their retries over time.
  4. 4. A request fails with a validation error (HTTP 400). What should the client do?

    Only transient errors are worth retrying. A validation error is permanent until the request itself changes.

Summary

  • An idempotent operation has the same effect once or many times; prefer "set to X" over "add X".
  • An idempotency key, chosen once per operation by the client, lets a server recognise retries and replay the stored result.
  • Store the key and the effect atomically, keep keys long enough, and reject a reused key with different content.
  • Retry only transient errors, with a maximum attempt count or deadline.
  • Exponential backoff spaces retries out; adding jitter breaks the lock step that causes retry storms.
  • Retry at one layer only, or the multiplication will surprise you.

Next: Queues, Pub/Sub and Event Logs