Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 40 ยท Payments

Idempotency and Reliable Payments

Make payments safe to retry: idempotency keys, duplicate submits, exponential backoff, exactly-once effects, deduplicating webhook events and the transactional outbox pattern.

Networks fail in the most annoying way: halfway. You send "charge this card $49.99", the connection drops, and you do not know whether the charge happened. If you retry, you may charge twice. If you do not retry, you may lose a sale. Customers add to the trouble by double-clicking the pay button, pressing refresh on a slow page, or tapping "back" and submitting again.

The cure is idempotency: an operation is idempotent if doing it several times has the same effect as doing it once. Setting a value to 5 is idempotent; adding 5 is not. Payments are naturally "add 5" operations, so we have to make them idempotent on purpose. This lesson covers the tools: idempotency keys, safe retries, deduplicating incoming events, and the outbox pattern for reliable messaging. It continues from Webhooks and Signature Verification.

What can go wrong

sequenceDiagram
    participant S as Your shop
    participant P as Payment provider
    S->>P: Charge $49.99
    P->>P: Card charged
    P--xS: Response lost (timeout)
    Note over S: Did it work? Unknown
    S->>P: Retry: charge $49.99
    P->>P: Card charged again

Without an idempotency key, the retry creates a second charge.

The same shape appears in many places: the browser sends a checkout request twice, a queue worker crashes after sending an e-mail but before acknowledging the job, a provider delivers the same webhook again. In every case one logical action produced two physical attempts.

Idempotency keys

An idempotency key is a unique string the client generates for one logical action and sends with every attempt of that action, usually in an Idempotency-Key header. The server remembers the result of the first attempt under that key. A later request with the same key does not run the action again; it returns the stored result.

The rules a good implementation follows:

  • The client makes the key, once per user intention (for example when the checkout form is first rendered or the cart becomes an order), and reuses it for every retry. A random UUID works well. A fresh key per retry defeats the whole idea.
  • The server stores key, request fingerprint and response. The fingerprint (a hash of the method, path and body) detects misuse: the same key with a different body is a bug, and should fail with an error such as 409 or 422 rather than silently return an unrelated result.
  • Keys expire. Keeping them for 24 hours to a few days is typical.
  • Failures are stored too, if they are final. A declined card replays as the same decline. A crash before the outcome was known is different: the key is still "in progress" and the retry must wait or be told to try again shortly.
  • Scope keys per account, so one customer's key can never collide with or reveal another's result.
Node.js A payment endpoint with idempotency keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three attempts, one charge. The mismatched request is rejected instead of being answered with the old payment, and a missing key is refused outright, which forces every client to think about retries.

This in-memory map is only a model. A real server stores keys in a database with a unique constraint, and "check, then charge, then save" must be protected against two simultaneous requests with the same key. Insert the key row first (the unique constraint lets only one request win), mark it "in progress", do the work, then store the result. The loser either waits or answers 409 ("request in progress, retry shortly").

Prevent double submits in the browser too

Server-side keys are the safety net, but reduce duplicates at the source. Disable the pay button after the first click, show a progress state, and generate the idempotency key when the checkout page loads or when the order is created, so a refresh reuses the same order. A common pattern is to create the order (status pending) first and use its id as the natural key for the payment attempt: one order, one successful payment. See The Order Lifecycle and State Machines for the states involved.

Retrying safely

Some failures are worth retrying and some are not.

Outcome Retry?
Network timeout, connection reset Yes, with the same idempotency key
429 Too Many Requests, 503 Yes, after waiting (honor Retry-After)
500 from the provider Yes, a few times
400/422 validation error No, the request itself is wrong
Card declined No, ask the customer for another method

Retries should use exponential backoff with jitter: wait 1 s, then 2 s, then 4 s and so on, each multiplied by a random factor. Without backoff a struggling service is hit harder exactly when it is weakest; without jitter all clients retry in the same instant and create a "thundering herd". Always cap the number of attempts and the total time.

The next example uses a fake provider that charges the card but loses the first response. Time is simulated, so the program is deterministic: the "random" jitter comes from a fixed list.

Node.js Retry with backoff: with and without a key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Without a key the customer is charged twice and the shop sees only the second payment id. With a key the retry receives the stored result of the first attempt. The key is created before the first attempt and reused by the retry loop, never inside it.

"Exactly once" is really "at least once plus dedupe"

Distributed systems cannot deliver a message exactly once over an unreliable network. What you can build is at-least-once delivery (retry until acknowledged) combined with idempotent processing (repeats have no extra effect). The result behaves like exactly-once for the user. Two common ways to make processing idempotent:

  • Natural idempotency. "Set order status to paid" repeated is harmless; "mark payment pay_77 as captured" is harmless. Prefer absolute updates over relative ones ("add 49.99 to the balance").
  • Deduplication. Keep a table of processed ids (event ids, message ids, idempotency keys) with a unique constraint. Insert the id and apply the effect in the same database transaction, so a crash cannot leave one without the other.

For incoming webhooks this means storing event.id as you saw in the previous lesson. For outgoing calls, it means sending an idempotency key. For effects you cannot dedupe (an e-mail already sent), use the id to make a second send unlikely and accept the rare duplicate.

The transactional outbox

Here is a subtler problem. When an order is paid you must update your database and tell other systems (send the receipt, notify the warehouse, publish an event). Doing both is a dual write:

  • Save to the database, then publish: if you crash between the two, the message is lost.
  • Publish, then save: if the save fails, you announced something that did not happen.

A database transaction cannot include a message broker or an e-mail provider. The outbox pattern solves this by writing the message to an outbox table in the same transaction as the business change. A separate relay process reads unsent rows, delivers them, and marks them sent. If the relay crashes after delivering but before marking, it delivers again, so consumers must be idempotent. That gives at-least-once delivery without ever losing a message.

flowchart LR
    A["Checkout service"] -->|"one transaction"| B[("orders +<br/>outbox table")]
    B --> C["Relay worker"]
    C -->|"at least once"| D["Message queue / e-mail / warehouse"]
    D --> E["Idempotent consumers"]

The outbox row and the order commit together; delivery happens afterwards and may repeat.

The Python example uses an in-memory SQLite database (Pyodide includes it). We mark the order paid and write an outbox row in one transaction, simulate a crash in the relay after delivery, and show how a consumer that deduplicates by message id absorbs the duplicate.

Python Outbox relay with a crash and a deduplicating consumer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The message was delivered twice, which is exactly what at-least-once means, but the consumer applied it once. The business change and the promise to announce it were never separated, so nothing was lost.

Putting it together

A reliable payment flow uses all of these ideas at once:

  1. The checkout page creates an order and a payment idempotency key.
  2. The server creates the payment at the provider with that key; retries reuse it.
  3. The provider's webhook, verified and deduplicated by event id, moves the order to paid and writes an outbox row in the same transaction.
  4. A relay delivers the "order paid" message to e-mail and warehouse systems, whose consumers are idempotent.
  5. A reconciliation job compares your records with the provider's to catch anything that slipped through.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Who should generate the idempotency key for a payment, and when?

    The key identifies the intention, so it must stay the same across attempts. A new key per retry would look like a new action and charge again.
  2. 2. A request arrives with a key you have stored, but the amount differs from the original request. What should the server do?

    Reusing a key with a different payload is a client bug. Returning the old result would hide it and running it would break the one-action-one-key rule, so an explicit error is correct.
  3. 3. Which failure should NOT be retried automatically?

    A decline is a final business answer; retrying the same card will not help. Timeouts and 429/503 responses are transient, so retry them with backoff and the same key.
  4. 4. Why add jitter to exponential backoff?

    Without jitter, many clients that failed together also retry together, creating bursts that can keep a recovering service down.
  5. 5. What problem does the transactional outbox pattern solve?

    Writing the message into an outbox table in the same transaction as the change makes the two atomic. A relay then delivers it at least once, and consumers deduplicate.

Summary

  • Idempotent operations can be repeated without extra effects. Payments are not naturally idempotent, so you add idempotency keys.
  • The client creates one key per logical action and reuses it for every retry; the server stores key, request fingerprint and response, and rejects a key reused with a different request.
  • Retry only transient failures, with exponential backoff, jitter and a cap. Never retry final answers such as declines or validation errors.
  • Exactly-once behavior is at-least-once delivery plus idempotent processing: absolute updates and deduplication by id in the same transaction as the effect.
  • The outbox pattern writes outgoing messages in the same transaction as the change; a relay delivers them later, possibly more than once.
  • Combine browser safeguards, server keys, deduplicated webhooks, an outbox and periodic reconciliation.

Next: Shipping Methods and Rates moves from taking the money to getting the parcel to the customer.