Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 65 ยท Case Studies: Classic Designs

Design a Payment System

Design a payment service: the payment flow, idempotency keys, a double-entry ledger, state machines, retries, reconciliation and failure handling, with Python simulations.

A payment system moves money: a customer pays a merchant, the merchant's account is credited, fees are taken, and later money is paid out to a bank. Correctness matters more than speed or elegance. A lost write means lost money; a duplicate write means a customer charged twice. Almost every design decision in this case study protects one of two things: never charge twice and never lose track of money.

Requirements

  • Accept a payment from a customer for an order, by card or wallet, through external payment providers.
  • Show payment status, support refunds, and pay out to merchants.
  • Strict consistency for money; every change must be auditable.
  • Reliable despite timeouts and crashes: a request may be repeated or lost at any point.
  • Scale: 1 million payments per day (about 12 per second on average, 100 per second at peak). Modest throughput, extreme correctness.

Architecture

flowchart LR
    C["Checkout"] --> PS["Payment service"]
    PS --> DB[("Payment DB + ledger")]
    PS --> RISK["Risk and fraud check"]
    PS --> PSP["Payment provider (card network)"]
    PSP -- "webhook" --> PS
    PS --> Q["Event queue"] --> OS["Order service"]
    PS --> W["Wallet / merchant balance"]
    REC["Reconciliation job"] --> DB
    REC --> PSP

We never store raw card numbers: the provider tokenizes them, which also reduces compliance scope.

The happy path:

  1. Checkout creates a payment record PENDING with an amount, currency and idempotency key.
  2. The payment service asks risk systems to approve it.
  3. It calls the provider to authorize and capture the card.
  4. The provider answers synchronously (success or decline) or later by webhook.
  5. The service updates the record to SUCCEEDED or FAILED and writes ledger entries in the same database transaction.
  6. An event tells the order service to ship.

Idempotency: never charge twice

Networks fail. If the checkout sends "charge $50" and the connection drops, it cannot know whether the charge happened, so it retries. Without protection the customer pays twice. The fix is an idempotency key: the client generates a unique key per intended payment, and the server stores the key with the result. A second request with the same key returns the saved result instead of charging again.

Python Idempotency keys stop double charges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Also pass the idempotency key on to the provider, since most providers support it, so the guarantee holds across the whole chain.

The ledger: double-entry bookkeeping

Do not store only a "balance" number that code increments. Keep an append-only ledger where every transaction is recorded as at least two entries that sum to zero: a debit from one account and a credit to another. Balances are derived by summing entries. Entries are never updated or deleted; a mistake is corrected by a new reversing entry. This gives a complete audit trail and makes errors detectable (the total of all entries must always be zero).

Python A double-entry ledger

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Store amounts as integers in the smallest currency unit (cents), never as floating point, and always record the currency.

State machine and failure handling

A payment moves through CREATED -> PENDING -> AUTHORIZED -> CAPTURED -> SETTLED, or ends in FAILED, CANCELLED, REFUNDED. Allow only valid transitions, and make the write that changes state atomic with the ledger entries (one database transaction, using a relational database with strong consistency).

Failure scenarios to cover out loud in an interview:

  • Timeout calling the provider: status unknown. Mark the payment UNKNOWN, then query the provider using the idempotency key until you learn the real state. Never assume failure.
  • Crash after charging but before saving: a recovery job scans payments stuck in PENDING and asks the provider.
  • Webhook arrives twice, or out of order: make the handler idempotent and check state transitions.
  • Duplicate click: the unique idempotency key and a database unique constraint stop it.
  • Distributed steps (reserve stock, charge card, create shipment): use the saga pattern with compensating actions (for example a refund) rather than a distributed lock.

Reconciliation

Even a perfect system can disagree with the outside world. Each day a reconciliation job compares your records against the provider's and the bank's settlement files, flagging missing, extra or mismatched payments for review. This final safety net catches bugs, fraud and provider errors.

Python Reconciling internal records with a provider file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other considerations

  • Security: use the provider's hosted fields or tokens so card data never touches your servers; encrypt secrets; apply strong authentication (3-D Secure) as required.
  • Risk: velocity limits, device checks and scoring before charging.
  • Currencies and taxes: store the currency with every amount; convert only at defined moments with recorded rates.
  • Payouts: batch merchant payouts on a schedule from the ledger.
  • Availability: active-passive databases with synchronous replication for the ledger; choose consistency over availability for the money path.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is an idempotency key for?

    The same key means the same intended payment, so retries are safe.
  2. 2. Why use an append-only double-entry ledger?

    Balances are derived from immutable entries and mistakes are fixed with reversing entries.
  3. 3. The provider call times out. What should the payment be marked as?

    A timeout does not tell you whether the charge happened. Resolve it before acting.
  4. 4. What is the purpose of daily reconciliation?

    It is the safety net that catches mismatches no code path anticipated.

Summary

  • Payments need correctness first: no double charges, no lost money.
  • Idempotency keys make retries safe end to end.
  • Use an append-only double-entry ledger with integer amounts, written atomically with state changes.
  • Model payment states explicitly; treat timeouts as UNKNOWN and recover by querying the provider.
  • Reconcile with providers and banks every day.

Next: Project: A Load Balancer Simulator