Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 37 ยท Data Consistency

The Transactional Outbox and Idempotent Consumers

Fix the dual-write problem in microservices: save state and an outbox row in one transaction, relay events to the broker, and deduplicate on the consumer.

Almost every service ends up doing two things at once: it changes its own database and it tells the rest of the system about it. "Save the order, then publish OrderPlaced." Those are two different systems, and no transaction spans both. If the process dies between the two steps, you get either an order nobody hears about or an announcement for an order that does not exist. This is the dual-write problem.

The transactional outbox solves it with a simple trick: write the event into a table in the same database and in the same transaction as the business change. A separate relay reads that table and publishes to the broker later. Because the relay can crash and retry, messages are delivered at least once, so the consumers must be idempotent. This lesson builds both halves, and you can run them.

The outbox in four steps

One transaction Save the order and the event together.
Relay A loop reads unsent rows.
Publish Send to the broker, then mark as sent.
Dedupe Consumers ignore repeats.

Atomic write, at-least-once delivery, idempotent handling.

1 / 2

The dual-write problem

Look at the naive version. It saves the order, then publishes. The stand-in below simulates a crash between the two lines, which in real life could be a deploy, an out-of-memory kill or a broker timeout.

Node.js Dual write: the event is silently lost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Order 2 exists, but billing and shipping will never learn about it. Swapping the two steps does not help: publishing first leaves you with an event for an order that was never saved. Retrying inside the request does not help either, because the process itself may be gone.

The outbox table

The fix is to make the intention to publish part of the local transaction. Next to your business tables you add an outbox table with one row per event. The business change and the event row commit together or not at all, and that is something a normal single database does very well.

The stand-in below uses SQLite through Node's built-in node:sqlite module, with an in-memory database. Real services use their regular database (PostgreSQL, MySQL, SQL Server) with the same table design.

Node.js Business row and outbox row in one transaction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The failed second attempt at order 1 left no trace in either table. Every committed order has exactly one pending event, and no event exists without its order. Each outbox row also carries a unique event_id; remember that, because the consumer will need it.

The relay

The relay (also called the message relay or publisher) is a small loop. It reads unsent rows in order, publishes each to the broker, and marks it as sent. There are two common designs:

  • Polling. Query WHERE sent_at IS NULL ORDER BY id every few hundred milliseconds. Simple and works with any database.
  • Change data capture (CDC). A tool such as Debezium reads the database's transaction log and streams new outbox rows to Kafka with no polling. Lower latency, more infrastructure.

Either way there is a window in which the message was published but not yet marked as sent. If the relay crashes inside that window, the next run publishes the row again. That is why delivery is at least once: you trade "might be lost" for "might repeat", and repeats can be handled.

Idempotent consumers

A consumer is idempotent when handling the same message twice has the same effect as handling it once. Some operations are naturally idempotent (SET status = 'paid'), others are not (balance = balance - 10, "send an email"). For the second kind, use a deduplication key: the consumer records the event_id of every message it has processed, in the same transaction as its own state change, and skips any id it has seen.

This stand-in runs the whole path: the relay crashes after publishing the first message, restarts, and publishes it again. Billing receives two copies but charges only once.

Node.js Outbox relay, crash, redelivery and deduplication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The broker holds four messages for three orders, because order-placed-1 was published twice. Billing ends up with exactly three charges. Two details make this correct: the dedupe record and the business change share one transaction (so a crash cannot record "seen" without charging), and event_id comes from the producer's outbox, so both copies carry the same id.

Practical details

  • Order of events. Read the outbox ordered by its auto-increment id and publish with a partition key (for example the order id), so events about one entity stay in order. Ordering across entities is not guaranteed.
  • Cleaning up. Delete or archive sent rows after a retention period so the table stays small.
  • Several relay instances. Use row locks (SELECT ... FOR UPDATE SKIP LOCKED in PostgreSQL and MySQL 8) or run a single relay per service, otherwise two relays can send the same row. Duplicates are tolerated, but needless.
  • Dedupe window. The processed_events table grows forever unless you expire old ids. Keep ids at least as long as the broker could redeliver.
  • Natural idempotency first. If the operation can be written as "set to this state" or "insert with this unique key", you may not need a separate table.
  • The inbox. The consumer-side table is sometimes called an inbox, the mirror of the outbox.

The outbox gives you reliable publishing, not a distributed transaction. The consumer still acts later, so the system is eventually consistent. Long business flows that may need undoing build on top of this in Sagas.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the dual-write problem?

    No single transaction covers both the database and the broker, so a failure between the two steps loses or invents an event.
  2. 2. Why is the outbox row written in the same transaction as the business change?

    A local transaction is atomic, so either both rows are saved or neither is.
  3. 3. The relay publishes a row, then crashes before marking it as sent. What happens after restart?

    The row is still unmarked, so the relay sends it again. That is at-least-once delivery.
  4. 4. Where should a consumer record the processed event id?

    If the id and the state change commit together, a crash cannot leave one without the other.
  5. 5. Which operation is naturally idempotent?

    Setting a value to a fixed state gives the same result however often it runs. The others change the world again on every repeat.

Summary

  • Writing to a database and publishing to a broker cannot be made atomic, so a crash between them corrupts consistency.
  • The outbox pattern stores the event in the same database transaction as the business change.
  • A relay (polling or change data capture) publishes unsent rows and marks them as sent, giving at-least-once delivery.
  • Consumers deduplicate with the producer's event id, recorded in the same transaction as their own work.
  • Reliable publishing is not a distributed transaction; the flow stays eventually consistent.

Next: Event Sourcing and CQRS.