Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 39 ยท Patterns

Event-Driven Patterns

Learn event sourcing, CQRS, the outbox pattern, change data capture, sagas and event design with Kafka, using runnable outbox and event-store stand-ins.

Kafka is a storage and transport layer. What you build on it is up to you, and a handful of patterns come up again and again. They all answer the same underlying question: how do several services, each with its own data, stay consistent without calling each other synchronously for everything?

This lesson covers six ideas: event design, event sourcing, CQRS, the outbox pattern with change data capture, and sagas. For each you will see the problem it solves and its price.

Real versus stand-in. These patterns are design techniques, and the Kafka client code you would write for them was introduced in earlier lessons. The TryIt blocks are JavaScript stand-ins that use arrays and objects in place of a real database and a real topic, so you can watch the behavior, including the failure cases, run deterministically.

Patterns in this lesson

Event sourcing Store the events, derive the state.
CQRS Separate write model from read models.
Outbox Save data and the event in one transaction.
Saga A long business process as steps and compensations.
1 / 2

Designing good events

Before any pattern, get the events right, because they are the long-lived contract between teams.

  • Name events in the past tense, in business language. OrderPlaced, PaymentCaptured, AddressChanged describe facts. UpdateOrder or OrderEvent describe nothing.
  • Make events immutable and self-describing. Include an event id, a type, a timestamp, the entity id and a schema version (see Schemas, Avro, Protobuf and JSON).
  • Choose a key that gives you the ordering you need. Events of one order should share a key such as the order id, so they land in one partition in order (see Message Keys and Partitioners).
  • Event-carried state versus notification. A thin event says "order 1001 changed" and forces consumers to call back for details. A fat event carries the data consumers need, so they can work alone. Fat events cost size but remove coupling and load.
  • Do not leak your database. Publish a deliberate event shape, not a raw table row, or every column rename becomes a breaking change for other teams.
  • One topic per kind of fact. Keep related events of one entity (OrderPlaced, OrderShipped) in the same topic when order between them matters, since ordering exists only inside a partition.

Event sourcing

Normally a service stores the current state of an entity and overwrites it on every change. With event sourcing you store the sequence of events that happened to the entity, and the current state is whatever you get by replaying them. A bank statement is the natural analogy: the balance is not stored, it is the sum of the transactions.

Benefits: a complete audit trail for free, the ability to answer "what did it look like last Tuesday?", and new views of the data built later by replaying history. Costs: replaying long histories is slow unless you take snapshots, changing event shapes takes care, and the programming model is unfamiliar. A Kafka topic fits as an event store because it is an ordered, replayable log (see The Commit Log Idea), though some teams keep the event store in a database and use Kafka to publish from it. Note that retention must be long (or infinite) for a topic that is your source of truth.

CQRS

Command Query Responsibility Segregation splits the model that handles changes (commands) from the models that answer questions (queries). Writes go through one component that validates rules and emits events. Separate projections consume the events and maintain read-optimized views: a table for the order history page, a search index, a dashboard counter. Each can have its own database and shape.

CQRS pairs naturally with event sourcing and Kafka, because projections are just consumers (often in different consumer groups). The trade-off is eventual consistency: right after a write, the read model may not have caught up yet, and your user interface has to cope with that.

Event sourcing and a projection

The stand-in stores events for bank accounts. The state is computed by replaying events, one projection tracks balances, another counts transactions per account, and both can be rebuilt from scratch.

JS Event store and projections

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The "balances after event 4" line is the time travel that event sourcing gives you. The withdraw function is the command side: it checks the rule against the current state and appends an event only when the command is valid. Events that were stored are facts and are never rejected afterwards.

The dual-write problem and the outbox pattern

Here is the most common source of silent inconsistency. A service handles "place order": it must save the order in its database and publish OrderPlaced to Kafka. Those are two systems, and no transaction spans both:

  • Save, then publish: if the process crashes in between, the order exists but no event was ever sent.
  • Publish, then save: if the save fails, other services react to an order that does not exist.

The outbox pattern removes the second system from the critical path. In one database transaction, the service writes the order and inserts a row describing the event into an outbox table. Either both are committed or neither is. A separate relay reads unsent outbox rows and publishes them to Kafka, then marks them as sent.

The relay can crash after publishing but before marking a row as sent. After restart it publishes that row again. So the outbox gives you at-least-once publishing, and consumers must tolerate duplicates, typically by remembering the event ids they have processed (see At-Most-Once, At-Least-Once, Exactly-Once).

JS Outbox, relay crash and deduplication

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Order 2 left no order and no event, which is the whole point of doing both inserts in one transaction. The duplicated event after the relay crash is the price, and the consumer's processed set pays it.

Change data capture

How does the relay read the outbox? One way is to poll the table on a schedule, which is simple but adds latency and query load. The other is change data capture (CDC): a tool reads the database's own transaction log (for example the PostgreSQL write-ahead log or the MySQL binlog) and turns every committed insert, update and delete into an event. Debezium is a well-known open source CDC system that runs on Kafka Connect.

CDC has two common uses. First, it can implement the outbox relay without polling: capture only the outbox table and route its rows to topics. Second, it can replicate whole tables into Kafka so other systems can follow them, which is the subject of Project: Database Sync with Change Events. Be aware that raw CDC events reflect your table structure, so for cross-team contracts the outbox with a deliberate event shape is usually the better interface.

Sagas

A business process such as "place order, charge card, reserve stock, ship" touches several services, each with its own database. There is no single transaction across them. A saga is the alternative: a sequence of local transactions, where each step publishes an event that triggers the next, and every step has a compensating action that undoes it if a later step fails (refund the card, release the stock).

There are two styles. In choreography, services react to each other's events and there is no central owner; it is simple for short flows but hard to follow as they grow. In orchestration, one coordinator service tracks the saga's state and tells each participant what to do next, which is easier to reason about and monitor. Either way you need idempotent handlers (a step can be delivered twice), timeouts (a participant may never answer) and a clear final state ("completed" or "compensated"). Kafka's transactions do not make a saga unnecessary: they cover Kafka reads and writes, not your other databases (see Transactions and Exactly-Once).

When not to use these patterns

Each pattern adds moving parts. If one service owns the data and others only need to be told about changes, an outbox is often all you need. Event sourcing is worth its cost where history itself is valuable (finance, audit, collaborative editing), not for every table. Start simple, and adopt a pattern when you can name the problem it solves for you.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does the outbox pattern solve?

    Writing the event to an outbox table in the same database transaction as the business data makes the two atomic. A relay then publishes it to Kafka.
  2. 2. After a relay crashes between publishing and marking a row as sent, what does the consumer see?

    The outbox is at-least-once. Consumers handle duplicates by remembering event ids or by making the handler naturally idempotent.
  3. 3. In event sourcing, how do you obtain the current state of an entity?

    The events are the source of truth. State is derived by applying them in order, and snapshots just shorten the replay.
  4. 4. Which event name follows good practice?

    Events are facts in the past tense, expressed in business language. OrderShipped says exactly what happened.
  5. 5. What is a compensating action in a saga?

    Since there is no distributed transaction, a saga repairs a failed process by running explicit undo steps such as refunding a payment.

Summary

  • Design events as immutable, past-tense facts with ids, versions and a key that matches the ordering you need.
  • Event sourcing stores the events and derives state by replay; CQRS builds separate read models, accepting eventual consistency.
  • The dual-write problem is solved by the outbox pattern: business data and the event row commit together, and a relay or CDC tool publishes them at-least-once.
  • Consumers deduplicate by event id.
  • Sagas coordinate multi-service processes with local transactions and compensations, in choreographed or orchestrated style.
  • Adopt a pattern for a concrete problem, not by default.

Next, see what to do when processing an event keeps failing in Retries, Dead Letter Topics and Poison Messages.