Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 65 ยท Asynchronous Systems

Event Sourcing and Event Streaming

Learn event sourcing (state as a log of events), snapshots, replay and projections, and how it differs from event streaming platforms like Kafka, with Python demos.

Most databases store the current state: the account balance is 70. When you update it, the old value is gone. Event sourcing stores something else: the full sequence of facts that led to the state. "Deposited 100", "Withdrew 30". The balance is not stored; it is computed by replaying the events. The log becomes the source of truth.

A similar-sounding idea, event streaming, is about moving a continuous flow of events between systems. The two are often used together but solve different problems, and mixing them up is a common interview mistake. This lesson explains both.

Event sourcing

In an event-sourced system:

  • Every change is recorded as an immutable event in an append-only event store.
  • Current state is derived by applying events in order (state = fold(events)).
  • Events are never updated or deleted. A correction is a new event ("Withdrawal reversed").

Compare a bank ledger with a single balance cell. The ledger explains why the balance is what it is, lets you audit any past moment and can be rebuilt if a bug corrupted a derived number.

Python Rebuilding state by replaying events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Benefits

  • Complete audit trail and the ability to answer "what did we know at 14:00 last Tuesday?" (temporal queries).
  • Debugging by replay: reproduce a bug by replaying real events into a fixed build.
  • New read models any time: build a new projection from history, even for questions you did not think of when the events were written.
  • Natural fit for CQRS and for publishing events to other systems.

Costs

  • Replay is slow for long histories. Solve it with snapshots: periodically save the state at event N, then replay only later events.
  • Event schemas live forever. You cannot change old events, so you version them and write upcasters that translate old versions when reading.
  • Eventual consistency in read models, and a steeper learning curve.
  • Privacy: an immutable log conflicts with "delete my data" laws, so use encryption with deletable keys or keep personal data out of events.
  • Querying current state by arbitrary fields needs a projection; you cannot just run SELECT.
Python Snapshots speed up replay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Event streaming

Event streaming means treating events as a continuous, ordered, durable flow that systems can publish to and read from in real time. Platforms such as Apache Kafka, Pulsar and Kinesis provide partitioned append-only logs with long retention. Consumers track an offset (their position) and can rewind.

Typical uses: feeding analytics from website clicks, change-data-capture from databases, IoT telemetry, log aggregation, and connecting microservices. Stream processors (Kafka Streams, Flink) compute over the flow: counts per minute, joins between streams, alerts.

The difference

Event sourcing Event streaming
What it is A way to store an application's state A way to move events between systems
Scope Inside one service or aggregate Across services and data systems
Source of truth The event log of that service Often a transport; truth may be elsewhere
Events are Domain facts used to rebuild state Any records: clicks, metrics, changes
Retention Forever (or snapshot plus archive) Time- or size-bound, often days
Typical tool Event store, or a table of events Kafka, Pulsar, Kinesis

They combine well: a service uses event sourcing internally and publishes its events to a Kafka topic so others can build views. But you can stream without event sourcing (a CRUD service emitting change events) and event-source without streaming (events in a database table).

Python A log with consumer offsets and replay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the log is retained, each consumer owns its offset and can fall behind, catch up or replay without affecting others. That is the main difference from a classic queue, where a consumed message is gone (see Publish/Subscribe and Message Brokers).

Stream processing basics

A stream processor reads events and keeps state (counts, windows). Common concepts:

  • Windows: tumbling (fixed, non-overlapping, such as each minute), sliding (overlapping) and session windows (grouped by activity gaps).
  • Event time vs processing time: events can arrive late or out of order. Use the timestamp in the event and a watermark to decide when a window is complete.
  • Exactly-once effect: achieved with transactional writes and idempotent sinks, because delivery itself is at-least-once.

Do not adopt event sourcing just because an auditable history sounds nice. A plain table with an audit_log table or a change-data-capture stream gives most of the audit benefit for far less complexity.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In event sourcing, how do you fix a wrong entry?

    Events are immutable facts. Corrections are recorded as new events, which keeps the history honest.
  2. 2. What is a snapshot for?

    A snapshot stores the state at some event number so replay starts from there rather than from the beginning.
  3. 3. Which statement is true?

    They solve different problems and are independent, though often used together.
  4. 4. Why can a Kafka consumer reprocess old data?

    The log is retained for a period, so a consumer can set its offset back and read again.

Summary

  • Event sourcing stores immutable events and derives state by replay; corrections are new events.
  • Snapshots shorten replay; versioning and upcasting handle schema changes forever.
  • Event streaming platforms are retained, partitioned logs with consumer-controlled offsets.
  • Sourcing is a storage strategy; streaming is a transport; they combine well but are independent.
  • Use them when audit, replay or many read models matter, not by default.

Next: Saga Pattern