Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 36 ยท Streaming

Streaming Fundamentals

Understand stream processing: events, append-only logs, topics, partitions, consumer offsets and consumer groups, and when streaming beats batch, using a Python stand-in.

Batch jobs look at a bounded pile of data and finish. A stream never finishes: events keep arriving, and your program keeps processing them. Streaming lets a platform react in seconds, power live dashboards, detect fraud while a payment is pending and feed machine learning with fresh features. It also introduces ideas that batch hides: ordering, time, and what to do when the same event is delivered twice.

Streaming platforms such as Apache Kafka, Kinesis or Pulsar and engines such as Flink cannot run inside this site. The examples here use a small in-memory stand-in log written in Python to demonstrate the real concepts: append-only logs, partitions, offsets and consumer groups. Real platforms add durability, replication and scale, but behave the same way conceptually. The Kafka and Flink tutorials on this site cover the real systems.

Events

An event is an immutable fact that something happened: "user 7 clicked buy at 09:00:03". Events are small, timestamped and are never edited; corrections are new events. Treating data as a sequence of events instead of the latest state keeps full history and allows replay.

The log

The central structure is the append-only log: new events are added at the end and receive increasing positions called offsets. Readers keep their own offset, so each reader can progress at its own pace and re-read from any earlier position.

Python A tiny append-only log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the log is not consumed destructively, many independent applications can read the same data: one feeds a dashboard, another a search index, another the warehouse.

Topics and partitions

A topic is a named log for one kind of event (orders, page_views). To scale beyond one machine, a topic is split into partitions, each its own log. The producer chooses the partition with a key: events with the same key go to the same partition, so their relative order is preserved. Ordering is guaranteed within a partition only.

Python Partition events by key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

(We used zlib.crc32 rather than Python's hash() because it is the same on every run.) All of ana's events share a partition and keep their order. Choosing the key is a design decision: key by user_id if you need per-user order; a bad key (such as a constant) puts everything into one partition.

Consumers and consumer groups

A consumer group is a set of workers that share the partitions of a topic. Each partition is read by exactly one worker in the group, so the group's workers process different data in parallel. A second, independent group gets its own full copy of the stream.

flowchart LR
    P0["Partition 0"] --> C1["Worker 1"]
    P1["Partition 1"] --> C2["Worker 2"]
    P2["Partition 2"] --> C2
    subgraph G["Consumer group: billing"]
    C1
    C2
    end

More workers than partitions leaves some idle: partitions are the unit of parallelism.

Python Assign partitions to a group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a worker joins or leaves, the group rebalances: partitions are reassigned. Each consumer tracks its committed offset per partition so a replacement worker resumes where the previous one stopped.

Batch or stream? Decide from the requirement

Requirement Choice
Daily report Batch
Hourly dashboard, minutes of delay acceptable Micro-batch or batch
Alert within seconds Stream
Need exact historical recomputation Batch (or replayed stream)
Many consumers need the same events Stream log

Streaming is harder to operate and debug than batch. Choose it for the latency you actually need. A "micro-batch" approach (run a small batch every minute) often meets "near real time" cheaply.

A stateful stream processor

Most useful streaming jobs keep state: a running count, the last value per key. Here is a consumer that maintains a count of events per user.

Python A consumer with state and offset commits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Committing the offset after processing means a crash re-processes the last batch rather than skipping it. That is "at-least-once" behaviour, and why duplicates are an unavoidable stream concern: the delivery guarantees lesson explains how to cope.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is an offset?

    Offsets number events in a partition; consumers track which offset they have processed.
  2. 2. Where is event ordering guaranteed in a partitioned topic?

    Each partition is an ordered log. Across partitions there is no global order.
  3. 3. Why use a key when producing events?

    The key determines the partition, so related events stay together and ordered.
  4. 4. What happens if a group has more workers than partitions?

    A partition is read by one worker in the group, so surplus workers have nothing to do.
  5. 5. Why commit the offset after processing a batch?

    Committing after processing gives at-least-once behaviour: no loss, but possible repeats.

Summary

  • Streams are unbounded sequences of immutable events kept in append-only logs with offsets.
  • Topics are split into partitions; order holds within a partition, chosen by the message key.
  • Consumer groups share partitions for parallelism; committed offsets allow resuming.
  • Choose streaming only for the latency you need; micro-batches are often enough.

Next: Event Time and Windows