Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 39 ยท Projects

Project: Real-Time Page-View Analytics

Build a streaming analytics pipeline: page views counted per minute in tumbling windows, late events, a state store with a changelog, and an output topic, with runnable stand-ins.

A website wants a live dashboard: how many times was each page viewed in each minute? Views come from many servers, they can arrive out of order, and the counting service must survive restarts without losing or double counting anything. This is the classic job for stream processing, and in this project you build it step by step.

You will meet event time, tumbling windows, late events and the grace period, a state store backed by a changelog topic, and an output topic that a dashboard can read. If any of those words is new, read Kafka Streams and Stream Processing first; this project puts it to work.

Real versus stand-in. The topic commands and the Kafka Streams Java code below are real usage, shown read-only because the site cannot reach a cluster or run the JVM library. The TryIt blocks are hand-written JavaScript stand-ins that model the same logic with arrays and maps: windows, stream time, a state store, a changelog and an output topic. They are not the Kafka Streams library.

The analytics pipeline

page-views One record per page view.
Windowed count Views per page per minute.
State store Counts live next to the code.
page-view-counts Output topic for the dashboard.
1 / 2

Step 1: design the topics and the event

Two topics carry the data:

  • page-views is the input. Each record is one view, keyed by the visitor id (so one visitor's views stay in order), with the page and the time the view happened in the value.
  • page-view-counts is the output. The key is page@windowStart and the value is the count so far.
JSON
{ "visitor": "v42", "page": "/pricing", "ts": 1767225641000 }

The ts field is the event time: when the view really happened in the browser or on the web server. It is not the time Kafka received the record. The two differ when a mobile app is offline for a while or a server batches its events, and windows must follow the event time, otherwise a delayed view would be counted in the wrong minute.

Terminal
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic page-views --partitions 4
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic page-view-counts --partitions 4

(Add --replication-factor 3 on a real cluster; see Replication, Leaders and ISR.)

Step 2: assign events to tumbling windows

A tumbling window has a fixed size and no overlap. With one-minute windows, an event at second 66 belongs to the window that starts at second 60 and ends at second 120. The start is the event time rounded down to a multiple of the window size.

JS Which window does an event belong to?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Windows are half-open: the event at exactly t=60 belongs to [60, 120), and t=119 is still in the same window. The pair page@windowStart is the natural key for the running count, which is why the output topic uses it.

Step 3: the Kafka Streams version

This is what the real application looks like. It is a few lines because the library takes care of the state store, the changelog and the recovery.

Java
StreamsBuilder builder = new StreamsBuilder();

KStream<String, PageView> views = builder.stream(
    "page-views",
    Consumed.with(Serdes.String(), pageViewSerde)
            .withTimestampExtractor(new PageViewTimestampExtractor())); // use the ts field

views
    .groupBy((visitor, view) -> view.page(), Grouped.with(Serdes.String(), pageViewSerde))
    .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(1), Duration.ofSeconds(10)))
    .count(Materialized.as("views-per-minute"))
    .toStream()
    .map((window, count) -> KeyValue.pair(window.key() + "@" + window.window().start(), count))
    .to("page-view-counts", Produced.with(Serdes.String(), Serdes.Long()));

Three things to notice. The timestamp extractor tells Streams to use the ts field instead of the record's own timestamp. groupBy with a new key (the page) makes Streams write the records through an internal repartition topic, so all views of one page are counted by the same task. And Materialized.as("views-per-minute") names the state store; Streams creates a changelog topic for it automatically.

Step 4: late events, stream time and the grace period

Events do not arrive in order. A view with t=55 can reach the processor after one with t=66. Should the window [0, 60) still accept it?

Streams keeps stream time: the largest event time seen so far. A window stays open for late events until stream time reaches the window's end plus the grace period. After that it is closed, and anything later for that window is dropped. This trades completeness against delay: a longer grace period accepts more stragglers but keeps state around longer.

The stand-in below is the whole counting service. It processes ten views in arrival order, keeps counts in a state store, writes every change to a changelog and to the output topic, and drops events that arrive too late. Then it simulates a crash and restores a fresh instance from the changelog.

JS Windowed counting with late events and a changelog

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the output from the top. The view /home at t=55 arrives after t=66, but stream time is only 66, which is below 70 (the end of [0,60) plus the 10 second grace), so it still raises /home@0 to 3. Then /docs at t=70 pushes stream time to 70 and closes the first window. When /pricing at t=50 shows up afterwards, it is dropped: nine records reach the output topic, one is dropped. Near the end /home at t=62 arrives while stream time is 125, yet its window [60,120) closes only at 130, so it is counted.

The last lines show the recovery idea. The changelog holds every update, so a replacement instance rebuilds an identical store just by replaying it. In real Streams that replay happens automatically before the new instance processes its first record, and the changelog is compacted so only the latest count per key is kept (see Log Compaction and Tombstones).

Step 5: updates versus final results

Look again at the output topic in the stand-in: it received /home@0 = 1, then = 2, then = 3. By default Streams emits an update every time a count changes. That suits a live dashboard, which just keeps the newest value per key. But a billing system wants one final number per window, emitted once the window is closed and can no longer change.

In Kafka Streams you ask for that with suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded())) after the count. The stand-in below does the same by hand: it keeps counts only for windows that are still open and emits a single record when stream time passes the window's end plus grace.

JS Emit one final count per window

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice the cost. Nothing is emitted for /home@0 until stream time reaches 70, and the windows at 60 and 120 appear only when a single view with t=200 arrives. Final results are delayed by the grace period, and they need new events to move stream time forward. On a quiet site a window may stay open for a long time. That is the price of a result that never changes.

Step 6: from prototype to production

  • Partitions and scaling. Run several instances of the application with the same application.id. They split the input partitions like a consumer group (see Consumer Groups and Rebalancing), and each owns its share of the state.
  • Standby replicas. Set num.standby.replicas=1 so another instance keeps a warm copy of each store and a failover does not wait for a long changelog replay.
  • Retention of windows. Old windows must eventually leave the store. Streams deletes a window from the store after its retention period, which must be at least the window size plus the grace period, so disk use stays bounded.
  • Exactly once. Counting must not double count after a crash. Set processing.guarantee=exactly_once_v2 (see Transactions and Exactly-Once); note that readers of the output topic then need isolation.level=read_committed.
  • Monitoring. Watch the consumer lag of the application's group and the restore time after restarts (see Monitoring, Lag and Troubleshooting).
  • Dashboard. A small service consumes page-view-counts and keeps the newest value per key in a cache or database, which is exactly the table view of a topic.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should windows use the event time instead of the time the record reached Kafka?

    Events can be delayed by offline clients or batching. Counting by arrival time would put them in the wrong window and change the results.
  2. 2. With 60 second windows and a 10 second grace period, until which stream time does the window [0, 60) accept late events?

    A window closes when stream time reaches its end plus the grace period, here 60 + 10 = 70. Events for it arriving after that are dropped.
  3. 3. How does a replacement instance get the counts of a crashed instance?

    Every update to the store is also written to a compacted changelog topic. Replaying it rebuilds the same state.
  4. 4. Why does an output topic with updates contain several records for the same key, such as /home@0?

    By default each change produces an update record. Consumers keep the newest per key, or the application uses suppression to emit one final record per closed window.

Summary

  • Windows group events by event time; a tumbling window start is the event time rounded down to a multiple of the window size.
  • Stream time is the largest event time seen; a window closes when stream time reaches its end plus the grace period, and later events for it are dropped.
  • Counts live in a local state store whose every update is logged to a compacted changelog topic, so a new instance can restore it.
  • By default the output topic carries updates; suppression gives one final result per window, at the cost of delay.
  • A production version adds standby replicas, window retention, exactly-once processing and lag monitoring.

Next, keep a database and a Kafka topic in sync in Project: Database Sync with Change Events.