Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 48 ยท Projects

Project: Real-Time Event Counter

Build a real-time page view counter with Structured Streaming: running counts, one-minute windows, late events, watermarks and output modes, with runnable stand-ins.

Dashboards that show "page views in the last few minutes" are a perfect fit for streaming. Events arrive continuously from websites and apps, and the numbers must stay up to date without re-reading everything. In this project you build a real-time event counter with Structured Streaming: first simple running counts per page, then counts in one-minute windows, then the hard part, late events, which arrive after the time period they belong to has already passed.

Streaming needs a live source, a running query and a place to write results, none of which exist on this site. So the Structured Streaming code below is read-only. The runnable examples are micro-batch simulators in plain Python: they take small batches of events, update state and print what Spark would emit after each batch. Everything about windows, watermarks and output modes can be explored this way. If any of the streaming vocabulary is new, read Structured Streaming Basics and Event Time, Windows and Watermarks first.

The design

Every event is a small JSON record: when it happened (event_time), which page and which user. Events travel through Kafka or land as files, Spark reads them in micro-batches, updates counts that it keeps as state, and writes the changed counts to a sink that the dashboard reads.

flowchart LR
    S["Events<br/>(Kafka or files)"] --> M["Micro-batch<br/>every 10 s"]
    M --> A["Windowed count<br/>per page and minute"]
    A <--> ST["State store<br/>+ checkpoint"]
    A --> K["Sink:<br/>database or topic"]
    K --> D["Dashboard"]

Spark reads a batch of new events, updates the state and writes only what changed.

Here is the whole query in PySpark. Again, this is the real code and it does not run here:

Python
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.types import StructType, StructField, StringType, TimestampType

spark = SparkSession.builder.appName("event-counter").getOrCreate()

schema = StructType([
    StructField("event_time", TimestampType()),
    StructField("page", StringType()),
    StructField("user", StringType()),
])

events = (spark.readStream
          .format("json")
          .schema(schema)                       # streaming file sources need an explicit schema
          .option("maxFilesPerTrigger", 1)
          .load("/data/events/"))

counts = (events
          .withWatermark("event_time", "2 minutes")
          .groupBy(F.window("event_time", "1 minute"), "page")
          .count())

query = (counts.writeStream
         .outputMode("update")
         .format("console")
         .option("truncate", False)
         .option("checkpointLocation", "/chk/event-counter")
         .trigger(processingTime="10 seconds")
         .start())

query.awaitTermination()

The three ingredients are withWatermark (how long to wait for late data), groupBy(window(...), "page") (the counting) and outputMode("update") (what to write after each batch). We will build each in turn.

Step 1: Running counts

Start without windows: count all events per page since the start. The state is one number per page, and Spark adds each batch to it.

Python
running = events.groupBy("page").count()
running.writeStream.outputMode("complete").format("console").start()

Output modes decide what is written after each batch. complete rewrites the whole result table every time, update writes only rows that changed in this batch, and append writes rows once, when they can never change again. A running count that never expires can only use complete or update. The trade-off: this state never shrinks. With millions of distinct pages, it grows for as long as the query lives.

The stand-in below simulates three micro-batches and prints both modes side by side.

Python Running counts: update versus complete output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Update mode sends little data to the sink, which is why it is the usual choice for dashboards. Complete mode is simple but gets heavier as the table grows.

Step 2: Tumbling windows

A dashboard rarely wants "since forever". It wants "per minute". A tumbling window is a fixed, non-overlapping slice of event time: [12:00, 12:01), [12:01, 12:02) and so on. F.window("event_time", "1 minute") assigns each event to the slice that contains its timestamp. Event time is when the event happened, which is not when Spark processes it: a phone that was offline sends yesterday's clicks today.

That difference is the reason for the next section.

Step 3: Late events and the watermark

If events always arrived in order, a window could be closed as soon as the clock passed its end. They do not. Network delays, retries and offline devices mean a click from 12:00:30 can show up after events from 12:03. Spark must keep each window's count in state in case more data arrives, and cannot do that forever.

A watermark is the rule that decides when to stop waiting. With withWatermark("event_time", "2 minutes"), Spark tracks the highest event time seen so far (the newest event) and sets:

TEXT
watermark = maximum event time seen so far - 2 minutes

A window whose end is older than the watermark is considered finished: its state is deleted and events that still belong to it are dropped. Windows that end after the watermark stay open and keep accepting late events. The watermark is computed from the previous batches, so it only ever moves forward.

flowchart LR
    E["New event<br/>time t"] --> Q{"Does its window end<br/>after the watermark?"}
    Q -->|"yes"| U["Add to window count"]
    Q -->|"no"| X["Drop: window already closed"]
    U --> W["After the batch:<br/>watermark = newest event time - delay"]
    X --> W

Each event is checked against the watermark from the previous batch; afterwards the watermark advances.

The trade-off is yours to choose. A long delay (10 minutes) tolerates very late events but keeps state for longer and delays final results. A short delay (30 seconds) saves memory and reports earlier but drops more late data. Pick it from how late your real data is, which you can measure.

The simulator below implements exactly this logic: tumbling windows of 60 seconds, a 120-second watermark delay, state per (window, page), and a log of dropped events. Times are seconds after 12:00:00. Batch 3 contains an event from 12:00:30, late but still within the delay, and batch 5 contains one from 12:00:50 that comes too late.

Python Windowed counts with a watermark

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output batch by batch. The late home event at 30 seconds in batch 3 is accepted, because the watermark is still negative. By batch 5 the watermark (90 seconds, computed after batch 4) is past the end of the first window, so a cart event at 50 seconds, belonging to a window that ended at 60, is dropped. The number of windows in state stays small, because finished windows are removed.

Step 4: Update versus append

With windows and a watermark, append mode becomes available. It emits a window's row once, after the watermark has passed the window end and the count can no longer change. update mode emits a row every time its count changes, which gives a live dashboard, but the same window appears several times with growing numbers.

update append
When a row is written Every batch in which it changed Once, after the watermark passes the window end
Latency Low: live numbers Higher: waits at least the watermark delay
Sink must handle Repeated rows for the same window (upserts) Plain inserts, never changes
Good for Live dashboards with a database sink Files, append-only topics, final reports

The second simulator shows the append behaviour: it only prints a window when it is final.

Python Append mode: results only when they are final

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Append mode is quiet in the first batches: it prints nothing until the watermark has passed the end of the first window. Each window then appears exactly once with its final count. Compare it with the update-mode output above, where the 12:00 window kept changing.

Make it reliable

Four settings turn the toy into a production query:

  • Checkpoint location: Spark stores the source offsets and the state there, so a restarted query continues where it stopped and does not recount. Never delete it casually, and do not reuse one path for two queries.
  • Replayable source and idempotent sink: Kafka or files can be re-read, and a sink that can handle the same batch twice (an upsert keyed by window and page) gives end-to-end exactly-once results. See Streaming with Kafka and Delta.
  • Triggers: processingTime="10 seconds" runs a batch every ten seconds. availableNow=True processes all waiting data and stops, which turns streaming code into an incremental batch job.
  • Monitoring: query.lastProgress and query.status report input rate, processing time and state size. Watch for processing time that grows beyond the trigger interval, and for state size that never drops.

For a database dashboard use foreachBatch, which hands you each micro-batch as an ordinary DataFrame:

Python
def upsert(batch_df, batch_id):
    (batch_df.select(F.col("window.start").alias("window_start"), "page", "count")
             .write.mode("append").jdbc(JDBC_URL, "page_views_staging", properties=PROPS))
    # then run a MERGE into the real table keyed by (window_start, page)

counts.writeStream.outputMode("update").foreachBatch(upsert).start()

For quick tests, use the rate source (spark.readStream.format("rate").option("rowsPerSecond", 5)) which generates data without any files, and the memory sink (.format("memory").queryName("counts")) so you can query the result with spark.sql("select * from counts"). Calling query.processAllAvailable() waits until the current data is processed, which makes streaming tests deterministic.

Real versus stand-in. The simulators model the logic of a single aggregation: event-time windows, a watermark that is the newest event time minus a delay, dropping closed windows and the three output behaviours. Real Structured Streaming also checks the watermark per query across partitions, stores state in a fault-tolerant state store, writes a checkpoint, plans batches from source offsets and runs on a cluster. The event numbers shown here are made-up data.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A query uses withWatermark("event_time", "10 minutes"). The newest event seen so far is at 12:30. What is the watermark?

    The watermark is the newest event time minus the delay: 12:30 minus 10 minutes is 12:20.
  2. 2. Which output mode writes a window's row only once, after it can no longer change?

    With a watermark, append mode waits until the watermark passes the end of the window and then emits its final count one time.
  3. 3. What happens to an event whose window ended before the current watermark?

    Closed windows have had their state removed, so events that belong to them are ignored.
  4. 4. Why does a running count without windows become a problem on a long-lived query?

    Without windows or a watermark, nothing tells Spark that old keys can be removed, so state grows with the number of distinct keys.
  5. 5. What makes a restarted streaming query continue from where it stopped?

    The checkpoint stores progress and state, so a restart resumes without recounting or losing data.

Summary

  • A real-time counter reads events in micro-batches, keeps counts as state, and writes only what changed to a sink the dashboard reads.
  • Running counts per key are simple but their state never expires; windows over event time bound the state.
  • A watermark is the newest event time minus a delay; windows that end before it are closed, their state is deleted and later events for them are dropped.
  • update mode writes changed rows each batch for live dashboards, append writes each window once when final, and complete rewrites everything.
  • Checkpoints, replayable sources and idempotent sinks (often via foreachBatch) give reliable exactly-once results.

Next: switch from counting to learning in Project: Movie Recommendations with ALS.