Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: Sessionizing Clickstream Events

Build a clickstream sessionization pipeline: group events into sessions with an inactivity gap, handle late events, compute session metrics, and see the Flink code and runnable stand-ins.

In this project you turn a raw stream of click events into sessions: bursts of activity by one user separated by a period of inactivity. Sessions are the unit analysts care about (session length, pages per session, bounce rate, conversion), and they are a perfect Flink exercise because a session has no fixed size, it can only be closed by silence, and late or out-of-order events can merge two sessions into one.

Flink code is real but read-only; the runnable parts are Java and SQL stand-ins that produce the same results on a fixed data set. It builds on the patterns of Patterns: Deduplication, Enrichment and CEP and the time concepts of the Apache Flink track.

The requirements

Input events: {user, page, ts} where ts is the event time in seconds. A user's session ends when 30 minutes (1800 s) pass without an event. For each session we want: user, start, end, duration, number of events, and whether it was a bounce (a single event).

flowchart LR
    K["Kafka: clicks"] --> W["Watermarks<br/>5 min lateness"]
    W --> B["keyBy user"]
    B --> S["Session window<br/>gap 30 min"]
    S --> M["Aggregate metrics"]
    M --> O["Sink: sessions table"]

The pipeline: key by user, group by inactivity, aggregate, write.

With the DataStream API, the session window is one line; the aggregation function builds the metrics incrementally so the window keeps a tiny accumulator, not every event.

Java
WatermarkStrategy<Click> strategy = WatermarkStrategy
    .<Click>forBoundedOutOfOrderness(Duration.ofMinutes(5))
    .withTimestampAssigner((c, ts) -> c.ts() * 1000)
    .withIdleness(Duration.ofMinutes(2));

DataStream<SessionSummary> sessions = env
    .fromSource(clicksSource, strategy, "clicks")
    .keyBy(Click::user)
    .window(EventTimeSessionWindows.withGap(Duration.ofMinutes(30)))
    .allowedLateness(Duration.ofMinutes(10))
    .aggregate(new SessionAggregate(), new AttachWindowInfo());

SessionAggregate is an AggregateFunction<Click, Acc, Acc> with fields for count, first and last timestamp (merging two accumulators sums counts and takes min and max, which is required because session windows merge). AttachWindowInfo is a ProcessWindowFunction that adds the user and the window's start and end to the result. In SQL, the same job reads:

SQL
SELECT user_id, window_start, window_end, COUNT(*) AS events
FROM TABLE(SESSION(TABLE clicks PARTITION BY user_id, DESCRIPTOR(ts), INTERVAL '30' MINUTE))
GROUP BY user_id, window_start, window_end;

(SESSION as a window table function exists in recent Flink versions; older ones use the SESSION(ts, INTERVAL '30' MINUTE) group window.)

Step 2: how session windows work

Flink does not wait for the gap to elapse before creating a session. Each event creates a tiny window [ts, ts + gap); overlapping windows of the same key are merged. A later event that falls into the gap of an existing session simply extends it, and an out-of-order event can even bridge two sessions into one. The runnable stand-in below sorts nothing in advance: it receives events out of order and merges windows exactly like that.

Java Stand-in: session windows by merging per-event windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Before the late event there are two sessions. After it, the event at 2000 falls within 1800 s of both neighbours (2000 - 600 and 3500 - 2000 are both below 1800), so the windows merge into a single session with five events. In Flink that is exactly why allowedLateness re-fires a window result: the downstream sink must therefore be able to update its earlier row (an upsert by user and session start) instead of only appending.

Step 3: session metrics

The metrics are plain aggregates over a session. You can also compute sessions in a batch query over stored events, which is a useful way to test your logic or to backfill. The trick is the classic "gaps and islands" pattern: flag each event whose previous event for the same user is more than 30 minutes earlier, then take a running sum of the flags to number sessions.

SQL Session metrics with window functions (portable SQL)

Press Run (or Ctrl + Enter) to see the result here.

The result shows ana as one session of five events (every consecutive gap is below 1800, thanks to the event at 2000 that bridges the two bursts) and li with a bounce session followed by a two-event session.

Step 4: production concerns

  • Idle partitions hold back watermarks: set withIdleness so a quiet Kafka partition does not freeze every session.
  • Lateness: allowedLateness re-emits updated sessions; events later than that go to a side output (sideOutputLateData) for a repair job.
  • State size grows with the number of open sessions, so a bot with a million events inside one session keeps an accumulator, not a million events, if you use aggregate rather than a window that buffers all elements.
  • Upsert sink: key the sink table on (user_id, session_start). When a session grows or merges, the new row replaces the old one (a merge also changes the key, so delete the absorbed row: emit a retraction or use a changelog sink).
  • Testing: Flink's test harnesses can drive the window operator with manual watermarks; write tests for the merge case above.

Extensions to try

  1. Add the referrer and compute conversion per session.
  2. Cap the session length at 4 hours by also using a timer.
  3. Emit a live "active users" count with a tumbling window next to the session job (two sinks, one source: a statement set or two branches of the same stream).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do session windows form?

    Merging overlapping per-event windows is how Flink builds sessions of variable length.
  2. 2. Why does allowedLateness require an updatable sink?

    The session result is fired again with the new content, so the sink must upsert.
  3. 3. Why use aggregate() instead of buffering all events in a window?

    An incremental aggregate keeps constant state per session.
  4. 4. In the SQL gaps-and-islands pattern, what does the running SUM of flags produce?

    Each new-session flag increments the counter, so all events of one session share a number.

Summary

  • Sessions are bursts separated by inactivity; Flink builds them by merging per-event windows.
  • Use bounded-out-of-orderness watermarks with idleness, allowedLateness and a side output for very late data.
  • Use aggregate with a mergeable accumulator, and key the sink so updated sessions overwrite earlier rows.
  • The same result is reproducible in batch SQL with LAG and a running sum, a good way to test and backfill.

Next: Project: Real-Time Fraud Alerts