Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 47 ยท Projects, Practice and Review

Project: Clickstream Sessionization

Group raw click events into user sessions with a secondary sort, a 30-minute gap rule, stream-style session windows and a columnar Parquet-style output.

A website records every click as an event: user, time, page. Analysts rarely care about single clicks. They want sessions: "Ada visited for 12 minutes, viewed four pages and bought something." A session is a run of events by the same user with no long pause between them. Sessionization turns an endless list of events into a table of visits, and it is a classic big data job because it needs events grouped by user and ordered by time.

In this project you build it with the secondary sort pattern, implement the 30-minute inactivity rule, see how the same logic looks as a stream-style session window, and write the result in a column-oriented layout like Parquet.

Everything runs in a Python simulator: a few dozen events in memory instead of billions of events in HDFS, and a toy columnar writer instead of a real Parquet file. The partitioning, sorting and grouping rules are the real ones that Hadoop applies to a composite key.

The challenge

What we need

Group by user All events of one user in one place.
Sorted by time Events arrive in order, without sorting in memory.
Cut at gaps Over 30 idle minutes starts a new session.
1 / 2

Step 1: Secondary sort

The reducer receives, for each key, a list of values. MapReduce sorts the keys, but the values of a key arrive in no particular order. If the values are user events that you need in time order, you have two options. You could load all of a user's events into memory and sort them, which fails for a user with millions of events. Or you can use a secondary sort, which makes the framework do the sorting for you:

  1. Make the key a composite: (user, timestamp).
  2. Use a partitioner that looks only at the user, so every event of one user reaches the same reducer.
  3. Sort by the full composite key, so each user's events arrive in time order.
  4. Use a grouping comparator that compares only the user, so one reduce call sees all of that user's events in a single iteration.

In Hadoop these are setPartitionerClass, setSortComparatorClass and setGroupingComparatorClass on the job (read-only):

Java
job.setPartitionerClass(UserPartitioner.class);          // user only
job.setSortComparatorClass(UserTimeComparator.class);    // user, then time
job.setGroupingComparatorClass(UserGroupComparator.class); // user only

The simulator below does the same: it partitions by user, sorts by the composite key, then groups by user without ever sorting inside the reducer.

Python Secondary sort: user first, time second

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check the output: each user appears under exactly one reducer, and each timeline is in ascending time order even though the input was shuffled.

Step 2: Cutting sessions at gaps

The reducer walks each user's sorted timeline and starts a new session whenever the gap to the previous event is larger than the inactivity timeout (30 minutes is the common default). For each session it emits the user, start, end, number of events and the pages. Because the walk is a single pass over a sorted stream, the reducer holds only the current session in memory, however long the user's history.

Python Session gap logic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ada has a gap of 168 minutes between 612 and 780, so she gets two sessions. Bob's step from 605 to 640 is 35 minutes, which is more than 30, so his visit splits as well. A session of one event has duration 0, a well-known limitation: time on page of the last page is unknown.

Step 3: The stream-processing view

If events arrive continuously, you cannot sort a whole day first. Stream processors (Flink, Spark Structured Streaming, Beam; see Batch vs Stream Processing and Spark, Flink and What Came After MapReduce) offer session windows: a window per key that stays open while events keep arriving and closes after GAP of silence, as judged by event time and the watermark.

The logic differs in one way: events can arrive out of order, so a late event may join two sessions that looked separate. Session windows therefore merge: when an event falls within GAP of two existing windows, it fuses them into one. The simulator below processes events one at a time in arrival order and merges windows as needed.

Python Session windows that merge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Watch the last two lines: before the late event there are three separate windows, and the event at minute 625 sits within 30 minutes of both the first (600) and the second (655), so those two fuse into one session while the one at 700 stays alone. Batch sessionization on the complete data gets this right automatically because it sorts first; a streaming job must be designed for the merge, and it pays for that with state that stays around until the watermark passes.

Step 4: Writing the result in a columnar layout

The sessions table is queried by analysts: "average session length by day", "share of sessions that reach the cart". Those queries read a few columns of many rows, so write the output in a column-oriented format such as Parquet or ORC (see Parquet and ORC Internals and Row vs Column Formats). Also partition the output directory by date, so a query for one day reads one folder (see Partitioning, Bucketing and File Layout).

The stand-in below turns session records into a toy columnar form: one list per column, with dictionary encoding for the repetitive user column and a min/max statistic per column that a reader can use to skip data.

Python A toy columnar output with statistics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A real writer adds row groups, compression and a footer, but the benefit is what you see here: the average-duration query reads one column instead of every cell of every row.

Putting it all together

The full pipeline: raw click logs land in HDFS, a map-only job parses and cleans them (and drops bots), a MapReduce job with the secondary sort cuts sessions, and a final step writes Parquet partitioned by session date. Downstream, Hive or Spark SQL computes funnels and engagement metrics. Edge cases to decide up front: sessions spanning midnight (assign to the start date), users without a login (use a cookie ID, and accept that one person may appear as several users), and bot traffic.

Extensions to try

  1. Add the inactivity timeout as a parameter and chart the number of sessions for 10, 30 and 60 minutes.
  2. Add a "conversion" flag when a session contains /checkout.
  3. Cap a session at a maximum length of two hours, even without a gap.
  4. Add a session id (user plus start time) to every event, producing the enriched event table.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the purpose of a secondary sort?

    Putting the timestamp into a composite key lets the framework's own sort order the values, so the reducer never holds them all in memory.
  2. 2. Which component must look only at the user part of the composite key?

    The partitioner routes all of a user's events to one reducer, and the grouping comparator makes them one reduce call. The sort comparator uses user and time.
  3. 3. A user has events at minutes 100, 120 and 170 with a 30-minute timeout. How many sessions?

    100 to 120 is a 20-minute gap (same session); 120 to 170 is 50 minutes (new session), so there are two sessions.
  4. 4. Why do streaming session windows need to merge?

    Events can arrive out of order, so a late event within the gap of two windows joins them into one session.
  5. 5. Why write sessions as Parquet partitioned by date?

    Columnar layout lets analytic queries read few columns, and date partitions let them skip whole folders.

Summary

  • Sessionization groups events by user, orders them by time and cuts at an inactivity gap.
  • The secondary sort uses a composite key, a user-only partitioner, a full-key sort and a user-only grouping comparator.
  • The reducer streams through the sorted events and keeps only the current session in memory.
  • Streaming session windows do the same with event time, watermarks and window merging for late events.
  • Write results in a columnar, date-partitioned layout so analysts read only what they need.

Next: Practice Exercises with Solutions.