Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 38 ยท Concepts and Data

The Ratings Dataset and Time-Based Splits

Build a MovieLens-style ratings dataset with timestamps and split it correctly: random, per-user, time-based and leave-one-out splits, and how leakage fools you.

How you split data decides whether your evaluation tells the truth. In ordinary machine learning a random split is often fine. In recommenders it can quietly lie to you, because the future should not be used to predict the past, and because each user must have enough training data to be modeled at all.

This lesson builds a MovieLens-style dataset (users, items, ratings, timestamps) with pandas, then shows four splitting strategies and when each is appropriate. Public datasets such as MovieLens use exactly this schema, so the code carries over.

The standard schema

A ratings dataset is a table of user, item, rating and timestamp. Real datasets add side tables: items (title, genres) and users (age, country).

Python A MovieLens-style dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Strategy 1: random split

Shuffle all interactions and hold out, say, 20%. Simple and fine for a first look, but it lets the model train on a user's later behavior and be tested on earlier behavior, which never happens in production.

Strategy 2: per-user random split

Hold out a fraction of each user's interactions, so every test user also has training data. This avoids test users with no history.

Strategy 3: time-based split

Choose a cutoff date: train on everything before, test on everything after. It mimics deployment ("we trained yesterday, what happens today?") and exposes drift. For per-user versions, hold out each user's most recent interactions.

Strategy 4: leave-one-out

Hold out only each user's last interaction as the test item. It is cheap to run, widely used for ranking evaluation, and easy to reason about ("can we put the next item in the top 10?").

Choosing a split

Random Fast, but leaks time
Per-user Everyone keeps history
Time-based Mimics production
Leave-one-out Predict the next item

Time-based or leave-last-out is the safest default.

1 / 2
Python Four splits side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last column shows the leakage clue: in the random and per-user splits some test interactions happened before the newest training ones, which is impossible in production. In the time-based and leave-one-out splits that column is False... almost: leave-one-out is per user, so one user's last event can still be older than another user's training events. For strict time realism use a global cutoff.

Make sure train covers the test

A model cannot score a user or item it has never seen. After splitting, check how many test rows involve unseen users or items. These are cold-start cases and are usually reported separately or removed for warm-start experiments.

Python Cold-start rows in a time split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Report how many rows you dropped; silently discarding them can make a model look better than it is.

Validation sets and leakage checks

  • Keep a validation split (also time-ordered) for tuning, and touch the test split once at the end.
  • Compute means, popularity counts, scalers and any learned item features from training rows only.
  • Never include the held-out interaction in a user's "history" input when predicting it.
  • For implicit data, held-out positives must be hidden from the model but the full set of seen items is still excluded from the candidate list when scoring (except the target).

The most common silent bug: computing item popularity or user averages on the whole dataset, then testing on part of it. The model has peeked at the answers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can a purely random split overstate performance?

    Random splits ignore time, so the model can learn from the future. Time-based splits respect the order of events.
  2. 2. What is leave-one-out evaluation?

    Each user's most recent interaction becomes the test target and the rest is training data.
  3. 3. How should test rows with unseen users or items be handled?

    Models cannot score unseen ids. Be transparent about the share, or evaluate cold start separately.
  4. 4. Which statistic must be computed from training data only?

    Statistics computed on all data leak test information into the model.

Summary

  • A ratings dataset has user, item, rating, timestamp, plus item and user side tables.
  • Random, per-user, time-based and leave-one-out splits answer different questions; time-based is the most realistic.
  • Check cold-start rows after splitting and report how you handled them.
  • Compute all statistics on training data; use a separate validation split for tuning.

Next: Similarity Measures for Recommenders