Skip to content
Tutorials Center
My learning

Topics

Lesson 59 of 64 ยท Projects

Project: Ratings and Recommendations

Build a small movie recommender in pandas: ratings tables, pivot to a user-item matrix, fair popularity ranking, correlation-based similar items and top-N lists.

"People who liked this also liked..." is one of the most recognizable features on the web, and the first version of it needs nothing more than the pandas you already know. In this project you will build a tiny movie recommender from three tables: users, movies and ratings. You will rank movies fairly by popularity, find movies that are rated alike by the same people, and produce a top-5 list for any user.

The project also teaches a habit that matters more than any single function: checking whether your clever method actually works. Near the end you will hide some real ratings, predict them, and compare your recommender with a much simpler baseline.

Along the way you will practice merges, groupby, pivot, correlation and missing data.

flowchart LR
    A["ratings<br/>(long: one row per rating)"] --> B["pivot"]
    B --> C["matrix<br/>(users x movies, many NaN)"]
    C --> D["subtract each user's mean"]
    D --> E["corr(): movie-to-movie similarity"]
    E --> F["similar movies"]
    E --> G["predicted ratings<br/>for unseen movies"]
    G --> H["top-N list per user"]

From a long table of ratings to a movie-by-movie similarity table and a personal top-N list.

Step 1: the three tables

Real rating data is long: one row per (user, movie, rating) event. It is also sparse: nobody has seen every movie. We generate a small version with a fixed seed: 24 invented movies in four genres, 80 users, and about 830 ratings. Each user has hidden tastes per genre, and some movies are far more talked about than others, so a few movies collect many ratings while others get only a handful. That is how real data behaves too.

Python Step 1: ratings, movies and users

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only about 43% of the possible user-movie pairs have a rating, and the number of votes per movie ranges from 2 to 77. That uneven spread drives everything that follows. (The last table of genre averages by age group is only there to show the joins at work: the generator does not use age, so any pattern in it is random noise.)

Step 2: fair popularity

The most obvious recommendation needs no personalization at all: show the best-rated movies. But sort by the mean rating and look at the top of the list:

Python Step 2: popularity, naive and fair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain mean puts "Last Convoy" (6 votes), "Iron Meridian" (4 votes) and "Salt and Sparrow" (2 votes) in the top five. A movie with two enthusiastic raters looks perfect, but two ratings tell you almost nothing. There are two standard fixes:

  • A minimum-votes filter: only rank movies with at least m votes. It is simple and easy to explain, but it hides new movies completely.
  • A weighted rating: blend each movie's mean with the overall mean, with a weight that grows with the number of votes. A movie with v votes gets v / (v + m) of its own mean and the rest from the global average, so a movie with few votes is pulled toward average and one with many votes keeps its own mean.

Here m is the median vote count, a reasonable default. In a real product you would choose it by thinking about how many votes you need before you trust a score.

Step 3: from long to a matrix, and similar movies

For personalized suggestions, we need to know which movies are rated alike. The natural structure is a matrix with one row per user and one column per movie, which pivot builds from the long table. Cells without a rating become NaN, and that is correct: "not rated" is not the same as zero.

Then comes the key idea. DataFrame.corr() computes the correlation between every pair of columns and, importantly, uses only the rows where both columns have values. So matrix.corr() gives a movie-by-movie similarity table: two movies score high when the same users tend to rate them high and low together.

Two refinements make the numbers trustworthy:

  • min_periods=8 returns NaN for a pair unless at least eight users rated both. Correlation computed from three people is noise.
  • Centering by user. Some people give everything a 4 or 5, others are stingy. Subtracting each user's own mean rating first (matrix.sub(matrix.mean(axis=1), axis=0)) removes that habit so that only relative preferences remain.
Python Step 3: the matrix and similar movies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With centering, all four neighbours of "Orbit Nine" are sci-fi movies and the neighbours of "The Quiet Orchard" are dramas. Without it, an action movie ("Night Fuse") sneaks into the list. Nothing told pandas about genres: the structure came purely from who rated what. The co_raters column is a built-in warning light. The matrix product seen.T @ seen counts, for every pair of movies, how many users rated both, so you can see when a high similarity rests on 11 people ("Echo Station") rather than 63 ("Deep Signal"). Finally, "Salt and Sparrow" shows the cold start problem: with only two ratings, there is nothing to compare, and the movie can never be recommended this way. That is when the popularity ranking from step 2 (or genre information) must fill in.

Step 4: top-N recommendations and an honest test

To predict how user u would rate an unseen movie m, look at the movies u has rated, keep those that are similar to m (positive similarity), and average the user's ratings of them, weighted by similarity. Writing it as "the user's own average plus a weighted average of how far above or below their average they rated the similar movies" keeps the prediction on the user's personal scale.

The recommendation for a user is then the unseen movies with the highest predicted rating. But a number on a screen is not evidence that the method works. So the code below also runs a hold-out test: it hides 15% of the real ratings, builds the similarity table from the remaining 85%, predicts the hidden ones, and measures the mean absolute error (MAE) against two baselines.

Python Step 4: recommend, then test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is honest rather than flashy. Predicting every rating as the overall average is off by about 0.95 stars on average; each movie's own average improves that to about 0.88, and the similarity method gets to about 0.80. It helps, but not dramatically, and with only 124 hidden ratings you should not read much into the third decimal. Real systems reach better numbers with far more data and much more machinery, yet they are evaluated with exactly this loop: hide, predict, compare with a simple baseline. If your fancy method cannot beat "each movie's average", do not ship it.

Taking it further

  • Genre filter. Merge movies back in and let the user ask for "top 5 comedies I have not seen".
  • Rating counts in the score. Multiply or blend the predicted rating with the weighted rating from step 2 so well-supported movies rank higher.
  • A user-based version. Transpose the matrix and find users who rate like you instead of movies that are rated alike.
  • Speed. The prediction loop is simple but slow for large catalogs. Replace it with matrix operations, as described in Writing Fast pandas Code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the plain mean rating produce a misleading top list?

    An average of two ratings is unreliable. A weighted rating or a minimum-votes filter keeps movies with little evidence from topping the list.
  2. 2. What does matrix.corr(min_periods=8) do for a pair of movies rated by only five of the same users?

    min_periods is the smallest number of rows in which both columns must have a value. Below it, the result is NaN, which protects you from similarities built on a handful of people.
  3. 3. Why subtract each user's mean rating before computing movie-to-movie correlations?

    A generous rater gives 4s and 5s to everything. Centering by the row mean makes ratings comparable across users, so the correlation reflects taste rather than how generous the rater is.
  4. 4. Why hide part of the ratings and compare the method with 'each movie's average'?

    A method is only useful if it beats a simple alternative on data it has not seen. A hold-out test gives that evidence; plausible-looking recommendations do not.

Summary

  • Rating data is long and sparse: pivot turns it into a user-by-movie matrix where NaN means "not rated", not zero.
  • Rank by popularity fairly: a minimum-votes filter or a weighted rating keeps movies with few votes from topping the list.
  • DataFrame.corr(min_periods=...) compares columns using only the rows where both have values, which gives item-to-item similarity; center each row first to remove rater habits.
  • Show how many people support a similarity (co-raters), and expect cold-start movies to need a fallback such as popularity.
  • Always evaluate with a hold-out set against a simple baseline before trusting a recommender.

Ready to test yourself on everything in this course? Continue with Practice Exercises with Solutions.