Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 38 ยท Concepts and Data

The User-Item Matrix

Turn interaction logs into a user-item matrix with pandas, store it efficiently as a SciPy sparse matrix, and mean-center ratings, with runnable examples.

Almost every collaborative method starts from the same object: the user-item matrix. Rows are users, columns are items, and each cell holds the interaction between them, such as a rating or a click count. An empty cell means "no interaction observed". The recommendation problem then becomes "fill in the blanks sensibly".

This lesson shows how to build the matrix from a log table, why you should almost never store it densely, how to read it row-wise and column-wise, and how to center it.

From a long table to a matrix

Logs arrive in long format: one row per interaction. A matrix is wide format: one row per user. pivot converts between them.

Python Pivot a long table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

NaN marks unobserved cells. Do not silently replace them with 0 for explicit ratings: 0 would look like the lowest possible opinion. For implicit counts, 0 is a reasonable "no interaction" value.

If the same user rated the same item twice, pivot raises an error. Aggregate first with groupby(["user", "item"])["rating"].mean() or keep the latest.

Why sparse storage matters

A music service with 20 million users and 5 million tracks has a matrix of 100 trillion cells. The average user has touched a few hundred tracks, so well over 99.99% of the cells are empty. A dense array would need hundreds of terabytes; a sparse format stores only the observed entries.

SciPy's csr_matrix stores three arrays: the values, their column indices, and the pointer where each row starts. It is fast for row slicing, which is exactly what "get this user's history" needs.

Python Dense versus sparse memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In production, user and item ids are arbitrary strings or large integers. You map them to consecutive row and column indices first; keep these maps, since you need them to translate results back.

Python Mapping ids to matrix indices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the matrix

  • A row is a user's profile: everything they interacted with. Two users are similar when their rows look alike.
  • A column is an item's profile: everyone who interacted with it. Two items are similar when their columns look alike.
  • The row sums or counts reveal how active a user is; column counts reveal item popularity.

Rows and columns

Row = user Everything one person did
Column = item Everyone who touched it
Empty cell Unknown, to be predicted

User-based methods compare rows; item-based compare columns.

1 / 2

Mean-centering

People use scales differently. One user gives 4 as their average, another 2.5. Subtracting each user's mean turns ratings into deviations: positive means "above this user's usual", negative "below". Many algorithms (cosine on centered data is the Pearson correlation) work much better after centering.

Python Center ratings by user mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "generous" user's 4 for Heat is below their mean (4.67), which is mild disapproval; the "strict" user's 2 for Heat is about average for them. Raw numbers hide this.

When you evaluate a model, compute means (and every other statistic) from the training matrix only. Using test ratings in the means leaks information. See The Ratings Dataset and Time-Based Splits.

Common mistakes

  • Filling missing explicit ratings with 0 before computing similarity (0 is an opinion, not a blank).
  • Building a dense matrix of a large catalog and running out of memory.
  • Forgetting to keep the id-to-index maps.
  • Duplicated (user, item) rows crashing pivot.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an empty cell in a user-item rating matrix mean?

    Empty means unobserved. The recommender's job is to estimate what would fill it.
  2. 2. Why is a sparse format used for large matrices?

    Over 99% of cells are typically empty, so storing only non-empty entries is essential.
  3. 3. What does mean-centering a user's ratings achieve?

    After centering, positive values mean 'above this user's usual' regardless of whether the user is generous or strict.
  4. 4. Which pandas problem occurs if a user has two rows for the same item?

    pivot requires unique (index, column) pairs; aggregate duplicates with groupby first.

Summary

  • The user-item matrix has users as rows, items as columns and observed interactions as values.
  • Build it from long-format logs with pivot, or directly as a SciPy CSR matrix.
  • Use sparse storage and keep id-to-index maps.
  • Rows describe users, columns describe items; neighborhoods compare either.
  • Mean-center explicit ratings and compute statistics on training data only.

Next: Sparsity and the Long Tail