Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 62 ยท Projects

Project: A Matrix-Factorization Recommender

Build a movie recommender with linear algebra: a sparse ratings matrix, low-rank user and item factors, alternating least squares, held-out error and a rank sweep.

"People like you also liked..." is one of the most visible uses of linear algebra. A streaming service has millions of users and thousands of titles, but each person rates only a handful. The ratings matrix is mostly empty, and the whole job is to fill in the blanks sensibly.

The trick is to assume that taste has a small number of hidden dimensions: maybe how much you like action versus romance, or how much you like slow films. If every user is described by a few numbers and every item by a few numbers, a rating is just a dot product of the two. Stacked together, that says the ratings matrix is approximately the product of two thin matrices, a low-rank factorization. In this project you will build it with nothing but NumPy: you will generate a ratings matrix, hide some entries, fit the two factor matrices with alternating least squares (ALS), and judge them on ratings the model never saw. It builds directly on least squares and low-rank approximation.

Taste has a few dimensions

User vector A few numbers describe what a user likes.
Item vector A few numbers describe what a film is like.
Dot product Their dot product predicts the rating.
1 / 2

Step 1: a ratings matrix with hidden entries

Real data would come from a database. We generate it, which lets us cheat in a useful way: we know every true rating, so we can measure how good the guesses for hidden entries are. Each of 30 users and 20 items gets two secret numbers, and the rating is 3 + taste . traits plus a little noise, clipped to the 1-5 range.

Then we hide 50 percent of the matrix (nobody saw those films) and set aside 20 percent of the visible ratings as a test set. The model must never train on the test set. That is how we find out whether it learned real patterns or just memorized.

Python Generate ratings and split them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

train, test and seen are boolean matrices of the same shape as the ratings, a convenient way to say "this entry counts". The last lines give us a number to beat: always guessing the average rating (about 3.02) is off by roughly 1.0 stars (RMSE, the root of the mean squared error). A recommender that cannot beat that is useless.

Step 2: why alternating?

We want matrices U (users by k) and V (items by k) so that U @ V.T matches the known ratings. The squared error over the known entries is

TEXT
sum over known (i, j) of (R[i, j] - U[i] . V[j])^2  +  lambda * (|U|^2 + |V|^2)

The lambda term is regularization: it keeps the factors small, which fights overfitting. Optimizing U and V together is a hard, non-convex problem. But freeze V and the error becomes an ordinary least squares problem in each row of U, and every user is an independent little problem. For one user, take the rows of V for the items they rated, call that V_o, and solve the (ridge) normal equations:

TEXT
(V_o.T @ V_o + lambda * I) u = V_o.T @ ratings_of_that_user

Then freeze U and do the same for every item. Alternate, and the error never goes up.

flowchart LR
    A["Random small U, V"] --> B["Fix V:<br/>solve each user row"]
    B --> C["Fix U:<br/>solve each item row"]
    C --> D{"Error still<br/>falling?"}
    D -- "yes" --> B
    D -- "no" --> E["Predict with U @ V.T"]

ALS alternates between two ordinary least-squares problems until the error settles.

Step 3: write ALS

We subtract the average rating first so the factors only need to explain deviations from "normal", and add it back when predicting. Watch the shapes: Vo is (items this user rated, k) and the right-hand side is a vector of that user's ratings.

Python Alternating least squares from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first sweeps do almost all the work: after five of them the test error is already about 0.35, down from 1.0 for the always-guess-the-average baseline. By sweep 15 nothing moves any more. The test error is not far above the noise we added (0.2), which tells us the two-dimensional model captured nearly all the structure that exists to be found.

Step 4: recommend something

To recommend films to user 0, predict a rating for every film they have not rated and list the best ones. Because this is a simulation we can also look up the true rating, which real systems cannot.

Python Top recommendations for one user

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three recommended films are predicted at 4.86, 4.70 and 4.37, and their true ratings are all 5.0 (the clipping at 5 hides the fact that this user would like them even more). The model has never seen those ratings. It inferred them from other users with a similar taste vector, which is the whole point.

Step 5: how many factors? Overfitting in action

Choosing the rank k is the central modelling decision. Too small and the model cannot represent the structure (underfitting); too large and it has enough freedom to memorize noise in the training ratings (overfitting). The honest way to choose is the test error. The regularization strength lambda is the second dial.

Python Sweep the rank and plot train versus test error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The curves tell the classic story. Training error keeps falling as k grows (from 0.64 at rank 1 to 0.03 at rank 8), but test error is lowest at rank 2, which matches how the data was generated, and then climbs again (0.37, 0.40, 0.52, 0.53). More factors let the model chase noise in the few training ratings it has. The second table shows regularization doing a similar job: with rank 8 fixed, raising lambda from 0.01 to 1 cuts the test error from 0.88 to 0.44, while lambda = 5 is so strict that the model underfits (about 0.75 on both sets).

Always measure on ratings the model did not train on. A training error of 0.03 looks wonderful and means almost nothing. In a real system you also need a third split for the final check, because choosing k and lambda by test error slowly leaks the test set into your decisions.

Where this connects

An SVD gives the best rank-k approximation of a complete matrix (the Eckart-Young idea). A ratings matrix is incomplete, so we cannot run an SVD directly; ALS finds a factorization that is only required to fit the entries we know. Production systems add per-user and per-item bias terms, use implicit feedback (clicks, watch time) and run ALS in parallel across thousands of machines, which is easy because each user row is an independent solve.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. If U has shape (30, 2) and V has shape (20, 2), what is the shape of U @ V.T?

    U @ V.T is (30, 2) times (2, 20), so the result has 30 rows (users) and 20 columns (items): one predicted rating for every user-item pair.
  2. 2. Why does ALS alternate between solving for U and for V?

    The joint problem is hard, but fixing V makes the error a quadratic function of U with a closed-form solution, and vice versa. Each step can only lower the error.
  3. 3. A model has training RMSE 0.03 and test RMSE 0.53. What does that suggest?

    A large gap between training and test error means the model learned patterns that exist only in the training data. Lower the rank or raise lambda.
  4. 4. What does the lambda term in the ALS objective do?

    Regularization adds lambda times the identity to the normal equations. That shrinks the factors toward zero and makes the solve stable even when a user has rated very few items.

Summary

  • A ratings matrix is mostly empty; a low-rank factorization R ~ U @ V.T gives every user and item a short vector and predicts the blanks with dot products.
  • ALS alternates two least-squares solves: fix V and solve each user row with (Vo.T @ Vo + lambda * I) u = Vo.T @ r, then swap roles.
  • Boolean masks pick the known entries, so only real ratings drive the fit.
  • Judge the model on held-out ratings; training error alone hides overfitting.
  • The rank k and the regularization lambda control the bias-variance trade-off.

Next project: A Tiny 3D Wireframe Renderer.