Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 38 ยท Matrix Factorization

Latent Factors and SVD for Recommendations

Understand why a ratings matrix has low rank, how latent factors describe users and items, and how truncated SVD builds a first matrix-factorization recommender.

Neighborhood methods compare users or items directly. Matrix factorization takes a different view: it assumes that a few hidden tastes explain most of the ratings. A movie might be "a bit of comedy, a lot of romance, not very violent". A user might love romance, tolerate comedy and avoid violence. If both are described by short vectors of numbers, a rating is simply how well the two vectors line up.

These hidden dimensions are called latent factors. Nobody labels them; the algorithm discovers them from the data. In this lesson you will see the idea on a tiny matrix, learn what the singular value decomposition (SVD) does, and use it to predict held-out ratings. The next lessons fix the weak points of plain SVD with SGD and ALS.

The ratings matrix and its low rank

Put users in rows and items in columns. Most cells are empty because nobody rates everything. The recommendation task is to fill the empty cells.

Factorization writes the matrix R (users x items) as a product of two thin matrices:

TEXT
R  โ‰ˆ  P  x  Qแต€
(users x items)  (users x k)  (k x items)

Row u of P is the user vector, row i of Q is the item vector, and the predicted rating is the dot product P[u] ยท Q[i]. The number k (often 10 to 200 in real systems) is the number of latent factors. If k is much smaller than the number of users and items, the model is forced to find shared patterns instead of memorizing each cell.

flowchart LR
    R["Ratings R<br/>users x items"] --> S["Factorize"]
    S --> P["User factors P<br/>users x k"]
    S --> Q["Item factors Q<br/>items x k"]
    P --> D["Dot product<br/>P[u] ยท Q[i]"]
    Q --> D
    D --> E["Predicted rating"]

Every prediction is one dot product between a user row and an item row.

SVD on a tiny, complete matrix

When every cell is known, the best rank-k approximation in the least-squares sense comes from the singular value decomposition: R = U ยท diag(s) ยท Vแต€. The singular values in s are sorted from large to small. Keeping only the first k columns of U, the first k values of s and the first k rows of Vแต€ gives the closest rank-k matrix. That is the Eckart-Young theorem.

Below, five users rate four items. Users 0, 1 and 4 like the first two items; users 2 and 3 like the last two. That is two "taste groups", so two factors should explain almost everything.

Python SVD of a small ratings matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the singular values: the first two are large and the last two are small, so a rank-2 matrix is almost the whole story. In the factors, the first column is the "overall rating level" (all negative here, a sign convention), and the second column separates the groups: users 0, 1 and 4 get a positive value, users 2 and 3 a negative one. The signs of an SVD factor are arbitrary (flipping a column in both matrices changes nothing), so judge the factors by their pattern, not by whether a number is positive.

Splitting sqrt(s) between the user and item side is a convention. You could also keep s on one side only. The product is the same.

The catch: real matrices have holes

np.linalg.svd needs a full matrix. A real ratings matrix is 95% to 99.99% empty. A tempting shortcut is to put 0 in the empty cells, but that tells the model "the user hated this item", which is wrong. The usual repairs for a first attempt are:

  1. Subtract the mean. Compute the global mean (or per-user or per-item means) from the training ratings and work with deviations from it.
  2. Fill the holes with the neutral value. After mean-centering, an empty cell becomes 0, which now means "average", not "terrible".
  3. Truncate. Keep only the top k singular values, then add the mean back.

This is imputation SVD. It is simple and fast, and it shows the shape of the method. Its weakness is that the guessed values in the holes still influence the result. Methods in the next lessons fit only the cells that were really observed.

Python Truncated SVD on a sparse synthetic matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice three things. The model beats the global-mean baseline at small k, so the latent structure is real. The error is lowest around the true number of factors and then gets worse as k grows, because extra factors start fitting noise and the filled-in zeros. And the best k is a hyperparameter you choose on validation data, never on the training error: training error always keeps falling as k rises.

Why not just use SVD on the sparse matrix?

Two good reasons keep plain SVD from being the production answer:

  • It fits the wrong thing. The loss counts every cell, including the imputed ones. Most of those cells are not real ratings.
  • It does not scale as written. A dense SVD of a 10-million-user matrix is impossible. Sparse solvers such as scipy.sparse.linalg.svds compute only the top k factors, but they still treat missing entries as zeros.

The famous solution from the Netflix Prize era is often called "SVD" in recommender articles, yet it is not an SVD at all. It learns P and Q by minimizing the error over observed ratings only, adds regularization and bias terms, and uses gradient descent. You will build it in SGD and Biased Matrix Factorization.

Matrix factorization in one view

Latent factors Hidden tastes learned from data.
User vector How much a person likes each factor.
Item vector How much an item has each factor.

Prediction = dot product of the two vectors.

1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In matrix factorization, how is a rating predicted for user u and item i?

    Each user and item gets a short latent vector. Their dot product is the predicted rating.
  2. 2. What does the Eckart-Young theorem tell us about truncated SVD?

    Truncated SVD is the closest rank-k matrix to a fully known matrix. That says nothing about test error or missing cells.
  3. 3. Why is filling missing ratings with 0 before an SVD a problem?

    Subtract the mean first so that a filled hole means 'average', and prefer methods that fit only observed ratings.
  4. 4. As k keeps growing, what usually happens to training and test error?

    Extra factors fit noise. Pick k on validation data.

Summary

  • Matrix factorization explains ratings with a few latent factors: R โ‰ˆ P ยท Qแต€.
  • A prediction is a dot product between a user vector and an item vector.
  • Truncated SVD gives the best low-rank fit of a complete matrix; for sparse data, subtract the mean and fill holes with "average" for a first baseline.
  • The number of factors k is a hyperparameter chosen on held-out data.
  • Plain SVD fits imputed cells too. SGD and ALS fit only observed ratings.

Next: SGD and Biased Matrix Factorization