Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 38 ยท Matrix Factorization

SGD and Biased Matrix Factorization

Build the classic Netflix-Prize style recommender from scratch in NumPy: fit user and item factors with SGD, add bias terms, tune regularization and rank unseen items.

Truncated SVD works on a complete matrix, but a ratings matrix is mostly empty. The method that made matrix factorization famous learns the factors by looking only at the ratings that exist. It minimizes the squared error over observed cells, adds a penalty so that the factors do not blow up, and walks downhill with stochastic gradient descent (SGD). People often call this "FunkSVD" after the blogger who popularized it during the Netflix Prize, although it contains no SVD.

In this lesson you will write the whole algorithm in about twenty lines of NumPy, add the bias terms that make it much stronger, and tune it.

The model and its loss

The prediction for user u and item i is:

TEXT
rฬ‚(u,i) = ฮผ + b_u + b_i + p_u ยท q_i
  • ฮผ is the global mean rating.
  • b_u is how generous user u is compared with the average. Some people give 5 stars to everything.
  • b_i is how well liked item i is overall. Some items are simply popular.
  • p_u ยท q_i is the personal part: the interaction between this user's taste and this item's character.

The model minimizes this over the observed ratings only:

TEXT
sum over observed (u,i) of (r(u,i) - rฬ‚(u,i))ยฒ  +  ฮป ( b_uยฒ + b_iยฒ + |p_u|ยฒ + |q_i|ยฒ )

The second term is L2 regularization. It pulls every parameter toward zero, which matters a lot because a user with three ratings cannot support a reliable 20-dimensional vector.

For one training rating, with error e = r - rฬ‚, SGD nudges the parameters by learning rate lr:

TEXT
b_u += lr * (e - ฮป b_u)          b_i += lr * (e - ฮป b_i)
p_u += lr * (e q_i - ฮป p_u)      q_i += lr * (e p_u - ฮป q_i)

Each update moves the prediction toward the true rating, and the ฮป term shrinks the parameters a little. One pass over all ratings is an epoch.

Implementing it

The code below creates a synthetic ratings matrix (with real user and item biases plus three hidden factors), splits it, and fits the model. The factors start small and random: if they all started at zero, every gradient would also be zero and nothing would learn.

Python Biased vs unbiased matrix factorization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the two traces. The biased model has a lower test error at every checkpoint (the unbiased one is still improving at the end and slowly catches up), because the bias terms soak up the "this user rates high" and "this item is great" effects cheaply, leaving the dot product free to capture real taste. This is why almost every production factorization model keeps ฮผ, b_u and b_i.

Watch the gap between train and test RMSE too. Training error usually keeps falling. If test error turns upward while training error falls, the model is overfitting and you need stronger regularization, fewer factors or fewer epochs.

Tuning: regularization, factors and learning rate

Three knobs decide most of the result:

  • reg (ฮป). Too small and the model memorizes; too large and every prediction collapses to the mean. Typical values are 0.01 to 0.1 for this kind of loop (their scale depends on how the loss is written).
  • k. More factors can capture finer taste but need more data and more regularization.
  • lr. If the error explodes or becomes nan, the learning rate is too high. If it falls very slowly, raise it or run more epochs.

A grid search must use a validation split, not the final test split. The next block reuses the data and the fit function above and compares a few settings. To stay self-contained it repeats the setup in compact form.

Python A small hyperparameter sweep

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table the way you would in a real project: pick the cell with the lowest validation error, then retrain on train plus validation and report one number on the untouched test split. In this run the middle column wins: reg=0.01 lets the 15-factor model overfit, and reg=0.5 shrinks everything so hard that no setting learns much. Single SGD runs are noisy, so in real projects average a few seeds before trusting a small difference.

Recommending items

Predicted ratings become recommendations by ranking items the user has not rated yet and keeping the top N. Never recommend something the user already rated or consumed.

Python Top-5 unseen items for one user

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For ranking, you do not need to clip to the 1 to 5 range: clipping only matters when you report an error in rating units.

Practical notes

  • Shuffle each epoch. Going through ratings in a fixed order makes SGD zigzag.
  • Learning-rate decay (for example multiplying lr by 0.95 each epoch) helps the end of training settle.
  • Early stopping on validation error is a cheap form of regularization.
  • Python loops are slow. This loop is for learning. Real libraries run the same idea in compiled code, or use mini-batches on a GPU, or switch to alternating least squares, which parallelizes better.
  • Side information. The same update rule extends to extra terms, for example a bias for the day of the week or implicit signals (SVD++ adds a term for the items a user interacted with).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In rฬ‚(u,i) = ฮผ + b_u + b_i + p_u ยท q_i, what does b_u capture?

    b_u is the user's average offset from the global mean. Personal taste is in the dot product.
  2. 2. Why does SGD matrix factorization start with small random factors instead of zeros?

    The factor gradients are proportional to the other side's factors, so an all-zero start never moves.
  3. 3. Training RMSE keeps falling but test RMSE rises. What should you try first?

    The model is overfitting. Regularization, fewer factors or early stopping reduce that.
  4. 4. Which data should you use to choose k and the regularization strength?

    Tuning on the test split leaks information and makes the final score too optimistic.
  5. 5. When ranking items for a user, which items must be excluded from the candidates?

    Recommending something the user already has is wasted space in the list.

Summary

  • Biased matrix factorization predicts ฮผ + b_u + b_i + p_u ยท q_i and learns all terms from observed ratings only.
  • SGD updates one rating at a time; L2 regularization (reg) keeps the factors small.
  • Bias terms capture generous users and popular items so the factors can focus on taste.
  • Tune k, reg and lr on a validation split; watch the train/test gap.
  • Rank unseen items by predicted score to produce top-N recommendations.

Next: ALS and Implicit Feedback