Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 48 ยท Projects

Project: Movie Recommendations with ALS

Build a movie recommender with Spark MLlib ALS: ratings matrix, latent factors, train and test split, RMSE, top-N recommendations and cold start, with a NumPy stand-in.

"Because you watched..." is one of the most visible uses of machine learning. A recommender system predicts how much a person would like something they have never seen, using the ratings of everybody else. Spark MLlib includes a well-tested algorithm for this called ALS, short for Alternating Least Squares, designed to run on very large ratings tables spread across a cluster.

In this project you learn how ALS works, how to train and evaluate it with MLlib, and how to turn the result into a top-N list per user. The MLlib code is read-only because there is no Spark runtime on this site. To let you run something real, the lesson also includes a compact NumPy implementation of the same algorithm on a small synthetic ratings table: it produces real factors, real error numbers and real recommendations, just on 30 users and 20 movies instead of millions. If you are new to MLlib's pipeline vocabulary, see MLlib Overview and Pipelines first.

The idea: hidden tastes

Imagine a table with one row per user and one column per movie, holding star ratings. It is mostly empty: nobody has watched everything, and a typical dataset is 99% blank. The blanks are not zeros. They mean "unknown", and predicting them is the whole task.

ALS assumes that tastes can be summarised by a few hidden numbers. Each movie gets a short vector of latent factors (perhaps "how much action", "how serious", "how old-fashioned", though the algorithm invents its own unnamed dimensions). Each user gets a vector of the same length describing how much they like each factor. The predicted rating is the dot product of the two vectors.

flowchart LR
    R["Ratings matrix R<br/>users x movies<br/>mostly unknown"] -.->|"approximated by"| P["User factors U<br/>users x k"]
    R -.-> Q["Movie factors V<br/>movies x k"]
    P --> X["prediction for user u, movie i<br/>= U[u] . V[i]"]
    Q --> X

The big sparse matrix is approximated by two small dense matrices of k columns each.

The number of factors, k, is called the rank. It is the main knob: a low rank is simple and may under-fit, a high rank can memorise the training ratings and generalise badly.

Why "alternating"?

Finding the best U and V together is hard, because the error depends on both and is not convex. But if you hold V fixed, finding the best vector for each user becomes an ordinary regularised least-squares problem with a direct solution. Likewise, if you hold U fixed, each movie is an easy problem. ALS simply alternates:

  1. Start with small random factors.
  2. Fix the movie factors; solve for every user's vector, using only that user's known ratings.
  3. Fix the user factors; solve for every movie's vector, using only that movie's known ratings.
  4. Repeat for a number of iterations (maxIter, commonly 10 to 20).

Every user's problem is independent of every other user's, so Spark can solve them in parallel on different machines. That is why ALS scales: each step sends the relevant factor vectors across the network (a shuffle) and solves millions of tiny systems at once. The regularisation term (regParam) keeps the factors small and protects users with only a few ratings from wild predictions.

The ratings table

First we build the data. The code below generates a synthetic ratings matrix with a hidden structure of two factors plus noise, hides about 55% of the entries and prints a corner of the matrix, with a dot for "unknown". It then holds out 20% of the known ratings as a test set, exactly as randomSplit([0.8, 0.2]) would, and computes a naive baseline: always predicting the average rating. Any model must beat it to be useful.

Python A sparse ratings matrix and a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

RMSE (root mean squared error) is the usual quality number for rating prediction: the typical size of the gap between predicted and actual stars, with big mistakes counted extra. Lower is better. The baseline RMSE shown is the number to beat.

Training ALS with MLlib

This is the real Spark version of the whole project, read-only:

Python
from pyspark.sql import SparkSession, functions as F
from pyspark.ml.recommendation import ALS
from pyspark.ml.evaluation import RegressionEvaluator

spark = SparkSession.builder.appName("recommender").getOrCreate()
spark.sparkContext.setCheckpointDir("/tmp/als-checkpoints")      # lets ALS truncate long lineage

ratings = spark.read.csv("/data/ml/ratings.csv", header=True, inferSchema=True)   # userId, movieId, rating
train, test = ratings.randomSplit([0.8, 0.2], seed=42)

als = ALS(
    userCol="userId", itemCol="movieId", ratingCol="rating",
    rank=10, maxIter=10, regParam=0.1,
    coldStartStrategy="drop",          # drop NaN predictions for unseen users or movies
    seed=42,
)
model = als.fit(train)

predictions = model.transform(test)     # adds a "prediction" column
rmse = RegressionEvaluator(metricName="rmse", labelCol="rating", predictionCol="prediction").evaluate(predictions)
print(f"test RMSE: {rmse:.3f}")

top5 = model.recommendForAllUsers(5)    # userId, recommendations: array<struct<movieId, rating>>
flat = (top5.select("userId", F.explode("recommendations").alias("rec"))
            .select("userId", "rec.movieId", F.col("rec.rating").alias("score")))

A few details worth knowing:

  • User and movie ids must be integers that fit into 32 bits. If your ids are strings, map them with StringIndexer first.
  • fit is the only expensive step. transform and recommendForAllUsers use the learned userFactors and itemFactors, which are ordinary DataFrames with columns id and features that you can save.
  • Implicit feedback (clicks, plays, purchases instead of star ratings) needs implicitPrefs=True and an alpha confidence parameter. There the "rating" is treated as a strength of evidence, not a score, and the evaluation uses ranking metrics (Spark 3 has RankingEvaluator) rather than RMSE.
  • Tune rank, regParam and maxIter with ParamGridBuilder and CrossValidator, or at least a loop over a small grid with a held-out set.
  • nonnegative=True constrains factors to be zero or positive, which sometimes makes them easier to interpret.
  • Set a checkpoint directory and checkpointInterval for long runs. ALS builds a long lineage over iterations, which can cause stack overflow errors without checkpoints (see Fault Tolerance and Lineage).

ALS from scratch

Now the stand-in. The function below is the algorithm described above, the same one MLlib distributes: for each user, solve a small regularised least-squares system using only that user's training ratings, then do the same for each movie, and repeat. It uses the weighted regularisation that MLlib uses (the penalty grows with the number of ratings a user or movie has). Watch the error fall on both the training and the test data.

Python Alternating Least Squares in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three things to look for. First, the training RMSE keeps falling with every iteration, but the test RMSE bottoms out around iteration 3 and then creeps back up: with only 222 training ratings, the model starts to memorise them. That is overfitting, and it is why the test number is the one that matters. On a real dataset with millions of ratings, the curve flattens instead and maxIter of 10 to 20 is usually enough.

Second, the grid search shows the rank and regularisation trade-off. A regParam of 1.0 squeezes every factor to nearly zero, so the model predicts the average and scores the baseline RMSE of 0.85: too much regularisation is as useless as none. Third, the grid is noisy, because the test set holds only 56 ratings, so one lucky setting can win by chance. This is why real projects use cross-validation (several train and test splits) instead of one split.

From scores to recommendations

A trained model predicts a score for every user and movie pair. To recommend, take one user, score all movies, remove the movies they have already rated and keep the top N. recommendForAllUsers(n) does this for everyone, and recommendForUserSubset does it for a chosen set. What is the situation for a brand new user? ALS has no vector for them, so MLlib returns NaN for their predictions (and coldStartStrategy="drop" removes those rows from evaluation). In practice you show popular items to new users until they have rated a few.

Python Top 3 recommendations for a user

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recommendations are movies the user has not rated, ranked by predicted stars. Notice that they follow the taste vector, not simple popularity: two users with different factors get different lists.

Evaluating beyond RMSE

RMSE measures rating accuracy, but a recommender is judged on the list it shows. Useful extra checks:

  • Precision at k: of the top k recommended items, how many does the user later rate highly or consume?
  • Coverage: what share of the catalogue ever gets recommended? A model that only suggests the ten most popular movies has a good RMSE and a poor product.
  • Time-based split: for real products, split by time (train on the past, test on the future) rather than randomly, because that mirrors how the model will be used.
  • A/B tests: offline metrics are a guide; the real test is whether users click and come back.

Real versus stand-in. The ALS lesson code with pyspark.ml.recommendation.ALS is real but did not run here. The NumPy version implements the same alternating least-squares updates on 278 known synthetic ratings, with a simple loop over users and movies on one machine. It also subtracts the average rating before fitting and adds it back afterwards, which keeps the tiny example stable; MLlib's explicit-feedback ALS has no such built-in bias term. MLlib distributes the same steps across blocks of users and movies, handles billions of ratings, supports implicit feedback and checkpoints state. The data, titles and numbers in the stand-in are invented for teaching.

A practical checklist

  1. Convert ids to integers and drop duplicate (user, item) pairs.
  2. Split into train and test (randomly for a first look, by time for the real thing).
  3. Fit ALS with a modest rank, 10 iterations and regParam near 0.1.
  4. Evaluate RMSE (or ranking metrics) on the test set and compare it with the baseline.
  5. Tune rank and regParam with cross-validation, then retrain on all data.
  6. Generate top-N lists with recommendForAllUsers, filter what users already saw and join item ids to titles.
  7. Save userFactors and itemFactors (or the model) and refresh them on a schedule.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a ratings matrix, what do the empty cells mean to ALS?

    Missing entries are unobserved. ALS fits its factors only to the known ratings and then predicts the unknown ones.
  2. 2. How is the predicted rating for a user and a movie computed after training?

    Each user and movie has a factor vector of length rank. Their dot product is the predicted rating.
  3. 3. Why does ALS alternate between solving for users and for movies?

    Holding one set of factors fixed turns each user (or movie) into an independent small regularised least-squares problem.
  4. 4. What does coldStartStrategy="drop" do?

    Users and movies absent from the training data have no factors, so their predictions are NaN. Dropping them keeps metrics like RMSE meaningful.
  5. 5. The training RMSE keeps falling but the test RMSE starts rising. What is happening?

    A growing gap between training and test error is overfitting. A smaller rank or stronger regularisation usually helps.

Summary

  • ALS approximates a sparse user-by-item ratings matrix with two small factor matrices; a rating prediction is the dot product of a user vector and an item vector.
  • It alternates between solving for user factors and item factors, each a small regularised least-squares problem, which is why it parallelises well on Spark.
  • In MLlib, ALS(rank, maxIter, regParam, coldStartStrategy) trains the model, transform predicts, RegressionEvaluator computes RMSE and recommendForAllUsers produces top-N lists.
  • Always compare against a baseline, evaluate on held-out data, and tune rank and regularisation to avoid overfitting.
  • New users and items have no factors (cold start), so combine ALS with popularity or content-based fallbacks.

Next: test everything you have learned in Practice: Exercises with Solutions.