Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 38 ยท Evaluation Metrics

Rating Metrics: RMSE and MAE

Measure rating-prediction accuracy with MAE and RMSE: formulas, worked examples, per-user breakdowns, baselines, and why a low RMSE does not guarantee good recommendations.

If your model predicts star ratings, the natural question is "how far off are the predictions?". The two classic answers are MAE (mean absolute error) and RMSE (root mean squared error). They were the headline metrics of the Netflix Prize and are still used for explicit-rating datasets. They are also easy to misuse, so this lesson covers both the formulas and their limits.

The formulas

For n held-out ratings with true value r and prediction rฬ‚:

TEXT
MAE  = (1/n) ฮฃ |r โˆ’ rฬ‚|
RMSE = sqrt( (1/n) ฮฃ (r โˆ’ rฬ‚)ยฒ )

Both are in rating units, so an RMSE of 0.9 means "typically about 0.9 stars off", and lower is better. The difference is how they treat big mistakes. Squaring makes a 3-star miss count nine times as much as a 1-star miss, so RMSE punishes large errors and is always greater than or equal to MAE. If the two metrics are very different, a few big errors dominate.

Python MAE and RMSE by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "risky" model is exactly right on six of the seven ratings and wrong by three stars on one. Its MAE (0.43) is only a little worse than that of the "good" model (0.33), but its RMSE is more than three times higher (1.13 against 0.36), because the one big miss is squared. The last lines show clipping: a prediction of 5.8 for a 5-star item counted as an error until it was pulled back to the valid range. Which model is better depends on whether one large miss hurts more than many small ones, which is a product question, not a math question.

Rules for computing them honestly

  1. Evaluate on held-out ratings only. Train error says nothing about unseen items. Use the time-based split from the dataset lesson.
  2. Clip predictions to the valid range (1 to 5) before computing the error. A prediction of 5.8 for a 5-star item should not count as an error.
  3. Compare against baselines. An RMSE of 0.95 is only meaningful next to the global mean predictor, per-item mean, or a bias model. The metric has no absolute scale.
  4. Say how you averaged. RMSE over all ratings is dominated by heavy users, because they own most ratings. Averaging the RMSE per user gives every user the same voice.
  5. Report spread. Repeat across seeds or folds and give a mean with a standard deviation. Differences in the third decimal are usually noise.

Baselines and per-user breakdowns

The next example builds a synthetic ratings table with user and item effects, then evaluates three predictors on a held-out part: the global mean, the item mean and a user-plus-item bias model. It also shows RMSE pooled over ratings versus averaged over users.

Python Baselines and pooled vs per-user RMSE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each step down the table should lower the error, and the two columns do not agree exactly because heavy users contribute most of the test ratings to the pooled number. If your product serves mostly casual users, the per-user average is the more honest summary.

Why low RMSE does not mean good recommendations

RMSE measures how well you predict ratings of items the user did rate, but a recommender must choose which items to show. Three problems follow:

  • Ranking is not accuracy. A model can be within 0.1 stars on average and still order the top of the list poorly. Only the order at the top matters to the user.
  • The data is biased. Users rate what they chose to watch, which is usually items they expected to like. Predicting those well is easier than predicting items they never tried.
  • Ratings are rarely available. Most modern systems learn from clicks and plays, where "error in stars" has no meaning.

For these reasons, the next lessons move to metrics that look at the top-N list: precision and recall at K and ranking metrics. Keep RMSE for what it is good at: checking that a rating-prediction model is calibrated, and comparing models on a rating-prediction benchmark.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statement about MAE and RMSE is correct?

    Squaring makes big errors count much more, so RMSE is at least as large as MAE. Both are in rating units and lower is better.
  2. 2. A model has RMSE 0.9 on the test set. What else do you need to judge it?

    RMSE has no absolute scale. Compare with simple predictors on identical data.
  3. 3. Why should predictions be clipped to the rating range before computing error?

    A 5.8 prediction for a true 5 should not be counted as an error because the valid maximum is 5.
  4. 4. What is the main reason RMSE alone is a poor metric for top-N recommendation?

    Users see a short ranked list, so ranking metrics describe what they experience better.

Summary

  • MAE averages absolute errors; RMSE averages squared errors then takes the root and punishes big misses.
  • Compute them on held-out ratings, clip predictions and compare with baselines.
  • Pooled RMSE is dominated by active users; report per-user averages when that matters.
  • Rating error does not measure the quality of a top-N list; use ranking metrics for that.

Next: Precision and Recall at K