Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 38 ยท Concepts and Data

Similarity Measures for Recommenders

Compare cosine, Pearson, adjusted cosine, Jaccard and Euclidean similarity on rating vectors in Python, see when each one works, and avoid common traps with missing values.

Almost every classic recommender rests on one question: how alike are two things? Two users who rate movies alike probably enjoy similar films. Two items that the same people like probably belong together. A similarity measure turns that vague idea into a number you can sort by.

This lesson builds five measures from scratch on a small rating matrix, shows where they disagree, and explains the main trap: what to do with the many empty cells. The later lessons on content-based and collaborative filtering all reuse these functions.

Vectors, not magic

A user is a row of the user-item matrix; an item is a column. Both are vectors, so any vector similarity works. The question is which one suits ratings.

Measure Idea Range
Cosine angle between two vectors -1 to 1 (0 to 1 for non-negative data)
Pearson cosine after subtracting each vector's mean -1 to 1
Adjusted cosine cosine after subtracting the user's mean (for item-item) -1 to 1
Jaccard overlap of two sets 0 to 1
Euclidean straight-line distance, turned into a similarity 0 to 1

Cosine similarity

Cosine compares the direction of two vectors and ignores their length:

cos(a, b) = (a ยท b) / (|a| |b|)

If you fill empty cells with 0, a user who rated nothing in common with you can still look "similar" in odd ways, because zeros are treated as real values. That is the central trap with ratings.

Python Cosine and Pearson by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that we only compare the items both users rated. Cosine is high for almost everyone because ratings are all positive numbers, so every pair of vectors points roughly the same way. That is why Pearson is popular.

Pearson correlation: remove the mean

A generous user who rates everything 4 or 5 and a harsh user who rates everything 1 to 3 may rank items identically. Subtracting each vector's mean removes that offset, so only the pattern remains. Pearson is exactly cosine on mean-centered vectors.

It has two weaknesses. With only one or two co-rated items it can be exactly 1 or -1 by accident, and it is undefined when a user has no variation (all the same rating). Both are handled by requiring a minimum overlap and by shrinkage.

Adjusted cosine for items

When you compare two items (columns), the right offset is each user's mean, not the item's mean. Subtract the user's average from every rating that user gave, then take the cosine of the two columns over users who rated both. This is adjusted cosine, and it usually beats plain cosine and Pearson for item-based filtering.

Jaccard for sets

With implicit data (watched or not) there is no rating to compare, only sets. Jaccard divides the size of the intersection by the size of the union:

J(A, B) = |A โˆฉ B| / |A โˆช B|

It treats all items equally, ignores popularity and is cheap, but it does not know that two obscure items in common say more than two blockbusters in common.

Euclidean distance as similarity

Euclidean distance is the straight-line gap between vectors. Smaller means more alike, so convert it: 1 / (1 + d). It cares about magnitude, which suits ratings on a shared scale, but it depends on how many items overlap (more co-rated items means a larger distance), so normalize by the count.

Python Five measures on one pair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the numbers carefully. User 1 is similar to user 0 on every measure. User 2 has the opposite taste (Pearson -1.00). User 3 agrees perfectly on the two items they share, so cosine, Pearson and Euclidean all say 1.00, yet Jaccard is only 0.33 because the two users overlap on few items overall. Two co-rated items are thin evidence, which is exactly why you set a minimum overlap.

Which similarity?

Cosine Direction, fast, ratings
Pearson Removes user generosity
Jaccard Sets, implicit data
Euclidean Magnitude gaps

For item-item on ratings, try adjusted cosine first.

1 / 2

Sparse matrices and scikit-learn

Looping over pairs is fine for teaching but slow in practice. sklearn.metrics.pairwise.cosine_similarity computes the whole similarity matrix at once and accepts sparse input. It treats missing cells as 0, so mean-center first (replace missing with 0 after centering, meaning "no deviation from average") to keep the meaning sensible.

Python Whole similarity matrix with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing in practice

  • Ratings, user-user: Pearson or centered cosine with a minimum overlap.
  • Ratings, item-item: adjusted cosine.
  • Implicit clicks or purchases: cosine on binary vectors or Jaccard, often with popularity damping.
  • Text or tags: cosine on TF-IDF vectors.
  • Always test: pick the measure that wins on your validation split, not the one that sounds best.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Pearson correlation is cosine similarity applied to what?

    Subtracting each vector's mean first removes generous or harsh rating habits, leaving only the pattern.
  2. 2. Why is filling missing ratings with 0 before cosine misleading?

    Missing means unknown, not disliked. Compare only co-rated items, or center first so missing becomes neutral.
  3. 3. Which measure fits purely implicit data (watched / not watched)?

    There are no rating values, only sets of items, so set-overlap measures are the natural choice.
  4. 4. What does adjusted cosine subtract before comparing two items?

    Item vectors are compared over users, so the per-user offset (generosity) is what must be removed.

Summary

  • Similarity turns users or items into comparable vectors; the choice of measure changes the neighbors you find.
  • Compare only co-rated entries and require a minimum overlap; never treat missing as 0 by accident.
  • Pearson and adjusted cosine remove rating-scale habits; Jaccard handles sets; Euclidean cares about magnitude.
  • Compute full matrices with scikit-learn after centering.

Next: TF-IDF Features for Items