Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 38 ยท Content-based Filtering

TF-IDF Features for Items

Turn item descriptions, tags and genres into TF-IDF vectors with scikit-learn, see why IDF down-weights common words, and find similar items with cosine similarity.

Content-based filtering recommends items that look like the ones a user already liked. "Look like" needs numbers, and for items that have text (titles, descriptions, tags, genres) the standard way to get them is TF-IDF. It turns each item into a vector where distinctive words weigh more and common words weigh less.

This lesson explains the idea, builds a tiny TF-IDF by hand to remove the mystery, then uses TfidfVectorizer and cosine similarity to find similar items.

Why raw word counts are not enough

If you count words, every description is dominated by "the", "a" and "movie". Two unrelated items then look alike simply because they share filler. We want words that are frequent in this item but rare overall.

  • TF (term frequency): how often a word appears in this item's text.
  • IDF (inverse document frequency): log(N / df) where N is the number of items and df the number of items containing the word. Rare words get a high IDF; a word in every item gets zero.
  • TF-IDF: TF ร— IDF.

scikit-learn uses a smoothed variant, idf = ln((1 + N) / (1 + df)) + 1, and then L2-normalizes each vector so lengths do not matter.

Python TF-IDF by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

"space" appears twice in item A and nowhere else, so it dominates A. "robot" and "adventure" appear in two items, so they count for less. A word in all three items would score 0.

TF-IDF with scikit-learn

TfidfVectorizer does tokenizing, stop-word removal, IDF and normalization, and returns a sparse matrix with one row per item.

Python Find similar items

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing text, tags and categories

A lesson on features is mostly a lesson on what text to feed in:

  • Descriptions and plots: rich, but noisy. Good for TF-IDF.
  • Tags and genres: clean and short. Use a plain CountVectorizer(binary=True) or the same TF-IDF.
  • Titles: very short; add them to the description rather than using alone.
  • Creators and brands: make each one a single token (christopher_nolan) so it is not split.

You can combine fields into one string, or build one vector per field and weight them before concatenating (see feature weighting).

Useful parameters

Parameter Effect
stop_words="english" Removes very common words
ngram_range=(1, 2) Adds two-word phrases such as "space station"
min_df=2 Ignores words found in fewer than 2 items (typos, one-offs)
max_df=0.8 Ignores words found in more than 80% of items
sublinear_tf=True Uses 1 + log(tf) so repeated words do not run away
Python Effect of n-grams and min_df

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fit the vectorizer on the training catalog only, then transform new items with it. New words the vectorizer has never seen are ignored, so refit when the catalog changes a lot.

TF-IDF in three steps

Count terms TF: words in this item
Weigh rarity IDF: rare words matter more
Normalize Unit length per item

The result is a sparse matrix: one row per item.

1 / 2

Limits to remember

TF-IDF knows only words. It misses synonyms ("film" vs "movie"), word order and meaning, and it cannot see images or audio. Embeddings fix some of this, at higher cost. For many catalogs, TF-IDF on tags and descriptions is a strong, cheap, explainable baseline.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does IDF do?

    Inverse document frequency rewards rare words and gives words found in every item a weight near zero.
  2. 2. A word appears in every item of the catalog. What is its (unsmoothed) IDF?

    log(N / N) = log(1) = 0, so the word carries no information for telling items apart.
  3. 3. Which setting ignores words that appear in only one item (often typos)?

    min_df=2 keeps only terms present in at least two items.
  4. 4. Why fit the vectorizer on training items only?

    IDF depends on the whole corpus. Computing it with test items would let the model peek.

Summary

  • TF-IDF weighs a word by how often it appears in an item and how rare it is across the catalog.
  • TfidfVectorizer produces L2-normalized sparse vectors; cosine similarity then finds similar items.
  • Choose fields carefully, combine tags with descriptions, and tune ngram_range, min_df and max_df.
  • TF-IDF is cheap and explainable but blind to meaning beyond the words.

Next: A Content-based Recommender