Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 63 ยท Counting and Classic Representations

TF-IDF

Learn how TF-IDF weights words by how informative they are. Compute it by hand, understand smoothing and normalization, then use TfidfVectorizer in scikit-learn.

In a bag of words, every count is treated as equally important. But the word the appears everywhere and tells you nothing about a document, while a word like sourdough might appear in two pages out of ten thousand and tells you a lot. Raw counts reward the wrong words: the most frequent ones are usually the least informative.

TF-IDF (term frequency times inverse document frequency) fixes this with one simple idea: a word is important in a document if it appears often there but rarely elsewhere. It has been the workhorse weighting for search engines and text classifiers since the 1970s, and it is still the first thing to try on a new text problem.

The two halves

Term frequency (TF) measures how much a word belongs to a document. The simplest version is the count of the word divided by the document length, so long and short documents are comparable.

Inverse document frequency (IDF) measures how rare a word is across the whole collection. If N is the number of documents and df is the number of documents that contain the word, the textbook formula is:

TEXT
idf(w) = log(N / df(w))
tfidf(w, d) = tf(w, d) * idf(w)

A word in every document has N / df = 1, so log(1) = 0: it gets zero weight. A word in only one of many documents gets a large IDF. The logarithm keeps the scale gentle, so a word that is ten times rarer is not ten times more valuable.

Two questions per word

Term frequency How often is it in this document?
Document frequency In how many documents does it appear?

High TF and low DF is the sweet spot.

1 / 2

TF-IDF by hand

Let us compute it for a four-document corpus without any library. We count document frequencies first, then score the words of the first document.

Python TF-IDF from the textbook formula

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

the appears in three of four documents, so its IDF is only 0.288, while bird appears once and gets 1.386. In document 1 the top word is mat: it appears once, in no other document. Notice that the, although it occurs twice in that document, ends up last: its high count is cancelled by a low IDF. This is the effect we wanted.

Smoothing and normalization

The textbook formula has two practical problems. First, a word with df = 0 would divide by zero, which happens when you score a query word that never occurred in training. Second, a word that is in every document gets exactly zero weight and disappears entirely, which can throw away useful signal.

Scikit-learn's TfidfVectorizer uses a smoothed IDF:

TEXT
idf(w) = ln((1 + N) / (1 + df(w))) + 1

Adding 1 inside the logarithm acts as if there was one extra document containing every word once, so the denominator is never zero. The trailing + 1 keeps universal words from collapsing to zero.

The vectorizer also normalizes each document vector to length 1 (the L2 norm). Without it, long documents would have large vectors and short documents small ones, so comparisons would favor length. After normalization, only the direction of a vector matters, which is exactly what cosine similarity measures. Instead of dividing by the document length, scikit-learn uses the raw count as TF and relies on this normalization to do the scaling.

The next example rebuilds the vectorizer's output by hand, to prove that nothing is hidden.

Python Rebuilding TfidfVectorizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two results are identical. In this smoothed version the has an IDF of 1.223 instead of nearly zero, and because it occurs twice in document 1 it is still the top-weighted word there (0.602). This surprises people: smoothing is gentler than the textbook formula, and very common words can stay near the top in short documents. The usual remedies are removing stop words, setting max_df (see feature hashing and large vocabularies), or damping counts with sublinear_tf, which we try next. The last line confirms the normalization: every row has length 1.

We passed token_pattern=r"\b\w+\b" so that one-letter words such as a are kept, to match our hand calculation. The default pattern drops them.

Options that change the weights

Four settings control most of the behavior. The comparison below shows the weight of python in two documents under each of them.

Python Comparing TfidfVectorizer options

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • sublinear_tf=True replaces the count tf with 1 + ln(tf). Saying python three times does not make a document three times as much about Python; it drops document 0's weight from 0.795 to 0.676.
  • use_idf=False keeps only normalized term frequency. With it, python is not discounted for being common (0.832 and 0.577).
  • norm=None shows the raw tf * idf products: document 0 is 3 * 1.223 = 3.669 and document 1 is 1 * 1.223 = 1.223. Unnormalized vectors are rarely what you want for similarity.
  • smooth_idf=False switches off the extra +1 document, giving the textbook formula plus 1.

Where TF-IDF is used

Its limits are those of the bag of words: no word order, no meaning, and no link between car and automobile.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the textbook formula, what IDF does a word get if it appears in every document?

    With df = N the ratio N / df is 1, and log(1) = 0, so the word gets zero weight everywhere.
  2. 2. Which word gets the highest TF-IDF weight in a document?

    TF rewards local frequency and IDF rewards rarity across the collection, so the combination points at words that characterize this document.
  3. 3. Why does TfidfVectorizer normalize each row to length 1 by default?

    Without normalization, longer documents have larger vectors. Dividing by the vector length leaves only the direction, so comparisons ignore document size.
  4. 4. What does smooth_idf=True protect against?

    Adding 1 to both the document count and the document frequency, as if one extra document contained every word, keeps the fraction well defined.
  5. 5. What does sublinear_tf=True do?

    It dampens repeated words, so saying a word three times counts for less than three times as much as saying it once.

Summary

  • TF-IDF weights a word by how often it appears in a document (TF) times how rare it is across the corpus (IDF).
  • The textbook IDF is log(N / df); words found in every document get zero.
  • Scikit-learn smooths the IDF to ln((1 + N) / (1 + df)) + 1 and normalizes each row to unit length.
  • TfidfVectorizer equals a counting step plus a weighting step; we reproduced its output exactly with NumPy.
  • sublinear_tf, use_idf, norm and smooth_idf change the weights; very common words can still rank high in short documents, so combine TF-IDF with max_df or stop words.
  • TF-IDF powers search, classification baselines, keyword extraction and similarity.

Next, handle huge vocabularies with Feature Hashing and Large Vocabularies.