Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Embeddings and Similarity

TF-IDF and LSA: Embeddings Without a Neural Network

Build sparse TF-IDF vectors and dense LSA embeddings with scikit-learn, see how dense vectors bridge vocabulary gaps, and learn why this still matters for RAG.

Before neural embedding models, search engines already turned text into vectors. Two classic methods are still worth knowing: TF-IDF, which makes a long sparse vector (one slot per word in the vocabulary), and LSA (latent semantic analysis), which compresses that vector into a short dense one. They run on a laptop in milliseconds, they need no training data except your own documents, and they teach the difference between matching words and matching meaning, the difference that vector databases are built around.

Everything in this lesson runs for real in the page with scikit-learn.

TF-IDF: weighting words by how special they are

A bag of words counts how often each word occurs in a document. Raw counts over-reward words like "the". TF-IDF multiplies two things:

  • TF (term frequency): how often the word appears in this document.
  • IDF (inverse document frequency): log(N / df), large for words found in few documents and small for words found everywhere.

A word that is frequent here but rare elsewhere gets a high weight; that is usually what makes a document distinctive. After weighting, each document is one sparse vector, and the cosine between two of them measures how many weighted words they share.

Python Sparse TF-IDF vectors with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rows of a TfidfVectorizer result are normalised to length 1, so the dot product equals the cosine (metrics). Most of the matrix is zeros, which is why sparse formats exist. The word "the" has a low IDF and "windowsill" a high one, so the query matches the documents that share the rare words.

The weakness: no word, no match

Sparse vectors match only the exact words (after tokenising). A question about a "kitten" does not retrieve a document about a "cat"; "failing" does not match "failed" unless you stem; "bug" does not match "defect". This is the vocabulary mismatch problem, and it is the main reason dense embeddings exist.

Python Vocabulary mismatch in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first query shares no words with any document, so every score is exactly 0 and the ranking is meaningless. A person sees that the cat document is the answer; TF-IDF cannot.

LSA: finding the hidden topics

Latent semantic analysis applies SVD to the TF-IDF matrix and keeps only the strongest k directions. Each direction is a latent topic: a weighted group of words that tend to occur together. A document becomes k numbers, "how much of each topic". Words that appear in the same kinds of documents (cat, kitten, yarn) end up pulling in the same topics, so they become interchangeable.

flowchart LR
    T["Documents"] --> V["TF-IDF matrix (sparse, huge)"]
    V --> S["SVD / TruncatedSVD, keep k"]
    S --> D["Dense vectors (k numbers each)"]
    Q["Query"] --> VQ["TF-IDF then same projection"] --> D

From sparse words to dense topics. Queries go through the same two steps.

Python LSA bridges the vocabulary gap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query "sofa windowsill" shares no word with the kitten-and-yarn document, yet that document scores 0.58 because it belongs to the same cat topic; plain TF-IDF would give exactly 0. The price is that the vectors are small, so details are lost, and k is a knob you must tune: too small merges different topics, too large lets noise back in.

Choosing k

Rules of thumb for LSA on a real collection: 100 to 300 components for tens of thousands of documents; fewer for small corpora. Check by measuring retrieval on test questions (metrics) instead of trusting a rule.

Sparse versus dense, side by side

Sparse (TF-IDF / BM25) Dense (LSA / neural)
Length Vocabulary size (tens of thousands), mostly zeros 100 to 3,000, all non-zero
Matches Exact words Meaning and related words
Rare names, codes, IDs Excellent Often poor
Synonyms, paraphrases Poor Good
Explainable Yes: you can list the matching words No
Needs training No Yes (a model)

Neither wins everywhere. Searching for an error code like ERR_4012 is a job for exact words; searching "my build keeps breaking" is a job for meaning. Real systems use both together (BM25, hybrid search).

Why this matters for RAG

  • A TF-IDF or BM25 index is a strong, cheap baseline. If your fancy dense retriever cannot beat it on your questions, something is wrong.
  • LSA shows in a dozen lines what a neural model does at scale: compress words into topics so that nearby points share meaning.
  • Sparse vectors have their own indexes (inverted indexes), and modern vector databases store sparse and dense vectors side by side.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the IDF part of TF-IDF do?

    IDF is the log of N over document frequency. Rare words get high weights, common words low weights.
  2. 2. Why does a TF-IDF search for 'feline' not find a document about a 'cat'?

    This is the vocabulary mismatch problem: without a shared term, the dot product is zero.
  3. 3. What does LSA do to the TF-IDF matrix?

    Truncated SVD compresses the matrix into k dense dimensions that capture co-occurring word groups.
  4. 4. Which query is the best fit for sparse (keyword) search instead of dense search?

    Exact identifiers such as error codes are matched literally by sparse methods; dense vectors often blur them.

Summary

  • TF-IDF gives sparse vectors that weight rare words highly; cosine over them measures shared weighted words.
  • Sparse methods cannot bridge vocabulary gaps (kitten vs cat).
  • LSA compresses TF-IDF with truncated SVD into dense topic vectors that do bridge gaps, with a tunable size k.
  • Sparse and dense each win on different queries, which is why production systems often combine them.

Next: Embedding Geometry and the Curse of Dimensionality