Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 53 ยท Tokens and Embeddings

Cosine Similarity and Semantic Search

Learn dot product, cosine similarity and Euclidean distance, then build a tiny semantic search engine in NumPy and see why approximate indexes like HNSW exist.

Once text is a vector (see Embeddings: Meaning as Numbers), "find documents about the same thing as this question" becomes a geometry problem: find the vectors closest to the query vector. This is semantic search, and it powers retrieval for chatbots, recommendations, duplicate detection and much of RAG.

This lesson covers the three common ways to measure closeness, builds a tiny search engine, and explains why big systems use approximate nearest-neighbor indexes instead of checking every vector.

Three ways to compare vectors

Take two vectors a and b.

  • Euclidean distance is the straight-line distance between the two points. Smaller means more similar.
  • Dot product multiplies matching coordinates and adds them up: a . b. Bigger means more similar, but it also grows with the vectors' lengths.
  • Cosine similarity is the dot product after dividing by both lengths. It measures the angle between the vectors and ignores their length: 1 means the same direction, 0 means unrelated (at right angles), -1 means opposite.
TEXT
cosine(a, b) = (a . b) / (|a| * |b|)

Length often carries no meaning for text. A long document and a short one on the same topic can have vectors pointing the same way but with different magnitudes, so the angle is the fairer measure. That is why cosine similarity is the most common default.

Python Dot product, cosine and Euclidean distance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Vectors a and b point the same way, so their cosine is exactly 1 even though their Euclidean distance is not zero. That is the difference in a nutshell. The last two lines show a handy trick: if you normalize every vector to length 1 once, then the dot product equals the cosine, and search becomes a single fast matrix multiplication.

Choosing a measure

Cosine Compares direction. The usual choice for text.
Dot product Fast. Same as cosine on unit vectors.
Euclidean Straight-line distance between points.

Use the measure your embedding model was trained for.

1 / 2

A tiny search engine

To search, we need a corpus, an embedding for each document, and an embedding for the query made the same way. To keep this runnable without a neural model, we use a toy embedding: counts of words from a small vocabulary, with a few synonyms merged into the same coordinate. That merge plays the role of "meaning". A real embedding model learns this from data.

Python Semantic search over five documents

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query "I forgot my credentials" shares no words with "How to reset your password and recover your account", yet it finds it, because "credentials" and "account" fall in the same concept coordinate. Plain keyword search would miss it, which is why hybrid search combines both approaches.

matrix @ q does the whole search in one step: one dot product per document. This is brute-force (or "exact", "flat") nearest-neighbor search.

When brute force is too slow

Brute force costs one comparison per stored vector, per query. That is perfectly fine for thousands of vectors or even a million, especially with NumPy or a GPU. At hundreds of millions of vectors, scanning everything for every query is too slow and too expensive.

Approximate nearest neighbor (ANN) indexes trade a little accuracy for large speedups. They return neighbors that are very likely among the true top results, but not guaranteed. The quality is measured by recall: of the true top-k, what fraction did the index return?

Common ideas:

  • Clustering (IVF): group vectors into clusters. At query time, find the nearest few cluster centers and only scan those clusters.
  • Graph indexes (HNSW): connect each vector to some near neighbors. A search starts at an entry point and repeatedly hops to the neighbor closest to the query, like navigating from a country to a city to a street. "Hierarchical" means upper layers have long links for big jumps and lower layers have short links for fine steps.
  • Quantization: store compressed vectors so more fit in memory (see LoRA, Quantization and Efficient Models).
flowchart LR
    Q["query vector"] --> E["entry point<br/>(top layer)"]
    E --> H1["hop to closer<br/>neighbor"]
    H1 --> H2["drop to a lower layer,<br/>shorter links"]
    H2 --> H3["greedy hops<br/>on the bottom layer"]
    H3 --> R["top-k candidates"]

An HNSW search zooms in layer by layer, visiting only a small fraction of the vectors.

The next example builds a simple cluster-based index and measures its recall against brute force. The data is random with a fixed seed.

Python Brute force vs a clustered index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the data has clear clusters, scanning just 1 of 20 clusters (about 5% of the vectors) already finds 98% of the true neighbors, and 2 probes reach 1.00. Real data is messier, so recall climbs more slowly. Scanning more clusters (more probes) raises recall and costs more work. Vector databases expose exactly this knob under names such as nprobe or ef_search. Choosing it is a balance between speed, memory and the fraction of good results you can afford to miss.

Start with exact search. If your corpus fits in memory and queries are fast enough, there is no accuracy loss and no tuning. Move to an ANN index when measurements, not guesses, say you need it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two vectors point in exactly the same direction but one is three times longer. What is their cosine similarity?

    Cosine similarity ignores length and measures the angle. Same direction means an angle of zero, so the cosine is 1.
  2. 2. Why do many systems normalize embeddings to length 1?

    For unit vectors, dividing by the lengths changes nothing, so the fast dot product gives the cosine directly.
  3. 3. What does recall@10 measure for an approximate index?

    Recall compares the approximate results with the exact brute-force results.
  4. 4. What is the main trade-off of approximate nearest-neighbor search?

    ANN indexes skip most vectors to save time, so a few true neighbors can be missed.

Summary

  • Euclidean distance compares positions; the dot product and cosine similarity compare direction, and cosine ignores length.
  • With unit-length vectors, dot product equals cosine, so searching is matrix @ query.
  • Semantic search embeds the query the same way as the documents and returns the nearest vectors, even without shared words.
  • Brute-force search is exact and fine up to millions of vectors; ANN indexes (clusters, HNSW graphs, quantization) trade a little recall for speed.
  • Tune ANN indexes by measuring recall against exact search.

Next: see how language models predict text in A Language Model in 20 Lines.