Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 39 ยท Memory and Retrieval

Embeddings and Vector Search

Understand embeddings, cosine similarity and nearest-neighbor search, then build a tiny vector index in Python to retrieve the most relevant text for an agent.

Keyword search fails when words differ: a user asks about "cancelling my subscription" and the help page says "ending your plan". An embedding solves this by turning text into a list of numbers (a vector) so that texts with similar meaning end up close together. Agents use embeddings to find relevant memories, documents and examples without relying on exact words.

This lesson explains what embeddings are, how similarity is measured, and how a vector index finds the nearest items. You will build the whole pipeline with a toy embedding so every step is visible and runnable.

Real embeddings come from a trained model behind an API or a local library, and none is available on this site. The "embedding" below is a deliberately simple stand-in (hashed word counts) that captures word overlap, not true meaning. The vector math, ranking and index code are exactly what you would use with real vectors.

What an embedding is

An embedding model maps text to a fixed-length vector such as 384 or 1536 numbers. Nobody reads the individual numbers. What matters is geometry: two texts about the same idea produce vectors that point in similar directions.

flowchart LR
  T1["cancel my subscription"] --> E["Embedding model"]
  T2["ending your plan"] --> E
  T3["best pasta recipe"] --> E
  E --> V1["[0.12, -0.40, ...]"]
  E --> V2["[0.10, -0.37, ...]"]
  E --> V3["[-0.55, 0.21, ...]"]

The first two vectors are close; the third is far away.

Measuring closeness: cosine similarity

The usual measure is cosine similarity: the cosine of the angle between two vectors. It is 1 when they point the same way, 0 when unrelated and negative when opposite. The formula is the dot product divided by the product of the lengths.

Python Cosine similarity by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Scaling a vector does not change the cosine, so a long document and a short one about the same topic can still match. Many embedding models return vectors already normalized to length 1, in which case the plain dot product equals the cosine.

A toy embedding

Our stand-in lowercases text, splits it into words, and hashes each word into one of 64 buckets, adding one to that bucket. It is crude, but it makes texts that share words similar.

Python Embed text and rank by similarity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The password page wins because it shares words with the query. With a real model it would also win for "I forgot my login", which shares no words at all. That is the whole reason to use real embeddings.

Building a vector index

A vector index stores one vector per chunk of text and returns the top k nearest to a query. The simplest version, brute force, compares the query to every vector. It is exact and perfectly fine up to tens of thousands of items.

Python A tiny vector index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two details deserve attention. The min_score threshold lets the index return nothing when nothing is relevant, which stops the agent from being fed unrelated text. The metadata lets you filter later ("only documents from the policy source") and show citations.

Approximate search at scale

Brute force slows down with millions of vectors. Production systems use approximate nearest neighbor (ANN) indexes such as HNSW graphs or IVF clusters. They give up a little accuracy (recall) for big speed gains. You rarely implement them yourself: a vector database or a library handles it. Choose one by size, filtering needs and whether you already run a database that offers a vector extension.

Chunking text

Embedding a whole book into one vector blurs everything together. Split documents into chunks of a few hundred words, often with a small overlap so an idea cut at a boundary appears whole in one chunk. Keep the title and source with each chunk as metadata. Smaller chunks give precise hits; larger ones carry more context. Test both on your own questions.

Common mistakes

  • Mixing models. Vectors from different embedding models cannot be compared. Re-embed everything if you change models.
  • No threshold. Without a minimum score, the top result is always returned, even if it is irrelevant.
  • Embedding queries like documents. Some models expect a special prefix or a different mode for queries; read the model's guidance.
  • Trusting similarity as truth. A close vector means related, not correct. The model must still read the text.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an embedding model produce?

    Each text becomes a vector; similar meaning gives nearby vectors.
  2. 2. Cosine similarity of 1 means the vectors...

    Cosine measures the angle; 1 is the same direction, 0 unrelated, -1 opposite.
  3. 3. Why use a minimum score when searching?

    Without a threshold, the nearest item is returned even if it is unrelated.
  4. 4. Why split long documents into chunks?

    Smaller chunks give more precise retrieval.

Summary

  • Embeddings map text to vectors so similar meaning lands close together.
  • Cosine similarity compares directions; normalized vectors make it a dot product.
  • A vector index returns the top-k nearest chunks; add a score threshold and metadata.
  • Brute force is exact and fine for small data; ANN indexes serve large collections.
  • Chunk documents, and never mix vectors from different models.

Next: Retrieval-Augmented Generation for Agents