Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 53 ยท Retrieval-Augmented Generation (RAG)

Vector Retrieval in Practice

Embed a corpus, run top-k retrieval, add metadata filters and keep an index up to date. Build a tiny vector store in NumPy and learn what real ones add.

In the last two lessons you turned documents into chunks and learned how cosine similarity finds nearby vectors. This lesson joins them into the retrieval half of a RAG system: embed every chunk once, embed each question at query time, and return the top-k most similar chunks.

On paper that is a few lines of code. In practice the details decide whether users get good answers: which embedding model you use, how many results you fetch, how you filter by metadata, and how you keep the index in step with changing documents. We will build a small vector store in NumPy so that each of those ideas is visible, then look at what production vector databases add.

Retrieval in four steps

Embed Turn every chunk into a vector, once.
Embed the query Same model, same settings.
Score Cosine similarity against every chunk.
Take top-k Keep the best few as context.

Ingestion happens once; only the query side runs for every question.

1 / 2

Embedding a corpus

Retrieval has two phases. Ingestion (offline) reads documents, splits them into chunks, embeds each chunk and stores the vectors with the chunk text and metadata. Querying (online) embeds the user's question with the same model, compares it with the stored vectors and returns the best matches.

Two rules prevent the most common bugs:

  • Use the same embedding model, and the same text preparation, for documents and queries. Vectors from different models live in different spaces and cannot be compared. If you change the model, you must re-embed everything.
  • Normalize the vectors to length 1 so that a dot product is the cosine similarity (see Cosine Similarity and Semantic Search).

A real embedding model is a neural network behind an API or a local library. Here we cannot call one, so we use a stand-in: scikit-learn's TF-IDF vectorizer. It builds a vector from word counts, down-weighting words that appear in every document. It is not a semantic model: it only matches shared words. That limitation is useful for teaching, because you will see exactly the failure that real embeddings are designed to fix.

Python Embed a small corpus and retrieve the top-k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two questions share words with the right chunk, so they work. The third ("how do I get my money back") scores zero everywhere: the refund chunk says "refunds" and "payment method", and nothing overlaps with "money back". A real embedding model places "money back" and "refund" near each other, which is the whole point of embeddings. Remember that when you read the scores in this lesson: the mechanics are real, but the stand-in is weaker than a neural embedder.

Choosing k and a score threshold

The number of results, k, is your main retrieval dial.

  • Too small and the answer may not be in the results at all (low recall).
  • Too large and you pay for extra tokens, push the relevant text into the middle of the prompt where models attend less (see Context Windows and the KV Cache), and give the model more distracting text.

Typical values are 3 to 10 chunks for generation, and a larger first-stage number (25 to 100) when a reranker will cut the list down afterwards.

A minimum score is a useful second guard: if even the best chunk is a poor match, it is better to say "I could not find that" than to hand the model irrelevant text. The right threshold depends on the embedding model and must be found by measuring, because score scales differ between models.

What a vector store gives you

A vector store (or vector database) is more than an array of numbers. It stores, for each item:

  • an id you can use to update or delete it;
  • the vector;
  • the original text (or a pointer to it);
  • metadata such as source, section, language, date, access group.

On top of that it provides similarity search with an index, metadata filters, and operations to add, update and delete items. Filters matter more than people expect. "Only search documents the current user may see" and "only the 2025 handbook" are filters, and for access control they are a security requirement, not an optimization: apply them inside the search, not by hiding results afterwards.

Filtering after retrieval ("post-filtering") can return fewer than k results, or none at all, if the top hits are all filtered out. Good stores filter during the search ("pre-filtering" or filtered ANN) so you still get k results from the allowed set.

Python A tiny vector store with filters, updates and deletes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the filtered query for topic=shipping returns nothing: a minimum score of 0 keeps only chunks with some overlap, and no shipping chunk matches. A real store would return the nearest shipping chunks regardless, so thresholds are your job. Also note that updates replace a vector in place, and a deleted document stops appearing at once.

Keeping the index fresh

Documents change. A reliable ingestion pipeline needs:

  • Stable ids built from the source and chunk position (for example handbook.md#section-3-chunk-2), so re-ingesting a file updates the same items instead of duplicating them.
  • Content hashes: store a hash of each chunk's text and re-embed only chunks whose hash changed. Embedding costs money and time.
  • Deletion: when a document disappears, or shrinks and loses chunks, remove the stale ids. Orphaned chunks keep answering questions with outdated facts.
  • Versioning: record the embedding model name and version in the metadata. When you switch models, build a new index next to the old one and swap over once the new one passes your tests.

This is the same idea as the incremental build step this website uses for its own content: skip what has not changed, remove what no longer exists.

Brute force, ANN and real vector databases

For a few thousand chunks, the brute-force matrix @ q above is already fast and exact. Beyond a few hundred thousand vectors, stores use approximate nearest-neighbor (ANN) indexes such as HNSW graphs or clustered (IVF) indexes, trading a little recall for large speedups (see Cosine Similarity and Semantic Search). You will meet these options:

  • Libraries (FAISS, hnswlib, ScaNN): fast in-process indexes with few features around them.
  • Dedicated databases (Qdrant, Weaviate, Milvus, Pinecone): add storage, filtering, replication and APIs.
  • Extensions to databases you already run (such as pgvector for PostgreSQL, or vector fields in search engines): simplest to operate when your data and permissions already live there.

Start with the simplest option that fits your scale. The retrieval quality problems you will actually hit (chunking, vocabulary mismatch, missing keywords) are not solved by a fancier index, which is why the next lesson adds keyword search and reranking.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must documents and queries be embedded with the same model?

    Each model defines its own coordinate system. Comparing vectors across models is like comparing latitude from one map with longitude from another.
  2. 2. What is the risk of choosing a very large k for the context you send to the model?

    Extra chunks cost tokens and can bury the useful passage. A larger first-stage k followed by reranking is a better way to improve recall.
  3. 3. A user may only see documents from their own team. Where should that rule be enforced?

    Anything retrieved can end up in the prompt. Access control must happen before the model sees the text.
  4. 4. A document is edited and now has fewer chunks than before. What must the ingestion pipeline do?

    Stale chunks keep matching queries and feed outdated facts to the model, so removed chunks must be deleted.

Summary

  • Retrieval embeds each chunk once (ingestion) and each query at question time, then returns the top-k nearest chunks.
  • Use one embedding model for both sides, normalize vectors, and re-embed everything if you change the model.
  • Tune k and a minimum score by measuring; use a larger first-stage k when a reranker follows.
  • A vector store adds ids, text, metadata, filters, updates and deletes around the vectors; apply access filters inside the search.
  • Keep the index fresh with stable ids, content hashes, deletions and a model version.
  • Brute force is fine for small corpora; ANN indexes and vector databases matter at scale.

Next: combine vector and keyword search in Hybrid Search and Reranking.