Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 63 ยท Neural NLP and Transformers

Retrieval and RAG

Learn how retrieval-augmented generation works: chunk documents, retrieve the best passages, assemble a grounded prompt and measure retrieval quality in Python.

A language model only knows what was in its training data, and that stops at a cutoff date. It has never seen your company's handbook, last week's tickets or your private notes, and it tends to hallucinate when it lacks a fact. You could try to paste everything into the prompt, but context windows are limited and every token costs money.

Retrieval-augmented generation (RAG) solves this in two steps: first retrieve the few passages that are relevant to the question, then generate an answer from a prompt that contains only those passages. The model reads the facts instead of remembering them, and you can show the user which passages the answer came from. The retrieval half is classic search, so you already know most of what you need from document similarity and building a small search engine. This lesson builds a complete miniature pipeline, using TF-IDF as a stand-in for the neural embeddings that production systems use.

The RAG pipeline

Prepare once

Chunk Cut documents into small passages.
Index Turn each chunk into a vector and store it.
1 / 3

Step 1: chunking

Whole documents are too big and too unfocused: a 40-page manual produces one vector that mixes everything. So you cut documents into chunks, typically 100 to 500 words, small enough to be specific and large enough to be understood alone. Good chunk boundaries follow the structure of the text (headings, paragraphs, sentences, see sentence segmentation). A simple fixed-size splitter also works, but it can cut an idea in half, which is why chunks usually overlap: the end of one chunk is repeated at the start of the next. The code splits a short policy into tiny 20-word chunks, with and without overlap, so you can see the difference.

Python Chunking with and without overlap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without overlap, the second chunk starts in the middle of "To start a return", and the sentence about starting a return is split across chunks 0 and 1, so neither chunk contains the full instruction. With an overlap of 5 words, the phrase "its original packaging. To start" appears in both chunks 0 and 1, so context straddling a boundary is not lost. The cost is more chunks (4 instead of 3), which means a larger index and some duplicated text in the prompt. Real systems tune size and overlap on their own data; small chunks give precise retrieval but lose context, large chunks keep context but dilute the match.

Step 2: retrieval

Each chunk is converted to a vector, and a question is converted the same way and compared with every chunk by cosine similarity. Production systems usually use dense embeddings from a neural model (the kind of vector from sentence embeddings), stored in a vector database or an index such as FAISS, because embeddings match meaning rather than exact words. Here we use TF-IDF vectors, which work the same way and run in the browser. The retrieval code is identical whichever vectors you plug in.

Python Retrieving the best chunks with TF-IDF

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two questions work: "refunds" and "forgot password" share rare words with the right chunks, so chunk 1 scores 0.45 and chunk 4 scores 0.67. The third question fails completely. The user says "arrive", but the shipping chunks say "takes" and "delivers", so there is no word in common and every score is 0.00. This is the vocabulary mismatch problem of keyword search, and it is the main reason production systems use dense embeddings, which place "arrive" and "delivers" close together. A common upgrade is hybrid search: combine a keyword score (such as BM25) with an embedding score, because keywords are good at exact names and codes while embeddings are good at paraphrases.

Step 3: evaluating retrieval

You cannot improve what you do not measure. To evaluate retrieval, collect real questions and mark which chunk contains the answer, then compute:

  • Recall@k: the fraction of questions where the right chunk is among the top k results. This matters because the generator can only use what you retrieve.
  • MRR (mean reciprocal rank): the average of 1 / rank of the first right chunk, which rewards putting it near the top.

The next program compares two ways of vectorizing the same chunks on six labeled questions. It also prints the rank of the correct chunk for every question, so you can see where each setup fails.

Python Recall@k and MRR for two retrievers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both setups find five of the six right chunks in the top 2 (recall@2 of 0.83). The one miss is the "arrive" question again: the word setup ranks the correct chunk 6th, and the character n-gram setup ranks it 4th, still too low to help. Character n-grams help a little with word forms (MRR 0.88 against 0.86) but cannot bridge "arrive" and "delivers". Six questions are far too few to trust small differences; a real evaluation set has dozens to hundreds. The per-question ranks are the most useful output: they tell you exactly which kinds of questions need a better retriever.

Step 4: assembling the prompt

Once you have the top chunks, you put them in a prompt with clear instructions. Three habits matter: number the passages so the model can cite them, tell the model to answer only from the passages and to say it does not know otherwise, and keep an eye on the context budget.

Python Assembling a grounded prompt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prompt is about 89 rough tokens, within our pretend budget of 120. With more or longer chunks you would trim from the bottom of the ranked list, because the lowest-ranked chunks are the least useful. Sending the prompt to a model is one more call, which depends on the provider you use, so we stop at the finished text. Notice that the prompt offers an escape ("I don't know"). Without it, a model asked something the passages cannot answer will often invent an answer anyway.

Practical tips

  • Put the most relevant passages first (models attend best to the start and end of long contexts).
  • Store metadata (title, URL, date) with each chunk so you can show citations and filter, for example by product or language.
  • Retrieve more than you need (say 20), then rerank with a stronger model and keep the top 3 to 5.
  • Keep the index fresh: when a document changes, re-chunk and re-embed it.
  • Log questions where retrieval failed; they are your best source of new evaluation cases.
  • Never retrieve data the user is not allowed to see. Apply access control at retrieval time, not in the prompt.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main purpose of RAG?

    RAG retrieves passages that are relevant to the question and puts them in the prompt, which grounds the answer and lets you cite sources.
  2. 2. Why do chunks usually overlap?

    A fixed-size cut can split a sentence or an instruction. Repeating a few words at each boundary keeps the surrounding context in both neighbors.
  3. 3. In our experiment, why did the question "when does my order arrive" fail with TF-IDF?

    Keyword methods need shared words. This is the vocabulary mismatch problem, which dense embeddings and hybrid search help solve.
  4. 4. What does recall@3 measure in a retrieval evaluation?

    If the right chunk is not retrieved, the generator cannot use it, so recall@k is the key upper bound for answer quality.
  5. 5. Why should a RAG prompt tell the model to say "I don't know" when the passages lack the answer?

    Models try to be helpful and fluent, so unless told otherwise they may fill gaps with guesses. An explicit fallback makes unsupported answers visible.

Summary

  • RAG retrieves relevant passages first and generates from them, which grounds answers and enables citations.
  • Chunks of 100 to 500 words with a little overlap keep ideas intact; chunk size trades precision against context.
  • Retrieval compares vectors with cosine similarity; TF-IDF works for exact words, dense embeddings and hybrid search handle paraphrases.
  • Evaluate retrieval separately with recall@k and MRR on labeled questions, and read the per-question ranks.
  • The prompt should number passages, restrict the answer to them and offer an "I don't know" way out.

Next, see how modern models handle specific tasks in Translation, Question Answering and Summarization.