Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Applications and Limits

Retrieval-Augmented Generation

Build retrieval-augmented generation step by step: chunking documents, TF-IDF retrieval with scikit-learn, assembling grounded prompts, and measuring recall@k and failure modes.

A language model only knows what was in its training data, up to its cutoff date, and it cannot see your private documents. Retrieval-augmented generation (RAG) fixes that at request time: it searches a document collection for passages relevant to the question, pastes them into the prompt, and asks the model to answer using that material. It is the most common way to build chat over company docs, support knowledge bases and codebases.

flowchart LR
    subgraph Indexing["Offline: indexing"]
    D["Documents"] --> C["Chunk"]
    C --> E["Embed or index"]
    E --> S[("Search index")]
    end
    subgraph Query["Online: each question"]
    Q["Question"] --> R["Retrieve top-k chunks"]
    S --> R
    R --> P["Prompt: question + chunks"]
    P --> L["LLM answers, citing chunks"]
    end

Retrieval quality limits answer quality: the model cannot use passages it never sees.

The retrieval code below is real and runs here. The generation step uses a small stand-in (it just assembles the prompt), because no LLM runs in the browser page.

Step 1: chunking

Documents are split into chunks because prompts have limited space and a focused chunk matches a question better than a whole manual. Common choices: 200 to 500 tokens with 10 to 20% overlap, split at headings or paragraph boundaries. Too small and a chunk loses context; too large and it dilutes the match and wastes the prompt.

Python Sliding-window chunker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Overlap helps when a sentence is cut at a boundary. Production pipelines split by structure (headings, code blocks, table rows) and attach metadata such as title, URL and date to every chunk.

Step 2: retrieval

Two families:

  • Lexical (keyword) search, for example TF-IDF or BM25: matches shared words. Great for exact terms, names and error codes.
  • Dense (embedding) search: each chunk and the query become vectors from an embedding model, and we take the nearest by cosine similarity. It matches meaning ("send back an item" finds "returns"). See the track's lessons on embeddings in the track overview.

Best systems use hybrid search (both, merged) and then a reranker model that rescores the top 20 to 50 candidates. Here we use TF-IDF from scikit-learn as a runnable lexical retriever.

Python TF-IDF retrieval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The retriever returns chunk numbers with similarity scores. A query with no word overlap would find nothing: for instance "send back an item" shares no words with "returns", which is exactly where embeddings help.

Step 3: build the grounded prompt

Tell the model to answer only from the passages, to cite them, and to say when the answer is not there.

Python Assemble the RAG prompt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The instruction to say "I don't know" is what turns missing retrieval into an honest answer instead of a hallucination.

Measuring retrieval

Evaluate the retriever on its own, with a small set of questions each labeled with the chunk(s) that contain the answer.

  • Recall@k: the share of questions whose correct chunk is in the top k.
  • MRR (mean reciprocal rank): average of 1 / rank of the first correct chunk.
  • Answer-level metrics: faithfulness (is the answer supported by the passages?) and answer correctness, often scored with a model judge.
Python Recall@k and MRR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the questions that ranked poorly: they usually reveal a chunking problem, a vocabulary mismatch, or a missing document.

Common failure modes

  • Wrong chunk retrieved: the answer exists but ranks low. Fix with hybrid search, reranking, better chunking, query rewriting.
  • Answer split across chunks: use overlap, larger chunks, or parent-document retrieval (match small, return the bigger section).
  • Stale or conflicting documents: store dates and versions; deduplicate.
  • Too many chunks: extra, loosely related passages distract the model ("lost in the middle"). Retrieve fewer, better ones.
  • Ignoring the context: the model answers from memory. Use clear instructions and check faithfulness.
  • Access control: retrieval must respect who may see which document; filter before the prompt, not after.
  • Prompt injection through documents: retrieved text is untrusted input.

RAG or fine-tuning?

RAG supplies knowledge that changes and needs citations; fine-tuning shapes behavior (Fine-tuning vs Prompting). They combine well: tune the model to follow the citation format, retrieve the facts.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main job of RAG?

    The model's weights stay unchanged; retrieved passages enter through the prompt.
  2. 2. Why use overlap when chunking?

    Overlap keeps boundary-spanning facts intact in at least one chunk.
  3. 3. A query shares no words with the right chunk. Which retriever can still find it?

    Embeddings compare meaning, so paraphrases land close together even without shared words.
  4. 4. Recall@5 is 0.6. What does that mean?

    Recall@k measures how often the retriever places a gold chunk in the top k.

Summary

  • RAG retrieves relevant chunks and gives them to the model as context so answers are current, private and citable.
  • Chunking, retrieval (lexical, dense, hybrid) and reranking decide most of the quality.
  • Instruct the model to use only the passages and to admit when they do not contain the answer.
  • Measure retrieval separately (recall@k, MRR) and answers separately (faithfulness, correctness).

Next: Tool Use and Agents