Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 32 ยท Retrieval and Vector Stores

Retrieval-Augmented Generation: The Big Picture

Understand retrieval-augmented generation (RAG): why models need your data, the two phases of indexing and answering, the LangChain components involved and typical failure points.

A model knows what was in its training data, up to a cutoff date, and nothing about your company handbook, last week's tickets or the PDF a user just uploaded. You could paste documents into the prompt, but context windows are finite and every token costs money. Retrieval-augmented generation (RAG) is the standard answer: store your documents in a searchable index, retrieve only the few passages relevant to each question, and put those into the prompt so the model answers from them.

RAG is the most common LLM application pattern, and this chapter builds it from the ground up: loading and splitting documents, embedding them, storing vectors, retrieving, and wiring the result into a chain.

Why not fine-tune, or paste everything?

Approach Strength Weakness
Paste all documents in the prompt Simplest Limited by context size; costly; models can overlook details in huge prompts
Fine-tune the model on your data Changes style and behavior Poor at adding precise, changing facts; expensive to update; cannot cite sources
RAG Fresh data, citeable sources, small prompts, easy updates (re-index a file) Quality depends on retrieval; more moving parts

Facts that change belong in an index, not in model weights. Behavior and tone are the better candidates for fine-tuning.

The two phases

RAG has an indexing phase that runs ahead of time and a query phase that runs per question.

flowchart TB
    subgraph Index["Indexing (offline)"]
        L["Load documents"] --> S["Split into chunks"] --> E["Embed each chunk"] --> V[("Vector store")]
    end
    subgraph Query["Answering (per question)"]
        Q["User question"] --> QE["Embed the question"] --> R["Retrieve top-k chunks"]
        R --> P["Build prompt:<br/>context + question"] --> M["Chat model"] --> A["Answer with sources"]
    end
    V --> R

Indexing prepares the knowledge; answering looks things up and asks the model to use them.

Mapping to LangChain components:

Step Component Covered in
Load Document loaders Document Loaders and Text Splitters
Split Text splitters same lesson
Embed Embeddings models Embeddings and Similarity
Store and search VectorStore Vector Stores and Retrievers
Retrieve Retriever (a Runnable) same lesson
Generate Prompt, chat model, parser Building a RAG Chain

The idea in miniature (runnable)

Before the real components, see the whole loop with the simplest possible "retriever": count shared words. It is crude, but the data flow is exactly RAG: documents in, relevant ones selected per question, a prompt built from them.

Python Stand-in: RAG in twenty lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Everything RAG adds is a better version of one piece: smarter loading, smarter chunking, meaning-based search instead of word overlap, and a real model at the end.

Where RAG goes wrong

Most RAG failures are retrieval failures, not model failures. If the right passage is not in the prompt, no model can use it.

  • Bad chunks. A split in the middle of an explanation leaves each half useless.
  • Weak matching. The question and the passage use different words for the same thing, and the retriever misses it.
  • Too few or too many results. Missing the answer, or burying it in noise.
  • Ignored context. The model answers from memory instead of the passages; the prompt must insist on using the context.
  • Stale index. Documents changed but were not re-indexed.
  • Unanswerable questions. Without an explicit "say you do not know" rule, models invent answers.

Diagnosing in order helps: first check what was retrieved (print it), then the prompt, then the model's answer.

What "good" looks like

A solid RAG system, built in the order of this chapter, has:

  1. Clean, structured source text with metadata (title, URL, date, section).
  2. Chunks sized to hold one idea, with a little overlap.
  3. Embeddings suited to your language and domain.
  4. A retriever tuned on real questions (how many results, optional filters, re-ranking).
  5. A prompt that requires grounding and citing sources.
  6. An evaluation set of questions with known answers, run after every change (see the evaluation chapter).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main purpose of RAG?

    RAG retrieves relevant passages and adds them to the prompt so the model can answer from your data.
  2. 2. Which phase runs ahead of time, before any user question?

    Indexing prepares the vector store; retrieval and generation happen per question.
  3. 3. A RAG answer is wrong. What should you check first?

    If the right passage was never retrieved, the model could not use it; retrieval is the usual culprit.
  4. 4. Why is RAG preferred over fine-tuning for frequently changing facts?

    Re-indexing a changed document is cheap and fast; changing model weights is not.

Summary

  • RAG retrieves relevant passages from your data and adds them to the prompt.
  • Two phases: index (load, split, embed, store) and answer (embed question, retrieve, prompt, generate).
  • Most failures are retrieval problems: bad chunks, weak matching, wrong k, stale index.
  • Insist on grounded answers and an "I do not know" path; evaluate with a question set.

Next: Document Loaders and Text Splitters