Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 53 ยท Retrieval-Augmented Generation (RAG)

RAG: Giving Models Your Data

Understand retrieval-augmented generation: why models need outside data, the retrieve-augment-generate pipeline, RAG vs fine-tuning vs long context, with a toy pipeline.

A language model knows what was in its training data, frozen at a cutoff date. It has never read your company wiki, last week's support tickets or the PDF you uploaded this morning. Ask about those and it will either say it does not know or, worse, produce a convincing guess.

Retrieval-augmented generation (RAG) fixes this without changing the model. Before answering, your program looks up relevant text in your own data and pastes it into the prompt. The model then answers from that supplied text. This chapter builds RAG step by step; this lesson gives the big picture and a complete toy pipeline.

Why models need help

Three limits make RAG useful:

  • Knowledge cutoff: the model cannot know events or documents after training.
  • Private data: internal documents were never in the training set, and you would not want them to be.
  • Verifiability: a model answering from memory cannot show where a claim came from. With RAG, you can show the source passage, which supports citations and reduces hallucinations.

The three-step pipeline

RAG's name is its recipe: retrieve, augment, generate.

flowchart LR
    Q["user question"] --> R["retrieve<br/>top matching chunks"]
    D[("your documents<br/>(indexed ahead of time)")] --> R
    R --> A["augment<br/>put chunks into the prompt"]
    A --> G["generate<br/>model answers from context"]
    G --> ANS["answer + sources"]

Documents are prepared once; every question runs retrieve, augment, generate.

  1. Retrieve. Find the few passages most relevant to the question, using the search ideas from Cosine Similarity and Semantic Search.
  2. Augment. Build a prompt that contains the question plus those passages and an instruction to rely on them.
  3. Generate. The model writes an answer grounded in the supplied text.

There is also an offline stage that runs ahead of time: load documents, split them into chunks, compute an embedding for each chunk and store them in an index. The quality of the whole system depends heavily on this preparation and on retrieval, much more than on the final generation step.

Offline: prepare the data

Load Read documents and clean the text.
Chunk Split into passages.
Embed Turn each chunk into a vector.
Index Store vectors for fast search.
1 / 2

A toy pipeline you can run

The model cannot run on this site, so the generation step below is a stand-in: it picks the retrieved sentence that shares the most words with the question. Retrieval uses simple word-overlap scoring over a tiny knowledge base. The structure is exactly the real thing: index, retrieve, build prompt, generate.

Python End-to-end toy RAG

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last question. Nothing in the corpus matches, so retrieval returns an empty list and the stand-in model says it does not know. A real system needs the same discipline: when retrieval finds nothing useful, the prompt should let the model say so instead of inventing an answer.

Next, a peek at the prompt that actually goes to a real model. Printing it is the best debugging habit in RAG, because most failures show up right there: either the right passage is missing or something confusing is in the way.

Python Inspect the augmented prompt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

RAG vs fine-tuning vs long context

RAG is one of three ways to get knowledge into an application, and they solve different problems.

Approach Best for Weak at
RAG Fresh, large or private facts; answers with sources; easy updates (re-index a file) Needs good search; answers depend on retrieved text
Fine-tuning Teaching style, format or a narrow skill (see Fine-Tuning vs Prompting) Poor way to add facts; updating means retraining
Long context Small document sets that fit in the prompt (see Context Windows and the KV Cache) Cost and latency per request; accuracy can drop on very long inputs

A practical rule: if all the needed text fits comfortably and the volume of questions is low, put it in the prompt. If the data is large, changes often or must be access-controlled, use RAG. If the problem is behavior and tone rather than knowledge, consider fine-tuning. Many products combine them: a lightly fine-tuned model, RAG for facts and a short system prompt for rules.

Where RAG goes wrong

  • Bad chunks: a passage cut mid-thought gives the model half an answer.
  • Retrieval misses: the right text exists but the search did not find it.
  • Irrelevant context: unrelated passages distract the model.
  • Ignored context: the model answers from memory despite the sources.
  • Stale index: documents changed but the index was not updated.
  • Access control: retrieval must respect who is allowed to see which documents, or RAG becomes a data leak.

The chapter that follows tackles these one at a time, starting with Chunking and Preparing Documents.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are the three steps of the RAG pipeline?

    RAG retrieves relevant text, adds it to the prompt (augment), then lets the model generate an answer.
  2. 2. Which problem is RAG best suited for?

    RAG supplies up-to-date or private facts at query time. Tone and format are better handled by prompts or fine-tuning.
  3. 3. What should a RAG prompt allow when retrieval finds nothing relevant?

    Without that option the model is pushed to invent an answer, which is exactly the failure RAG should reduce.
  4. 4. When might long context beat RAG?

    For small sets, pasting everything is simpler and avoids retrieval errors, though it costs more per request.

Summary

  • RAG gives a model your data at question time by retrieving relevant passages and adding them to the prompt.
  • The pipeline is offline preparation (load, chunk, embed, index) plus online retrieve, augment, generate.
  • Retrieval quality usually decides answer quality; always inspect the final prompt.
  • Use RAG for large, fresh or private knowledge, fine-tuning for behavior, long context for small document sets.
  • Let the model say "I don't know" and respect access control in retrieval.

Next: Chunking and Preparing Documents.