Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 38 ยท Retrieval and Context Engineering

Retrieval-Augmented Generation (RAG) Basics

Understand retrieval-augmented generation: find relevant text first, then ask the model to answer only from it. Build a tiny retriever and grounded prompt.

A model's built-in knowledge is frozen at training time, knows nothing about your private documents, and sometimes invents plausible nonsense. Retrieval-augmented generation (RAG) fixes this by splitting the job in two: first retrieve the few passages that matter for this question, then generate an answer from the question plus those passages.

For prompt engineers, RAG is mostly a prompt-assembly problem. The retriever picks the evidence; your prompt tells the model how to use it, what to do when the evidence is missing, and how to cite it.

As in the rest of this chapter there is no real model on this site. The retriever below is real working Python (simple word scoring); the "model" is a clearly labelled stand-in.

The RAG loop

flowchart LR
    Q["User question"] --> R["Retriever"]
    D[("Documents")] --> R
    R --> P["Prompt: rules + passages + question"]
    P --> M["Model"]
    M --> A["Grounded answer with sources"]

Retrieval narrows many documents to a few passages; the prompt wraps them with rules.

Offline, you split documents into chunks (next lessons) and index them. Online, every question triggers retrieval, prompt assembly and generation.

A tiny retriever

Production systems often use embeddings (vectors that capture meaning) or a search engine. The idea is the same as this keyword scorer: give every chunk a relevance score for the question and keep the top few.

Python A keyword retriever with TF-IDF scoring

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last question retrieves nothing. That is a feature: a good pipeline knows when it has no evidence, and the prompt must handle that case.

The grounding prompt

The prompt does four jobs: set the role, restrict the model to the supplied passages, say what to do when they do not contain the answer, and require sources.

TEXT
You are a support assistant for Acme Shop.
Answer the question using ONLY the passages between <passages> tags.
If the passages do not contain the answer, reply exactly:
"I don't have that information."
After each claim, cite the passage id in square brackets.

<passages>
[refunds-1] Refunds are issued to the original payment method within 5 business days.
</passages>

Question: How long do refunds take?

Labelled passages with ids make citations possible and let you check later whether the answer matches its sources. The refusal sentence gives the model a safe exit instead of a guess.

Python Assemble a grounded prompt (stand-in model)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where RAG goes wrong

Most RAG failures are not model failures:

  • Retrieval miss: the right passage was never found, so the model improvises. Fix chunking, scoring or the query.
  • Distractors: off-topic passages appear and the model uses them. Retrieve fewer, better passages.
  • Stale or conflicting sources: two passages disagree. Add dates or versions to chunks and tell the model which wins.
  • Ignored instructions: the model answers from memory despite "only use the passages". Tighten the rule, add the refusal text and test it.
  • Missing citations: require ids and verify them in code (see output guardrails).

When an answer is wrong, look at the retrieved passages first. If they did not contain the answer, no prompt rewrite will fix it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem does RAG mainly address?

    RAG supplies relevant documents at question time, so the model can answer from information it was never trained on.
  2. 2. Why give each passage an id?

    Ids allow citations, and you can verify in code that cited ids exist and support the claim.
  3. 3. What should the prompt say when the passages lack the answer?

    A defined fallback gives the model a safe exit and makes missing evidence easy to detect and test.
  4. 4. A wrong answer arrives and the retrieved passages were off-topic. Where do you fix it?

    If the evidence is missing, rewording the generation prompt cannot help; improve the retrieval step.

Summary

  • RAG retrieves relevant passages first, then generates an answer from them.
  • The prompt restricts the model to the passages, defines a refusal and asks for citations.
  • Always handle the zero-results case explicitly.
  • Debug RAG by inspecting retrieved passages before touching the prompt.

Next: Chunking Documents for Prompts