Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 36 ยท Generation and Prompting

RAG Prompt Templates and Context Assembly

Learn how to write a RAG prompt, label sources, fit chunks into a token budget, order them to avoid the lost-in-the-middle effect, and handle empty context.

Retrieval gives you chunks. Generation turns them into an answer, and the bridge between the two is the prompt. A RAG prompt has to do four jobs: tell the model its role and the rules, present the retrieved evidence clearly, carry the user's question, and say what to do when the evidence is not enough. Small changes here often matter more than swapping the model.

This lesson builds the prompt in code. Calling a hosted language model requires a network and an API key, which the browser sandbox does not allow, so the examples stop at the assembled prompt string and use a tiny stand-in function where an answer is needed. The assembly logic is exactly what you would run before a real API call (see citations and grounded answers).

Anatomy of a RAG prompt

  1. System instructions: role, tone, language, and the grounding rule ("answer only from the sources").
  2. Context block: the chunks, each with an identifier and often a title or URL.
  3. User question.
  4. Output format: length, citation style, JSON schema if needed.
  5. Fallback rule: what to say if the sources do not contain the answer.

Most chat APIs let you put 1 and 5 in a system message, and 2, 3 and 4 in the user message. Keep the context and the question clearly separated with delimiters so the model (and you, when debugging) can see where each begins.

Python Assembling a prompt from retrieved chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the pieces: the rule is stated once, sources carry numeric ids that the answer can cite, and XML-style tags separate evidence from the question. Tags are a convention, not a requirement; models follow clear structure of almost any kind.

Instruction design that works

  • Be specific about grounding. "Use only the sources" beats "be accurate".
  • Give an exit. Without an explicit "say you don't know" instruction, models tend to fill gaps with plausible text.
  • State conflicts policy. "If sources disagree, say so and prefer the most recent."
  • Fix the format. Length, language, bullet or prose, citation style.
  • Keep it short. Long rulebooks are followed less reliably than five clear sentences.
  • Treat context as data. Tell the model that text inside <sources> is reference material and not instructions (prompt injection).

Fitting the context budget

A model has a context window (the maximum tokens of prompt plus answer), and you pay per token. You also want some room for the answer. A budget might be: system 300 tokens, question 200, answer 800, leaving the rest for sources. Count tokens with the model's own tokenizer when you can; a rough rule for English is one token per 0.75 words, or four characters.

Strategy: sort chunks by final relevance, add them in order until the budget is full, and skip (rather than truncate mid-sentence) a chunk that no longer fits, then try smaller ones.

Python Packing chunks into a token budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here a (about 200 tokens) fits, b (160) does not, but the smaller c and d still do. A greedy loop like this is enough in practice. If the loop skips your best chunk, that is a signal to re-chunk at a smaller size or to compress the chunk, not just to raise the budget.

Ordering: the "lost in the middle" effect

Studies of long-context models found that they use information at the start and end of the context more reliably than information in the middle. Two practical consequences:

  • Do not stuff 30 chunks and hope. Fewer, better chunks beat many.
  • When you must send many, put the strongest evidence first and last (for example order by relevance, then move the weakest to the middle).

Newer models are better at long context, but the effect has not disappeared. Test it on your own questions with the evaluation methods later in this course.

Python Ends-first ordering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The most relevant (1) leads, the second most relevant (2) is last, and the weakest (5, 6 region) sit in the middle.

Merging neighbours and adding provenance

  • Merge adjacent chunks from the same page into one passage in reading order; it reads better and wastes fewer tokens on repeated headers.
  • Include titles and section headings in each block. A chunk that says "It costs 5 dollars" is useless without knowing what "it" is. Many pipelines prepend a breadcrumb such as Billing > Team seats.
  • Include dates or versions when freshness matters.
  • Keep identifiers stable so citations can link back to the source.

When there is no context

Sometimes retrieval returns nothing above your threshold. Do not call the model with an empty sources block and the same instructions; it may answer from its training data. Either return a fixed "not found" message without calling the model, or switch to an explicit prompt for the no-evidence case (for example, ask a clarifying question or offer to contact support).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why include an explicit 'say you don't know' instruction?

    An explicit fallback rule gives the model a safe option when the sources lack the answer.
  2. 2. What is the 'lost in the middle' effect?

    Evidence in the middle of a very long context is used less reliably.
  3. 3. A chunk does not fit the remaining budget. What is a sensible action?

    Cutting mid-sentence can change meaning; skipping or re-chunking is safer.
  4. 4. Why prepend a title or breadcrumb to each chunk in the prompt?

    Headings give the chunk the context it lost when it was split from its document.

Summary

  • A RAG prompt has instructions, labelled sources, the question, an output format and a fallback rule.
  • Treat sources as data; keep instructions short and specific.
  • Pack chunks into a token budget greedily, leaving room for the answer.
  • Put the strongest evidence at the start and end; fewer, better chunks beat many.
  • Handle the empty-context case explicitly instead of calling the model blindly.

Next: Citations and Grounded Answers