Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 53 ยท Projects

Project: Question Answering over Documents

Build a small RAG pipeline in Python: chunk documents, index them, retrieve the best passages and answer with citations using an extractive fake generator.

This project builds a complete question-answering system over your own documents: chunk, index, retrieve, answer, cite. It is the pipeline from RAG: Giving Models Your Data in one small program. The corpus is a handful of short help-center articles, and the "generator" is an extractive stand-in that picks the best sentence instead of writing new text. That keeps everything deterministic, and the code around it is the same as for a real system.

By the end you will have functions for each stage, citations that point back to the source, and an honest "I don't know" for questions the documents do not answer.

The pipeline

flowchart LR
    A["Documents"] --> B["Chunk"]
    B --> C["Index<br/>(TF-IDF vectors)"]
    D["Question"] --> E["Retrieve top-k"]
    C --> E
    E --> F["Build prompt<br/>with sources"]
    F --> G["Generate answer<br/>+ citations"]

Indexing happens once, ahead of time. Retrieval and generation happen for every question.

We will keep each stage a small function with a clear input and output, so you can replace one without touching the others: a better chunker, real embeddings, a real model.

Step 1: the corpus and chunking

Real documents are long, so we split them into chunks small enough to be specific and large enough to make sense (see Chunking and Preparing Documents). Here each article is split into sentences, and we group sentences into chunks of up to two. Every chunk keeps metadata: the document title and its position, which we need for citations later.

Python Step 1: documents to chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: an index

To search, we turn each chunk into a vector. Real systems use embeddings; to stay dependency-free and deterministic we use a TF-IDF bag of words: each word becomes a dimension, weighted by how often it appears in the chunk (term frequency) and how rare it is across all chunks (inverse document frequency). Rare, distinctive words such as "warranty" count more than "the". Retrieval is then cosine similarity between the question vector and each chunk vector, exactly as in Vector Retrieval in Practice.

Python Step 2: TF-IDF index and retrieval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the second question first: it scores zero although the Returns article is clearly relevant, because "return" and "returned" are different words to a bag-of-words index. That is the weakness embeddings and stemming fix. Now look at the third question. Nothing in the corpus is about France, yet search still returns its "best" two chunks, just with a score of zero. A retriever always returns something. Deciding that it is not good enough is the job of the next stage.

Step 3: answer with citations

The generator receives the question plus the retrieved chunks and must answer only from them, cite which chunk supports each claim, and admit when the sources do not contain the answer (see Answering with Context and Citations). A real system builds a prompt like this and sends it to a model. Our extractive stand-in makes the same decisions with simple rules: if the best score is below a threshold, refuse; otherwise return the sentence that shares the most words with the question, followed by its citation.

Python Step 3: the full pipeline with citations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two behaviors make this a trustworthy system rather than a toy: every answer carries a citation a user can check, and weak evidence produces "I don't know" instead of a guess. A real generator writes fluent text, which makes both safeguards even more important, because a fluent wrong answer is the worst outcome (Hallucinations and How to Reduce Them).

Step 4: test it like a product

A pipeline has many parts that can fail quietly, so test the stages separately:

  • Retrieval: for a list of questions with known source chunks, is the right chunk in the top 3? (Metrics: Retrieval, Benchmarks and Statistics)
  • Citations: does every cited id exist in the retrieved set?
  • Refusal: do out-of-scope questions get "I don't know"?
Python Step 4: checking retrieval and citations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The run reports valid citations: 4/5 on purpose: for "How do I get a refund?" the right chunk is in the top 3, but its score is too low (the question says "refund", the chunk says "Refunds"), so the generator refuses and cites nothing. The test points straight at the weak spot. If a test fails, you know which stage to fix: a miss in retrieval means better chunking or embeddings, a bad citation means a prompt or post-processing problem, and a wrong answer with the right chunk points at the generator.

Where to go next

  • Swap TF-IDF for real embeddings and add BM25 with hybrid search and reranking.
  • Store metadata (document, section, date) and use it as a filter.
  • Replace the extractive generator with a real model, and add a check that every claim in the answer is supported by a cited chunk.
  • Grow your test list into a proper evaluation harness.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the pipeline need a threshold on the best retrieval score?

    Nearest-neighbor search always has a nearest neighbor. Without a threshold (or a model that refuses), irrelevant chunks would be used to produce confident wrong answers.
  2. 2. What is metadata on a chunk used for?

    Title, position or date let you cite sources, filter by document or date and debug where an answer came from.
  3. 3. In TF-IDF, why does a rare word like 'warranty' count more than 'the'?

    IDF rewards words that distinguish one chunk from the others. A word in every chunk tells you little about which chunk matches.
  4. 4. A test shows the right chunk is retrieved but the answer is wrong. Which stage should you inspect?

    Retrieval did its job, so the failure is in how the answer was produced from the context: the prompt, the model or post-processing.

Summary

  • A RAG question-answering system has five stages: chunk, index, retrieve, generate and cite.
  • Chunks carry metadata so answers can cite their sources.
  • TF-IDF vectors with cosine similarity are a simple, deterministic stand-in for embeddings.
  • A retriever always returns results, so use a score threshold or instruct the model to say "I don't know".
  • Citations make answers checkable; validate that every cited id was actually retrieved.
  • Test retrieval, citations and refusals separately to know which stage to fix.

Next: measure quality systematically in Project: An Evaluation Harness.