Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Question Answering over Documents

Build an end-to-end question-answering pipeline: chunk documents, retrieve with TF-IDF, abstain below a threshold, cite sources and measure accuracy, using a labelled stand-in generator.

In this project you assemble a complete question-answering system over a small document set: chunking, retrieval, a decision to answer or abstain, a prompt with citations and an evaluation. It follows the pipeline from retrieval-augmented generation and finishes with numbers you can compare after each improvement.

The retriever (TF-IDF in scikit-learn) is real. The final "generator" is a stand-in: it builds the grounded prompt a real model would receive, then picks the best-matching sentence as an extractive answer. Replace generate() with an API call to use a real LLM.

The knowledge base

We use a handful of short policy documents about a made-up shop. Each has an id so answers can cite it.

Python Documents and chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One sentence per chunk keeps this example transparent. Real systems use larger, overlapping chunks (see the chunking section of the RAG lesson).

Retrieval with an abstain threshold

The retriever scores every chunk. If even the best score is low, the question is probably outside the documents, and the system should say so rather than guess.

Python Retrieve or abstain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The threshold is a tuning knob: too high and the system refuses answerable questions, too low and it answers from irrelevant text. Choose it using your evaluation set, not by feel.

Prompt, stand-in generator and citations

Python The full pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at "When will I get my money back?": it shares almost no words with "Refunds go back to the original payment method", so lexical retrieval finds nothing and the system abstains even though the answer exists. That is the vocabulary gap that embeddings and hybrid search address.

Evaluate it

Create labelled questions: each has the document it should come from, or None when the correct behaviour is to abstain. Measure retrieval hits, correct abstentions and false abstentions.

Python Evaluation of retrieval and abstention

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sweep the threshold and watch the tradeoff: raising it removes answers to unanswerable questions but starts refusing real ones. Pick the value that fits your product (a medical bot should abstain more often than a casual FAQ bot).

Ideas to extend it

  1. Replace TF-IDF with an embedding model and merge both rankings (hybrid search).
  2. Add a reranker over the top 20 candidates.
  3. Chunk by paragraph with overlap and keep titles as metadata.
  4. Check faithfulness: does every sentence of the answer appear in a cited passage?
  5. Put the whole thing in the evaluation harness and gate changes on it.
  6. Treat retrieved text as untrusted (safety).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should a RAG system be able to abstain?

    Without abstention the model fills gaps from memory, which is where hallucinations come from.
  2. 2. What happens if you set the retrieval score threshold too high?

    A high threshold filters out true matches along with noise, raising false abstentions.
  3. 3. Which label should an unanswerable test question carry?

    Unanswerable questions test abstention, which must be measured explicitly.
  4. 4. 'When will I get my money back?' is wrongly refused because it shares no words with the refunds chunk. Which fix targets this?

    The question and chunk share few words, so a meaning-based retriever helps.

Summary

  • The pipeline is chunk, retrieve, decide to answer or abstain, build a cited prompt, generate.
  • The abstain threshold trades false refusals against answers to unanswerable questions; tune it on labelled data.
  • Lexical retrieval misses paraphrases; embeddings and hybrid search close that gap.
  • Evaluate retrieval and abstention separately from answer quality.

Next: Project: A Tiny Language Model