Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 36 ยท Projects and Practice

Project: A Support FAQ Search Engine

Build a complete FAQ search engine step by step: embed questions, add BM25 hybrid retrieval, a no-answer threshold and an evaluation run, all in Python.

In this project you build the retrieval half of a support assistant: a FAQ search engine that takes a customer's message and returns the best matching FAQ entries, or says that it found none. There is no language model in it, which is intentional. Many support tools should start here: matching a question to a vetted answer is cheaper, faster and safer than generating text, and the same retrieval engine later feeds a full RAG assistant.

You will apply almost everything from this chapter of the course: top-k and thresholds, BM25, hybrid fusion, and retrieval metrics.

Stand-in notice: the "dense" half uses TF-IDF followed by a truncated SVD (LSA) from scikit-learn, because a neural embedding model cannot be downloaded in the browser. The architecture is identical to a production system; to upgrade it, replace the embed function with calls to a real embedding model and keep everything else.

Requirements

  1. Index 12 FAQ entries (question plus answer).
  2. Search by meaning and by exact keywords (hybrid).
  3. Return the top 3 with scores, but return nothing when the best score is too low.
  4. Evaluate on labelled customer messages and report recall@1, recall@3 and MRR.

Step 1: the data and a dense baseline

Index each FAQ as question + answer, because customers phrase things like either one. First a baseline with only the LSA vectors.

Python Step 1: FAQ data and dense search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inspect the output: where is the right FAQ ranked, and are the scores for wrong results close to the right ones? Pure LSA has trouble with words that never co-occur in a tiny corpus, which is why the next step adds keywords.

Step 2: hybrid search, a threshold and the full engine

This version adds BM25, fuses with RRF, and returns an empty list when no result looks relevant. The no-answer rule uses the dense and keyword evidence: the query must share at least one term with a result or have a decent dense similarity.

Python Step 2: hybrid engine with a no-answer rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The engine is about fifty lines. Notice the design: BM25 and LSA each propose candidates, RRF merges them, and a separate decision rule decides whether to answer at all. Keep "ranking" and "should we answer?" as separate concerns, as the observability lesson recommends.

Step 3: evaluate it

A search engine you cannot measure is a guess. Write 15 or more labelled customer messages (different wording from the FAQ questions, with some out-of-scope ones) and compute recall@1, recall@3 and MRR, plus the correct-refusal rate for out-of-scope messages. Below is the evaluation harness; to keep it short, it re-creates the engine on a smaller FAQ so you can focus on the loop. Replace the data with your own.

Python Step 3: evaluation harness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is the most useful output: a list of failing queries is your to-do list. For each, decide whether the fix is content (add a FAQ), vocabulary (synonyms, hybrid, better embeddings), or threshold.

Stretch goals

  1. Replace the LSA embed function with a real embedding model if you run this locally with sentence-transformers, then re-run the evaluation and compare.
  2. Add metadata filters such as product and language (filtering lesson).
  3. Add a re-ranker stage (re-ranking) and measure MRR before and after.
  4. Log every query with its top score, and review the lowest ones weekly.
  5. Feed the top result's answer to a language model only when the question needs rephrasing; otherwise return the FAQ answer verbatim.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why index the FAQ question together with its answer?

    Both parts contribute vocabulary that real queries may match.
  2. 2. Why keep the 'should we answer?' rule separate from ranking?

    Nearest neighbours always exist; a separate rule decides whether any of them is good enough.
  3. 3. Which evaluation number checks the no-answer behaviour?

    Out-of-scope messages should produce no answer; measure how often they do.
  4. 4. A message fails in evaluation. What is the first useful step?

    Each failure points to a specific fix; deleting it hides the problem.

Summary

  • A FAQ search engine is retrieval without generation: cheap, fast and safe.
  • Hybrid (BM25 plus dense) with RRF handles both keywords and paraphrase.
  • A separate no-answer rule prevents confident wrong results.
  • An evaluation harness with in-scope and out-of-scope messages makes improvement measurable.

Next: Project: A Documentation Assistant with Evaluation