Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 63 ยท Projects

Project: Rule and Retrieval Chatbot

Build a customer-support chatbot in Python without a language model: regex intents with slots, TF-IDF retrieval over an FAQ, a safe fallback and tests.

Large language models can chat about anything, but plenty of real chatbots do not need one. A support bot for a shop, a clinic or a bank has a limited job: answer common questions correctly, collect a few details, and hand over to a human when it is unsure. For that job, a bot built from rules and retrieval is fast, cheap, predictable and easy to test, and it never makes up an opening hour.

In this project you build such a bot for an imaginary coffee shop called Bean There. You combine three techniques: regular expressions for intents that need exact details (like an order number), TF-IDF retrieval over a list of FAQ entries for everything else, and an honest fallback when nothing fits. Then you test it like any other piece of software. You will need regex extraction, TF-IDF and document similarity.

The plan

Three layers, in order

Rules Exact patterns with details to extract.
Retrieval Find the closest FAQ answer.
Fallback Say "I do not know" and offer help.

The first layer that is confident wins.

1 / 2

Step 1: rules with slots

An intent is what the user wants (check an order, ask about opening hours). A slot is a detail the intent needs (the order number). Regular expressions are a good fit when the user's wording is predictable and you must pull out exact values.

Each rule below pairs a pattern with a function that builds the reply from the match. Rules are tried in order, so the specific ones come first. If no rule matches, the function returns None, which will let later layers take over.

Python Intent rules with slot extraction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output as a test report. Greetings, order lookups with a number (even written as #50977) and unknown orders are handled. "Where is my order" has no number, so the bot asks for it. Three things deserve attention:

  • Order matters. "thanks a lot, bye" matches both thanks and goodbye, and only the first rule in the list answers. That is fine here, but when two rules can fire you must decide on purpose which one wins.
  • Rules do not generalize. "do you have oat milk" matches nothing, because nobody wrote a rule for it. Writing a rule for every question does not scale, which is why the next layer exists.
  • A pattern can be too loose or too strict. "my phone number is 555-1234" correctly does not trigger the order rule, because the rule requires the word order before the digits. Without that, any number in any message would be taken for an order number.

None means "this layer has no answer", and later layers take over.

Step 2: retrieval over an FAQ

Most questions are not worth a custom rule. Instead, keep a list of questions with their approved answers, turn every FAQ question into a TF-IDF vector, and answer a new message with the answer of the most similar FAQ question. This is the same cosine-similarity search you used for documents, applied to short questions.

Retrieval needs a confidence threshold. A similarity of 0.9 means "this is nearly the same question"; 0.05 means "nothing related". If the best match is below the threshold, the bot must not guess.

Python FAQ retrieval: one question versus several examples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two changes make this version work better than a first attempt would. The tokenizer drops common words (do, you, can) and strips simple endings, so opening, open and hours can match. And every answer has several example questions instead of one, so there are more ways to match how people really ask. The test set holds ten messages we wrote after the FAQ, three of them out of scope (laptop, bitcoin, the capital of France), where the correct behavior is no answer.

The comparison is dramatic: with a single question per answer the retriever is right on 50 percent of the messages, and with several examples it reaches 90 percent. "when do you open?" shows why. Against the single question "What are your opening hours?" it shares no useful word, but against the extra example "When do you open?" it matches perfectly.

The one remaining mistake is the most instructive. "do you sell bitcoin" is matched to the coffee-beans answer with similarity 0.54, only because both contain sell. The threshold cannot fix it: raise it high enough to reject this match and you would also reject correct answers such as "are dogs welcome inside" (0.63). Bag-of-words similarity knows nothing about meaning. The standard remedy is cheap: teach the bot what out of scope looks like with a few example questions that map to "no answer". We do that in the final step.

Step 3: put it together and test it

Now combine the layers in one class. The reply method follows the plan: rules first, then retrieval above the threshold, then a fallback that is honest and useful. The fallback names what the bot can do and offers a human, instead of repeating "I don't understand".

The bot also keeps a little context: it remembers when it asked for an order number, so that the user's next message, just "48213", is understood as the answer. Without that, a bare number would reach the retrieval layer and fall through to the fallback. Small bits of state like this make a bot feel far more natural.

Finally, we write tests. A chatbot is software, and every behavior you promised should be checked, especially the ones that protect users: no invented answers, correct slot extraction, safe handling of empty input.

Python A complete chatbot with context and tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The transcript shows the whole design at work. "hi" and "thanks, bye" are answered by rules. "when do you open?" is answered from the FAQ. "where is my order" makes the bot ask a question, and the bare number "48213" in the next message is understood because the bot remembers what it asked. Out-of-scope "do you sell bitcoin" now reaches the fallback instead of the coffee-beans answer, thanks to the out-of-scope examples (Do you sell electronics?, Do you sell cars? ...), which pull the bitcoin question toward the "no answer" group.

The nine assertions are the regression suite. Each one protects a promise: right answers for common questions, exact slot extraction, a safe answer for unknown orders, no invented answers for out-of-scope input (two tests), correct handling of context, empty input and a request for a human. Whenever a user finds a bug, first write a failing test for it, then fix the bot.

Notice that the out-of-scope list is a patch for one kind of failure and is itself a limited list. A different out-of-scope question could still be matched to the wrong FAQ entry. That is why you also measure on real logs and keep the threshold conservative: a bot that says "I am not sure" too often is annoying, but one that answers wrongly is worse.

Step 4: evaluate and improve

A bot should be measured on real messages, not on the ones its author thought of. Collect conversation logs (with permission, and with personal details removed), label what the right behavior should have been, and track:

  • Answer accuracy: when the bot answered, was the FAQ entry correct?
  • Fallback rate: how often does the bot say it does not know? Too high means missing knowledge; too low may mean wrong answers slipping through.
  • False answers on out-of-scope input: messages such as "can you recommend a laptop" must hit the fallback. These errors are the dangerous ones, and the threshold controls them.
  • Handover rate and what users did after a fallback.

When you read the failures, they tend to fall into three groups: a missing FAQ entry (add it), a paraphrase the retrieval missed (add the new wording to the entry as an extra example question), and a wrong rule (fix the pattern and add a test). Adding several example questions per answer is the cheapest big improvement for retrieval bots.

You can also raise the quality of matching. Replace TF-IDF with sentence embeddings so that "do you take credit cards" matches "Can I pay by card" by meaning, or use retrieval and RAG to let a language model phrase the answer from the retrieved text. Keep the rule layer and the fallback even then: they are what makes the bot dependable.

Never let a support bot guess about money, medicine, safety or legal questions. For those, a deliberate rule that routes to a human is better than any statistical answer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a slot in a chatbot?

    An intent often needs specific values (slots) to act on, like the order number for an order status request.
  2. 2. Why does the retrieval layer need a confidence threshold?

    The most similar FAQ question always exists, even for nonsense input. The threshold decides when that match is too weak to trust.
  3. 3. Why check the order rule before the generic "my order" rule?

    A message with an order number should be answered at once. If the general rule came first, it would ask for the number the user already gave.
  4. 4. What is a good way to improve a retrieval bot that misses paraphrases?

    More example wordings per answer give the retrieval step more chances to match how real users ask.
  5. 5. Why remember that the bot just asked for an order number?

    A message such as "48213" means nothing alone. Small dialogue state lets the bot interpret it as the answer to its own question.

Summary

  • A rules plus retrieval bot is fast, cheap, predictable and testable, a good fit for narrow support tasks.
  • Regex rules handle intents with exact slots; order specific rules before general ones.
  • TF-IDF retrieval over FAQ questions answers the rest, but only above a confidence threshold.
  • An honest fallback that lists what the bot can do, plus a path to a human, beats a wrong guess.
  • A little dialogue state makes follow-up messages work.
  • Test every promised behavior, measure on real logs, and improve by adding rules, FAQ entries and example wordings.

You have now built five complete NLP applications. Time to practice the building blocks with Practice Exercises with Solutions.