Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 38 ยท Projects and Practice

Project: A Support Ticket Classifier

Build a support ticket classifier end to end: write the prompt, validate JSON output, add a fallback, build a labelled test set, measure accuracy and fix the failures.

In this project you put the track together: a clear prompt, structured output, validation with a safe fallback, a labelled test set and a measured improvement. The task is realistic and small: read a customer message and return a category and a priority as JSON so a help desk can route it.

No real model runs on this site, so the "model" is a deterministic stand-in whose behaviour you can read. The pipeline around it (prompt building, parsing, validation, scoring) is exactly what you would use with a real model; to switch, replace one function. Treat the stand-in as a teaching device: a real model will make different mistakes, which is why the test set matters.

Step 1: Define the task precisely

Write the contract before the prompt:

  • Input: a customer message (plain text, up to 1,000 characters).
  • Output: JSON with category (one of billing, bug, howto, account, other) and priority (low, normal, urgent).
  • Rules: urgent only for outages, security problems or a customer unable to pay or log in. When unsure, use other and normal.

Step 2: Write the prompt

Python Build the prompt from a template

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

It uses what you learned: a role and task, an output format, rules for edge cases, two examples, a safe fallback instruction, and a delimited data block.

Step 3: Call, parse and validate

Never trust the reply. Parse it, check every field, and fall back to a safe default when anything is wrong (validating and repairing JSON).

Python The classifier with validation and fallback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three of the five replies were awkward (a code fence, a wrong-case enum, plain prose). The validation layer repaired two and fell back on the third, so the help desk never receives a malformed record.

Step 4: Build the test set

A test set is what turns "seems fine" into a number. Cover each category, both priorities, tricky wording, empty and weird input, and at least one injection attempt. Label by hand.

Python Score the classifier on labelled cases

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the failures, not only the scores. Typical findings, which a real model reproduces in different forms:

  • Priority errors are the costly ones: "The app crashes on launch for everyone" is an outage but was tagged normal and would wait in line. The rules say "urgent for outages", but the stand-in looks for keywords, not meaning.
  • "Please resend my invoice" expected low but got normal, and the same for the student-discount question: your labels and your rules may disagree. Decide which is right and make the prompt and the labels consistent.
  • The injection case passes only because the stand-in ignores instructions. Keep the case anyway; against a real model it is a real test.

Step 5: Improve, then re-measure

Change one thing at a time and rerun the whole set (see iterating and regression tests). Typical fixes in order of cost:

  1. Clarify the rule that failed ("urgent when many users are blocked or nothing loads").
  2. Add an example that is near the failure.
  3. Split the task: one call for category, one for priority, if they interfere.
  4. Use a stronger model for the hard class only (routing).

Record each change as a new prompt version with its score, and refuse any version that breaks a case that used to pass.

Step 6: Prepare for production

  • Log version, tokens, latency and fallbacks (monitoring).
  • Alert if the fallback rate rises: it means the model's format drifted.
  • Add agent feedback ("re-routed") as a signal and feed corrections back into the test set.
  • Redact personal data before sending (privacy).

Extensions

  • Add a confidence field and send low-confidence tickets to a human.
  • Support multiple languages and add test cases for each.
  • Add a summary field and judge it with a rubric (LLM-as-judge).
  • Compute a confusion matrix to see which categories get mixed up.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the pipeline validate the model's JSON instead of trusting it?

    Validation plus a safe fallback guarantees downstream systems only see well-formed records.
  2. 2. Which error type is usually costliest in this classifier?

    A missed urgent ticket waits in the queue while customers are blocked.
  3. 3. A test case fails because your label disagrees with your own prompt rules. What do you do?

    Inconsistent labels make scores meaningless; fix whichever is wrong.
  4. 4. How many things should you change between two test runs?

    Single changes make cause and effect visible.

Summary

  • Start from a precise contract (input, output enums, rules for edge cases).
  • Prompt = role/task, format, rules, examples, delimited data; wrap the call in parse, validate, fallback.
  • Measure on a labelled set that includes edge and attack cases, inspect failures, then change one thing at a time.
  • Version prompts, log fallbacks and feed production corrections back into the test set.

Next: Project: Grounded Question Answering