Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 38 ยท Projects and Practice

Project: A Support-Ticket Classifier, Data to Demo

Build a complete text-classification project: data splits, tokenization, a baseline, fine-tuning with Trainer, error analysis, saving and a demo, with a runnable numpy baseline.

This project ties the course together. The task: route incoming customer-support tickets to one of three queues (billing, shipping, account). You will plan the work, build a quick baseline you can actually run here, write the real fine-tuning code, evaluate it properly, and ship it.

Fine-tuning a Transformer needs transformers, datasets, PyTorch and ideally a GPU, none of which run on this site. The real code is shown read-only; the output of those steps is not shown because it depends on your data and hardware. The runnable parts are a stand-in baseline in numpy (a bag-of-words softmax classifier on a tiny synthetic dataset) that follows the same project steps.

Step 0: plan before you code

Write down, before any training:

  • The decision the model supports. Wrong routing costs a few minutes; a wrong refund decision costs money. This sets how much accuracy you need and which errors matter.
  • The labels, with a one-line definition each and rules for tickets that fit two queues. Ambiguous labels cap your accuracy no matter the model.
  • The metric. Here: macro F1 (every queue matters equally), plus per-class recall for billing.
  • A golden test set of real tickets that is never used for training or tuning (see error analysis).
  • A baseline. A trivial model tells you how much the Transformer adds.
flowchart LR
  A["Define labels and metric"] --> B["Split train / validation / test"]
  B --> C["Baseline: keyword or bag of words"]
  C --> D["Fine-tune a small encoder"]
  D --> E["Evaluate and analyse errors"]
  E --> F["Save, push, serve or demo"]
  E -->|"fix data or labels"| B

The loop back from evaluation to data is where most improvement comes from.

Step 1: data and splits

Split before looking at test results, stratified by label, and deduplicate first so the same ticket cannot land in both train and test. Real code (read-only):

Python
from datasets import load_dataset

raw = load_dataset("csv", data_files="tickets.csv")["train"]
raw = raw.class_encode_column("label")                       # str -> ClassLabel
split = raw.train_test_split(test_size=0.2, seed=42, stratify_by_column="label")
valtest = split["test"].train_test_split(test_size=0.5, seed=42, stratify_by_column="label")
ds = {"train": split["train"], "validation": valtest["train"], "test": valtest["test"]}
labels = raw.features["label"].names
id2label = dict(enumerate(labels)); label2id = {l: i for i, l in id2label.items()}

Step 2: a baseline you can run

A bag-of-words model with a linear classifier is fast and surprisingly strong when the classes use different vocabulary. The code below is complete: it builds a tiny synthetic dataset, vectorizes it, trains softmax regression with gradient descent and reports per-class results.

Python Baseline: bag of words + softmax regression (numpy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real project you would run this kind of baseline on your actual tickets first. If it already reaches the accuracy you need, you may not need a Transformer at all; if it plateaus because of word-order, negation or paraphrase, that is exactly where a pre-trained encoder helps.

Step 3: fine-tune a small encoder

Pick a small, fast model first (a DistilBERT- or MiniLM-sized encoder). Real code (read-only), using the pieces from earlier lessons (tokenizing with map, dynamic padding, Trainer):

Python
import numpy as np, evaluate
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DataCollatorWithPadding, Trainer, TrainingArguments, set_seed)

set_seed(42)
name = "distilbert/distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
enc = {k: v.map(lambda b: tok(b["text"], truncation=True, max_length=256), batched=True)
       for k, v in ds.items()}

model = AutoModelForSequenceClassification.from_pretrained(
    name, num_labels=len(labels), id2label=id2label, label2id=label2id)

f1 = evaluate.load("f1")
def compute_metrics(p):
    preds = np.argmax(p.predictions, axis=-1)
    return f1.compute(predictions=preds, references=p.label_ids, average="macro")

args = TrainingArguments(
    output_dir="out", learning_rate=2e-5, num_train_epochs=3,
    per_device_train_batch_size=16, per_device_eval_batch_size=32,
    eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True,
    metric_for_best_model="f1", weight_decay=0.01, bf16=True, report_to="none")

trainer = Trainer(model=model, args=args, train_dataset=enc["train"],
                  eval_dataset=enc["validation"], data_collator=DataCollatorWithPadding(tok),
                  compute_metrics=compute_metrics)
trainer.train()
print(trainer.evaluate(enc["test"]))        # touch the test set once, at the end

Compare the test macro F1 with your baseline. Report both. If the gain is small, spend your time on data quality instead of a bigger model.

Step 4: error analysis and iteration

Look at the confident mistakes, slice by ticket length and source, and read the confusion matrix. Typical findings and fixes:

Finding Likely fix
Two classes confused in both directions Merge them, or sharpen label definitions
Errors on long tickets Raise max_length, or classify the first and last parts
Wrong labels among confident errors Clean the labels, retrain
Rare class has low recall Collect more examples; class weights; threshold tuning

Fix one thing at a time and rerun on the validation set. Touch the test set only for the final number.

Step 5: ship

Save the model with its tokenizer, write a model card, push privately, then pick a delivery route: a Gradio Space for a quick demo (Gradio and Spaces) or an API (FastAPI and batching). See saving, pushing and model cards. In production, log predictions and confidence, send low-confidence tickets to a human, and re-evaluate on fresh labelled tickets every month, because language and products drift.

Project checklist

  • Labels defined, golden test set fixed, metric chosen.
  • Baseline measured and recorded.
  • Fine-tuned model compared on the same splits, with seed noted.
  • Error analysis written down (top three failure types).
  • Model, tokenizer and card saved; label maps present in config.json.
  • A monitoring and retraining plan exists.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why build a simple baseline before fine-tuning a Transformer?

    Without a baseline you cannot tell whether the extra cost and complexity are justified.
  2. 2. When should the test set be used?

    Tuning on the test set leaks information and makes the final score optimistic.
  3. 3. Why is macro F1 a good headline metric for three equally important queues?

    Macro averaging gives each class the same weight regardless of its size.
  4. 4. You find confident errors where the label itself looks wrong. What next?

    Noisy labels cap accuracy; cleaning them is usually the cheapest improvement.

Summary

  • Plan labels, metric, splits and a golden test set before training.
  • Run a cheap baseline first, then fine-tune a small encoder with Trainer and compare on the same splits.
  • Use error analysis to decide whether the fix is data, labels, length or model.
  • Ship with a card, a demo or API, confidence-based fallback and ongoing monitoring.

Next: Project: semantic search with embeddings