Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 38 ยท Evaluation and Metrics

Error Analysis, Calibration and Benchmarks

Go beyond one accuracy number: slice errors, read confusion matrices, measure calibration with ECE, and learn how benchmark scores for language models are produced and misread.

A single score such as 91% accuracy tells you that a model is good on average. It does not tell you where it fails, whether its confidence can be trusted, or whether the number will hold up on real traffic. Error analysis is the habit of looking at the mistakes themselves, grouped in useful ways, before you decide what to fix: more data, a different label scheme, a bigger model or a threshold change.

This lesson covers three tools: slicing and confusion matrices, calibration, and a short guide to reading public benchmarks.

The Hugging Face tools used for scoring (evaluate, Trainer.predict) are not available on this site. The runnable examples use numpy on small, seeded, synthetic predictions so the logic is real and the numbers are reproducible; the model outputs are stand-ins.

Start with the mistakes, not the average

After trainer.predict(test_dataset) you have logits and labels. Keep them together with the original text and any metadata, then look at the wrong rows. The real workflow looks like this (read-only):

Python
preds = trainer.predict(tokenized["test"])
probs = softmax(preds.predictions, axis=-1)
df = raw["test"].to_pandas()
df["pred"] = probs.argmax(-1)
df["conf"] = probs.max(-1)
errors = df[df["pred"] != df["label"]].sort_values("conf", ascending=False)
errors.head(30)

The most informative rows are the confident mistakes: the model was sure and wrong. They usually reveal one of four things: a wrong label in the dataset, an ambiguous example, a missing kind of training data, or a shortcut the model learned (for example, "contains the word refund means complaint").

Slicing: where does the model fail?

Average metrics hide weak groups. A slice is any subset you can define: text length bucket, language, product category, presence of negation, source of the data. Compute the metric per slice and compare it with the overall value.

Python Slice-based evaluation (numpy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overall number looks healthy while the long slice is clearly worse. In a real project that is your cue to inspect long inputs: are they being truncated at max_length? If yes, the fix is a long-context model or a chunking strategy, not more epochs.

Confusion matrices

For classification, a confusion matrix counts how often each true class is predicted as each class. Rows are the truth, columns are the predictions, so the diagonal is correct and every off-diagonal cell is a specific kind of mistake. It tells you which classes get confused, which precision and recall alone do not (see precision, recall and F1).

Python Confusion matrix from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading it row by row answers "what happens to real shipping emails?"; reading it column by column answers "when the model says billing, what was it really?".

Calibration: can you trust the confidence?

A model is calibrated if, among predictions made with 80% confidence, about 80% are right. Neural networks are often overconfident, which matters whenever you act on the probability: auto-approve above 0.9, send to a human below 0.6, show a percentage to users.

Expected Calibration Error (ECE) measures the gap. Split predictions into confidence bins, compare average confidence with accuracy in each bin, and take the weighted average of the gaps.

Python Expected calibration error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In every bin the accuracy sits below the confidence, the signature of overconfidence. A simple, cheap repair is temperature scaling: divide the logits by a single number T > 1 fitted on a validation set. It does not change which class wins (so accuracy is unchanged) but softens the probabilities.

Reading benchmarks

Public leaderboards (GLUE and SuperGLUE for classification, MMLU-style multiple choice, code and math sets for language models) are useful for choosing a candidate model, and easy to misread:

  • Contamination. Test questions may have appeared in the pre-training data, inflating scores.
  • Prompt and format sensitivity. The same model can differ by several points with a different prompt template, few-shot count or answer-extraction rule. Compare scores only when the harness and settings match.
  • Saturation. When many models score 95%+, differences are noise.
  • Mismatch with your task. A strong average on general knowledge says little about your support tickets.

The practical answer is a small, private golden set of real examples (a few hundred is plenty), labelled carefully, never used for training, and scored the same way every time. Run it for every candidate model and every release; track slices, not only the average.

flowchart LR
  A["Model predictions"] --> B["Overall metric"]
  A --> C["Slices and confusion matrix"]
  A --> D["Calibration (ECE)"]
  C --> E["Read confident mistakes"]
  D --> F["Set thresholds"]
  E --> G["Fix data, labels or model"]

A good evaluation looks at the same predictions from several angles.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which rows of the test set are usually the most informative to read first?

    Confident mistakes often expose wrong labels, ambiguity or shortcuts the model learned.
  2. 2. In a confusion matrix with true labels on the rows, what does a large off-diagonal cell mean?

    Cell (i, j) counts items of true class i predicted as class j.
  3. 3. A model says 90% on many predictions but is right only 75% of the time on them. It is...

    Confidence exceeds accuracy, which is overconfidence; ECE measures this gap.
  4. 4. Why can two published scores for the same model on the same benchmark differ?

    Evaluation harness details can move a score by several points.

Summary

  • Look at mistakes, especially confident ones, before changing anything.
  • Slice metrics by length, language, category or source to find weak groups hidden by the average.
  • A confusion matrix shows which classes get mixed up; calibration (ECE, temperature scaling) shows whether probabilities can be trusted.
  • Treat public benchmarks as a shortlist tool, and keep a private golden set for the decisions that matter.

Next: Saving, pushing and documenting models