Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 38 ยท Evaluation and Metrics

The Evaluate Library and Metric Basics

Compute metrics with Hugging Face Evaluate: load, compute, add_batch, combine, compute_metrics for Trainer, and how a metric works inside, with a runnable mini version.

Training reduces a loss. Evaluation answers a different question: is the model good at the thing you care about? Loss is hard to interpret ("is 0.41 good?"); metrics such as accuracy, F1, BLEU or exact match have meaning for people. Hugging Face ships the evaluate library to compute them consistently, with the same implementations everyone else uses, so your numbers can be compared with papers and leaderboards.

evaluate downloads metric scripts from the Hub and is not available on this site. Plain code blocks are real, read-only code. The TryIt stand-ins implement the same interface (add_batch, compute, combine) in plain Python so you can see exactly what it does.

Loading and computing a metric

Python
import evaluate

accuracy = evaluate.load("accuracy")
result = accuracy.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 0])
print(result)            # {'accuracy': 0.75}

f1 = evaluate.load("f1")
print(f1.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 0]))
print(f1.compute(predictions=[0, 2, 1], references=[0, 1, 1], average="macro"))

Every metric has the same two arguments: predictions (what the model said) and references (the truth). Extra options, such as average="macro" for F1, are passed to compute. Each loaded metric has a description, citation and inputs_description you can print: always read the inputs description, because metrics disagree about whether they expect integers, strings, token lists or tokenized references.

Streaming over batches

On big evaluation sets you cannot keep all predictions in memory, and in distributed evaluation each process sees only a slice. The metric object solves both: call add_batch repeatedly, then compute once.

Python
metric = evaluate.load("accuracy")
for batch in eval_loader:
    logits = model(**batch).logits
    preds = logits.argmax(dim=-1)
    metric.add_batch(predictions=preds, references=batch["labels"])
print(metric.compute())

Combining metrics

evaluate.combine(["accuracy", "f1", "precision", "recall"]) returns one object whose compute returns all four numbers, so a single pass gives a full report. One caution: options that belong to a single metric (such as average="macro" for F1) are awkward to pass through a combined object, so when you need per-metric options it is often clearer to load and call each metric separately.

Using a metric with the Trainer

The Trainer calls your compute_metrics function with an EvalPrediction, a pair of arrays: model outputs (logits) and label ids. You convert logits to predictions and return a dictionary.

Python
import numpy as np, evaluate

clf = evaluate.combine(["accuracy", "f1"])

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    out = clf.compute(predictions=preds, references=labels)
    return out

Return only plain Python numbers (not arrays) so they can be logged. The Trainer prefixes each key with eval_, so "f1" is reported as eval_f1, and you can use it as metric_for_best_model="f1".

For generation tasks the logits are not the text. Use Seq2SeqTrainer with predict_with_generate=True so compute_metrics receives generated token ids, which you decode before scoring. See text generation metrics.

A metric from the inside

A metric is just a function over two lists plus some bookkeeping. This stand-in reproduces the interface: it accumulates batches, computes on request, and combine merges several.

Python Stand-in: a mini evaluate library

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The real library also validates input types, supports distributed accumulation across processes, and caches intermediate files on disk, but the contract is the same.

Picking a metric

Choose by the type of task, then by the failure you fear:

Which metric for which task?

Classification Accuracy, F1, precision, recall, ROC-AUC.
Token tagging seqeval entity-level F1.
Extractive QA Exact match and token F1.
Generation ROUGE, BLEU, BERTScore, perplexity.

Accuracy alone hides class imbalance.

1 / 2

Another common library choice is scikit-learn metrics directly (accuracy_score, classification_report); they are equivalent for many tasks. The advantage of evaluate is breadth (generation, QA, summarization, translation metrics) and sharing metric definitions through the Hub, including community "comparisons" and "measurements" such as word length or toxicity.

Common pitfalls

  • Mismatched label order. Predictions and references must use the same integer mapping.
  • Averaging. For multi-class F1 you must choose average="macro", "micro" or "weighted"; the default "binary" errors out with more than two classes.
  • Strings versus ids. Text metrics want decoded strings, not token ids; classification metrics want ids.
  • Metric on padded positions. In token tasks, drop positions whose label is -100 before scoring.
  • Evaluating on the training set. It flatters the model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What two arguments do all evaluate metrics take?

    predictions are model outputs; references are the ground truth.
  2. 2. Why use add_batch and then compute?

    It lets evaluation stream and works with distributed evaluation.
  3. 3. In the Trainer, how does a metric called f1 returned by compute_metrics appear in the logs?

    The Trainer adds the eval_ prefix to every key returned during evaluation.
  4. 4. What must you do before scoring predictions in a token classification task?

    -100 marks ignored positions such as padding and special tokens; they are not real predictions.

Summary

  • Metrics turn model outputs into human-meaningful numbers; evaluate provides shared implementations.
  • evaluate.load(name).compute(predictions=..., references=...) is the core call; add_batch streams; combine merges metrics.
  • In the Trainer, compute_metrics receives logits and labels, so apply argmax and return plain numbers.
  • Check input types, label order, averaging mode and ignored positions.
  • Pick metrics by task and by the error you most want to avoid.

Next: Classification metrics: precision, recall and F1