Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 40 ยท Model Evaluation and Selection

Evaluation Metrics for Classification and Regression

Pick the right scikit-learn metric: confusion matrix, precision, recall, F1, ROC AUC and average precision for classifiers; MAE, RMSE and R-squared for regressors.

"How good is my model?" has no single answer. A model can have 99% accuracy and still be useless, and a model with modest accuracy can be exactly what you need. The metric you choose defines what "good" means, so choosing it is a modelling decision, not a formality.

scikit-learn collects metrics in sklearn.metrics. Most take y_true first and the predictions second. This lesson walks through the ones you will use daily for classification and regression, and shows how to plug them into cross-validation with the scoring argument.

The confusion matrix

For a binary classifier every prediction is one of four things: a true positive (TP), false positive (FP), true negative (TN) or false negative (FN). The confusion matrix counts them. Almost every classification metric is a ratio of these four numbers.

Python Confusion matrix and the basic metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read them as questions:

  • Accuracy: what fraction of all predictions were right?
  • Precision: when the model says "positive", how often is it right? (Cost of false alarms.)
  • Recall (sensitivity): of all real positives, how many did we find? (Cost of misses.)
  • F1: the harmonic mean of precision and recall; it is low if either is low.

Which error hurts more?

False alarm Spam filter hides a real email.
Miss Screening overlooks a disease.
Trade-off Raising one usually lowers the other.

Choose precision or recall by the cost of each mistake.

1 / 2

Why accuracy can lie

If 99% of transactions are legitimate, a "model" that always answers "legitimate" scores 99% accuracy and catches zero fraud. On imbalanced data use precision, recall, F1, balanced accuracy or the ranking metrics below.

Python The accuracy trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A DummyClassifier is a useful baseline: any real model has to beat it on the metric you care about.

Scores, thresholds and ROC AUC

Most classifiers can output a probability (predict_proba) or a score (decision_function). predict simply cuts that at 0.5. Metrics that look at the ranking of scores, independent of any threshold, are:

  • ROC AUC: the probability that a random positive gets a higher score than a random negative. 0.5 is chance, 1.0 is perfect.
  • Average precision (area under the precision-recall curve): preferred when positives are rare.
Python ROC AUC, average precision and a threshold sweep

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As the threshold rises, precision usually goes up and recall goes down. Picking the operating point is covered in calibration and thresholds.

Multiclass averaging

With more than two classes, precision_score, recall_score and f1_score need an average argument:

  • macro: compute per class, then average (every class counts equally).
  • weighted: average weighted by class size.
  • micro: pool all decisions (for single-label problems it equals accuracy).

Regression metrics

For numeric targets the error is y_true - y_pred, and the metrics summarize it.

Python MAE, RMSE, R-squared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • MAE is the average absolute error, in the units of the target, and is robust to outliers.
  • RMSE squares errors before averaging, so large misses hurt more.
  • R-squared compares the model with always predicting the mean: 1 is perfect, 0 is no better than the mean, negative is worse.

Metrics inside cross-validation

scoring accepts a string name. Metrics where lower is better are negated (neg_mean_absolute_error) so that "higher is better" always holds and tools like grid search can maximise.

Python
from sklearn.model_selection import cross_val_score
cross_val_score(model, X, y, cv=5, scoring="f1")
cross_val_score(reg, X, y, cv=5, scoring="neg_root_mean_squared_error")

Use sklearn.metrics.get_scorer_names() to list every valid string, and make_scorer to wrap your own function.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A fraud detector must catch as many frauds as possible. Which metric should you watch most closely?

    Recall is the share of real positives that were found. With rare fraud, accuracy is dominated by the majority class.
  2. 2. What does precision measure?

    Precision is the fraction of predicted positives that are truly positive.
  3. 3. Why is the scoring string neg_mean_absolute_error negative?

    Search tools pick the highest score, so losses are negated to keep 'higher is better'.
  4. 4. A ROC AUC of 0.5 means the model

    AUC is the probability a random positive outranks a random negative; 0.5 is random ordering.

Summary

  • The confusion matrix underlies precision, recall, F1 and accuracy; choose by the cost of each error type.
  • Accuracy misleads on imbalanced data; compare against a DummyClassifier baseline.
  • ROC AUC and average precision judge scores independent of a threshold.
  • Multiclass metrics need average= (macro, weighted, micro).
  • Regression: MAE for typical error, RMSE to punish big misses, R-squared for relative fit.
  • Use scoring= strings in cross-validation; losses appear negated.

Next: Hyperparameter Tuning