Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 111 ยท Classification

Multiclass and Multilabel Classification

Handle more than two classes or several labels per sample: one-vs-rest, one-vs-one, softmax, macro, micro and weighted averages, and MultiOutputClassifier.

So far the examples had two classes: spam or not, pass or fail. Real problems often have more. A photo shows one of ten digits. A support ticket belongs to one of twelve categories. A movie can be tagged comedy and romance and french all at once. These are different situations with different tools:

  • Binary: two classes, one is chosen.
  • Multiclass: three or more classes, exactly one is chosen per sample.
  • Multilabel: several yes/no labels, any number of which can be true per sample.

This lesson shows how binary classifiers are extended to many classes (one-vs-rest, one-vs-one, softmax), how to average metrics across classes, and how to predict several labels at once. It builds on Logistic Regression and Classification Metrics.

Which problem do you have?

Binary Two classes. Spam or not spam.
Multiclass Many classes, exactly one per sample.
Multilabel Many yes/no tags, any number per sample.
1 / 3

Multiclass with scikit-learn

Many scikit-learn classifiers handle multiclass data out of the box: you simply pass labels with more than two distinct values. LogisticRegression does it with softmax regression (also called multinomial logistic regression): the model computes one score per class and the softmax function turns the scores into probabilities that sum to 1. The class with the highest probability wins.

Python Softmax: from scores to probabilities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here is a three-class problem using the iris flowers. predict_proba now has one column per class.

Python Three classes with LogisticRegression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One-vs-rest and one-vs-one

Some models are inherently binary (for example LinearSVC and SVC in Support Vector Machines, or any model you have written yourself). Two wrapper strategies turn them into multiclass classifiers:

  • One-vs-rest (OvR, also one-vs-all): train one classifier per class, "this class against everything else". For K classes that is K models. At prediction time the model with the most confident "yes" wins. It is cheap and the default for many linear models.
  • One-vs-one (OvO): train one classifier for every pair of classes, K * (K - 1) / 2 models, and let them vote. Each model sees only two classes' data, so each is faster to train, which helps for algorithms that scale badly with data size.
Python OneVsRestClassifier and OneVsOneClassifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 10 classes OvR trains 10 models and OvO trains 45. Accuracy is usually similar; choose by speed and by what the base algorithm supports.

Metrics for many classes: macro, micro and weighted

With more than two classes, precision, recall and F1 exist per class. To get a single number you average them, and the average you pick changes the story:

  • Macro: compute the metric for each class, then take the plain mean. Every class counts equally, so a poor result on a rare class pulls the score down. Use it when all classes matter equally.
  • Weighted: the mean weighted by each class's number of true samples. Close to accuracy when classes are imbalanced.
  • Micro: pool all individual decisions (total TP, FP, FN) before computing. For single-label multiclass problems micro precision, micro recall, micro F1 and accuracy are all the same number.
Python Macro, micro and weighted averages on imbalanced classes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Micro F1 equals accuracy here, as promised. Macro F1 is much lower because the two smaller classes are handled poorly (recall below 0.3), and weighted F1 sits near accuracy because the big class dominates. Report the per-class numbers whenever classes are unequal in size or importance.

Note: In scikit-learn, binary metrics default to average="binary" for label 1. For three or more classes you must choose average= yourself ("macro", "micro", "weighted") or pass None to get one value per class.

ROC curves and AUC for many classes

A ROC curve needs one score and a yes/no question, so for three or more classes you ask several yes/no questions and combine the answers:

  • One-vs-rest (OvR): for each class, treat "this class" as positive and everything else as negative, and use the model's probability for that class as the score. You get one ROC curve and one AUC per class. This is the usual choice and answers "how well can the model pick out class k from all the others?"
  • One-vs-one (OvO): compute the AUC for every pair of classes, using only the rows of those two classes, and average the results. It is less affected by class imbalance, but there are K(K-1)/2 pairs.

The per-class AUCs are combined by macro averaging (plain mean, all classes count equally) or weighted averaging (weighted by class frequency); roc_auc_score(y, probabilities, multi_class="ovr", average="macro") does both steps. Note that the model must supply probabilities for all classes that sum to 1, with the columns in the order of model.classes_.

Python One-vs-rest and one-vs-one AUC on three classes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each class has its own curve and its own AUC. Here the rare class (10% of the data) is actually the easiest to separate (0.949) and the most common one the hardest (0.859): rarity alone does not make a class hard, it depends on how the classes overlap. The macro average (0.911) counts the three classes equally, while the weighted average (0.888) leans toward the common class and therefore comes out lower in this example. Always look at the per-class numbers, not only the average. As with binary problems, a high ROC AUC can flatter a model on rare classes; check precision-recall curves as well.

Two more facts about ROC curves: the curve you get from a model's scores is a non-parametric estimate (it just follows the data), while a parametric ROC curve assumes a distribution for the scores (for example two normal distributions, the "binormal" model) and fits its two parameters; the second is smoother and needs fewer samples, but only as good as its assumption. And because ROC compares ranks, it says nothing about whether the probabilities are calibrated (calibration).

Multilabel: several tags per sample

In multilabel problems the target y is a matrix with one column per label, filled with 0s and 1s. A sample can have none, one or many labels on. Because the labels are not mutually exclusive, we do not use softmax (which forces probabilities to sum to 1). Instead each label gets its own independent yes/no model.

MultiOutputClassifier wraps any binary classifier and trains one copy per column. Some estimators, such as RandomForestClassifier and KNeighborsClassifier, accept a 2D y natively.

Python MultiOutputClassifier for three tags

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multilabel metrics

Evaluating several labels needs care, because "exactly right" is a harsh standard:

  • Subset accuracy (accuracy_score on a label matrix): a sample counts only if every label is correct. Very strict.
  • Hamming loss: the fraction of individual label slots that are wrong. Gentler.
  • Per-label precision, recall and F1, then macro, micro or samples averaging (average="samples" computes the metric per sample, then averages).
Python Evaluating multilabel predictions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Subset accuracy is clearly lower than the label-wise scores, because one missed tag fails the whole sample. Pick the metric that matches how the predictions will be used. If a recommendation system shows the top few tags, per-label precision matters most; if a document must be routed to every right department, recall does.

One caveat: MultiOutputClassifier treats labels as independent. When labels depend on each other (comedy and romance often co-occur), ClassifierChain feeds earlier predictions into later models to capture that. The basics above are the right starting point, and you can reach for chains when the independent version is not enough.

With multiclass and multilabel covered, the next lesson tackles a problem that distorts almost every metric in this chapter, rare classes: Imbalanced Classes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A news article can carry the tags 'politics', 'economy' and 'europe' at the same time. What kind of problem is this?

    Several labels can be true for the same sample, so it is multilabel. Multiclass means exactly one label per sample.
  2. 2. With 6 classes, how many binary classifiers does one-vs-one train?

    One-vs-one trains one model per pair of classes: 6 * 5 / 2 = 15. One-vs-rest would train 6.
  3. 3. Which average treats every class equally, however rare it is?

    Macro averaging computes the metric per class and takes the plain mean, so a poorly handled rare class pulls the score down.
  4. 4. Why is subset accuracy usually much lower than hamming-based or per-label scores in multilabel problems?

    One wrong label among many makes the whole row wrong, which makes subset accuracy a very strict measure.

Summary

  • Binary has two classes, multiclass has several with one per sample, multilabel allows any number of tags per sample.
  • Softmax regression scores all classes at once; one-vs-rest trains K binary models, one-vs-one trains K(K-1)/2.
  • For several classes pick an averaging scheme: macro (classes equal), weighted (by size), micro (pool decisions; equals accuracy in single-label multiclass).
  • Multilabel uses a 0/1 label matrix and independent models, for example MultiOutputClassifier; ClassifierChain models label dependence.
  • Judge multilabel results with several views: subset accuracy (strict), hamming loss, per-label and samples-averaged F1.

Next up: Imbalanced Classes.