Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 63 ยท Text Classification

Multi-class and Multi-label Classification

Understand multi-class and multi-label text classification, build one-vs-rest models with MultiLabelBinarizer, choose thresholds and score them with the right metrics.

So far most of our examples had two answers (spam or not spam) or several answers with exactly one correct choice (sports, tech or food). Real text is messier. A news article can be about politics and economy at once. A support ticket can mention billing and login. A film can be a comedy and a romance. Forcing such documents into one box throws information away.

This lesson separates three problem types, shows how scikit-learn handles each, and covers the part that is new in the multi-label case: turning label sets into a matrix, choosing thresholds, and scoring predictions that can be partly right.

Three kinds of problems

How many labels per text?

Binary Two classes, one is correct. Spam or not.
Multi-class Many classes, exactly one is correct.
Multi-label Many classes, any number can be correct.

The number of labels per document decides the model setup.

1 / 2

In multi-class classification the model scores every class and picks the best one. With logistic regression this uses the softmax function, which turns the scores into probabilities that sum to 1. In multi-label classification the labels are independent questions: "is it sports?", "is it tech?", and so on. The probabilities do not sum to 1, and zero, one or several labels can be switched on.

Multi-class: one label per document

A linear model can handle several classes in two ways. One is a true multinomial model (softmax). The other is one-vs-rest (OvR): train one binary classifier per class ("this class versus all the others") and pick the class whose classifier is most confident. Scikit-learn's LogisticRegression uses the multinomial form for several classes by default, while LinearSVC uses one-vs-rest internally, and OneVsRestClassifier wraps any binary classifier in this scheme.

Python Multi-class: softmax and one-vs-rest

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three score between 0.944 and 0.958 on the 72 test documents, a difference of one document, so none is clearly better. For the mixed query (columns food, sports, tech), softmax distributes probability over the classes (0.124, 0.296 and 0.580) and the numbers add up to 1; the one-vs-rest wrapper trained three separate binary models (one per class), and decision_function returns one score per class for every test document, a 72 by 3 matrix. In multi-class prediction you take the largest of those scores with argmax.

Multi-label: label sets

For multi-label data, each document carries a list of labels. Models need a numeric target, so we turn the lists into a binary indicator matrix with MultiLabelBinarizer: one column per label, with 1 where the label applies. Then OneVsRestClassifier trains one independent logistic regression per column, and each produces its own probability. The threshold turns probabilities into yes/no decisions, by default 0.5.

Training one classifier per label ignores correlations between labels, such as "food" often co-occurring with "health". It is a strong baseline anyway. Methods that model label dependence (classifier chains, neural networks with a sigmoid output layer) can help when correlations are strong.

Our synthetic documents carry one or two of four topics (sports, tech, food, health). Each document has two words per true topic, three words from random topics as noise, and filler words, so the task is not trivial. The metrics are different from single-label ones:

  • Exact match (subset accuracy): the share of documents whose entire label set is right. Harsh.
  • Hamming loss: the fraction of individual label decisions that are wrong. Lower is better.
  • Micro F1: pool all label decisions, then compute F1 (dominated by frequent labels).
  • Macro F1: compute F1 for each label, then average (every label counts the same).
  • Samples F1: compute F1 per document, then average.
Python Multi-label classification with MultiLabelBinarizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The binarizer turned each tag list into a row of four 0/1 columns (alphabetical: food, health, sports, tech). Look at the scores together. About half of the test documents (0.500) get their complete label set exactly right, yet only 17.2% of single label decisions are wrong (hamming loss 0.172) and micro F1 is 0.768. All four per-label F1 scores are close to each other (0.76 to 0.78), so macro F1 equals the micro value here. These numbers measure the same predictions from different angles: exact match is the strictest, and the others give partial credit. Pick the one that matches what users experience, and report more than one.

The query contains goal (sports) and server, update (tech), but the model assigns tech 0.77 and sports only 0.18, so with the default threshold the predicted labels are ['tech'] only. The sports evidence was weaker. That raises the next question.

Choosing thresholds

The 0.5 cut is a convention. Because the labels are independent, you can choose any threshold, even a different one per label, to favor recall or precision. Lowering it assigns more labels per document; raising it makes the model more selective. The best value depends on the cost of errors: tagging articles for a search facet can be generous, while auto-routing support tickets should be conservative.

Python Threshold sweep for multi-label output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As the threshold falls from 0.5 to 0.2, the average number of labels per document climbs from 1.46 to 2.45 (the truth is 1.50), recall rises and precision drops. Micro F1 peaks at about 0.3 (0.789) while exact match keeps falling, because predicting too many labels breaks the exact set. If you need a guaranteed label for every document, a common trick is to always add the highest-scoring label even when it is below the threshold, so no document is left unlabeled. The last line shows the price of forcing exactly one label per document on this data: micro F1 is only 0.650, well below the 0.789 that thresholding reaches, because half of the documents really have two labels.

Practical advice

  • Decide the problem type first. Ask: can a document have more than one right answer? Then use multi-label.
  • Keep a label vocabulary. MultiLabelBinarizer stores classes_; save it with your model so columns keep their meaning.
  • Check label frequency. Rare labels get few positive examples; look at per-label scores (macro F1) to see them.
  • Tune thresholds on validation data, per label if you have enough examples, never on the test set.
  • Use stratified splitting carefully. Plain train_test_split ignores label sets; for very rare labels look at iterative stratification libraries.
  • Report several metrics (exact match, hamming loss, micro and macro F1), because they answer different questions. See Evaluating Classifiers.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A news article can be tagged "politics", "economy" or both. Which setup fits?

    More than one label can be correct, so each label is its own yes/no question. Softmax would force the probabilities to sum to one and pick a single winner.
  2. 2. What does MultiLabelBinarizer produce?

    Each row is a document and each column a label, with 1 where the label applies. This is the target format that multi-label estimators expect.
  3. 3. Exact match is 0.50 but hamming loss is 0.17. How is this possible?

    A document with 1 wrong decision out of 4 still counts as a complete miss for exact match, but only as a 25% error for that document in hamming loss.
  4. 4. You lower the decision threshold from 0.5 to 0.2 in a multi-label model. What is the most likely effect?

    A lower cut-off switches on more labels, so you recover more true labels but also admit more wrong ones. The probabilities themselves are unchanged.
  5. 5. How does one-vs-rest work for a three-class problem?

    Each binary classifier answers "this class or not". For multi-class prediction you pick the most confident one; for multi-label you keep every one that passes its threshold.

Summary

  • Binary: two classes. Multi-class: many classes, exactly one correct. Multi-label: many classes, any number correct.
  • Multi-class models use softmax or one-vs-rest and predict the highest-scoring class.
  • Multi-label models use MultiLabelBinarizer to make a 0/1 target matrix, then one binary classifier per label via OneVsRestClassifier.
  • Thresholds turn probabilities into label sets; tune them on validation data to balance precision and recall.
  • Evaluate with exact match, hamming loss, and micro, macro and samples F1, because each measures something different.

Next, discover themes in unlabeled text with Topic Modeling with LDA and NMF.