Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 63 ยท Text Classification

Text Classification Overview

Get the big picture of text classification: common tasks, how labels work, train/test splits, baselines and data leakage, with runnable scikit-learn code.

Sorting text into categories is the most common job in applied NLP. Is this email spam? Is this support ticket about billing or about login problems? Is this review positive? Does this sentence ask for a refund or a human agent? All of these are text classification: take a piece of text, return one label from a fixed set.

This chapter walks through the main classifiers and metrics. This first lesson gives you the map: what kinds of tasks exist, how data and labels are organized, how to split data honestly, why a baseline comes first, and the leakage mistake that makes bad models look great. Everything here builds on the representations from earlier chapters, such as bag of words and TF-IDF.

Common tasks

The mechanics are the same across tasks; only the labels change.

One idea, many tasks

Spam detection Two labels, spam or not spam.
Topic labeling Sports, politics, tech, and so on.
Intent detection What does the user want to do?
Sentiment Positive, neutral or negative.

Same pipeline, different labels.

1 / 2

The labels define the type of problem:

  • Binary: two classes (spam or ham).
  • Multi-class: several classes, exactly one per document (a news article is about sports or politics).
  • Multi-label: several tags can apply at once (a post can be both "python" and "tutorial"). You will see this in multi-class and multi-label classification.

The supervised workflow

Text classification is supervised learning: you need examples whose correct label is already known, collected from user reports, human annotators or a database column. The model learns the connection between text and label from those examples, then predicts labels for new text.

flowchart LR
    A["Labeled texts"] --> B["Split into<br/>train and test"]
    B --> C["Turn text into features<br/>(bag of words, TF-IDF)"]
    C --> D["Fit a classifier<br/>on the training part"]
    D --> E["Predict the test part"]
    E --> F["Compare with true labels<br/>(metrics)"]

The test set is touched only once, at the end.

Setting up data and splitting it

Below is a tiny spam data set of 20 short messages, ten of each label. Real projects have thousands or millions, but small data keeps every number visible.

The train/test split is the most important habit in machine learning: hold back part of the data, never let the model see it while learning, and measure on it afterwards. A model that scores perfectly on the text it studied tells you nothing about new text. Two details matter. random_state makes the split repeatable. stratify=labels keeps the class proportions the same in both parts, so a rare class does not vanish from the test set by bad luck.

Python Labeled data and a stratified split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fourteen messages go to training (seven of each label) and six to testing (three of each). The seed, not us, chose which messages are held back, and the model must never see them while it learns. If your data has a natural order, such as dates, split by time instead of randomly, so that the test set really looks like the future.

Many projects use three parts: train to fit, validation to choose settings, and test for one final report. With very little data, cross-validation (covered in scikit-learn pipelines) reuses the data more efficiently.

Always start with baselines

Before you build anything clever, find out how hard the problem is with two cheap baselines:

  1. The majority baseline always predicts the most common label. It shows how much accuracy you get for free. On imbalanced data it can be deceptively high: if 95% of emails are not spam, a model that never flags anything is 95% accurate and useless.
  2. A simple rule, such as "contains the word free". It shows what a few minutes of hand-crafted logic can achieve, and it is the standard your model must beat to justify its complexity.

Then fit a first learned model: bag-of-words counts fed to a Naive Bayes classifier. Do not worry about the details yet; the next lesson builds it from scratch.

Python Two baselines and a first model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The majority baseline scores 0.5 (the classes are balanced, so guessing is a coin flip). The keyword rule scores 0.667: it catches the spam messages that contain one of its words, but misses "Cheap meds online" and "send your bank details" because those words are not on the list. The learned model scores 1.0 on these six messages and gets both new sentences right. Be careful before celebrating, though: six test messages are far too few to trust a perfect score. One more mistake would drop it to 0.83. On real work you need hundreds or thousands of test examples, and metrics beyond accuracy, which evaluating classifiers covers.

The leakage trap

Data leakage means information from the test set sneaks into training. It makes the score look better than reality, and it is the most common way beginners fool themselves. In text work the classic version is fitting the vectorizer (vocabulary, IDF weights) on all the data before splitting. The test messages then contribute words to the vocabulary, and the evaluation is no longer a fair rehearsal for new text.

The correct rule is: every step that learns something from data (vocabulary, IDF values, scalers, feature selection, the classifier itself) is fitted on the training part only, and then applied unchanged to the test part. Pipelines enforce this rule automatically, which is a big reason to use them.

Python Why you fit on training text only

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The vectorizer fitted on training messages learns 80 words. Fitted on everything it knows 113. In real use the model will meet words it has never seen, and 33 of the words in the six test messages are unknown to the training vocabulary. A fair test must include that situation, because that is what deployment looks like. Other leaks to watch for: near-duplicate texts landing on both sides of the split (the same email forwarded twice), and features that reveal the label, such as a "[SPAM]" tag added by an earlier filter.

Choosing your approach

You now have a complete workflow in miniature. For the rest of the chapter the pieces get better, in this order: Naive Bayes (fast, strong on small data), logistic regression and linear SVMs with TF-IDF (usually the best classic default), pipelines to keep everything tidy, and evaluation metrics to judge honestly. Later in the course, fine-tuned transformers replace the features and classifier with a pretrained network; the workflow around it (labels, splits, baselines, metrics, no leakage) stays the same.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do we keep a test set that the model never sees during training?

    A model can memorize the text it studied. Only data it has never seen shows how it will behave in real use.
  2. 2. What does stratify=labels do in train_test_split?

    Stratification splits each class separately, so both parts have the same share of every label. This matters most when one class is rare.
  3. 3. A data set is 95 percent not-spam. A model that always predicts "not spam" gets 95 percent accuracy. What does this show?

    High accuracy is easy when one class dominates. The majority baseline reveals that, and you need metrics like precision and recall to see how well the rare class is found.
  4. 4. Which of these is an example of data leakage?

    Fitting the vocabulary or IDF weights on all the data lets information about the test messages shape the features. Everything that learns from data should be fitted on the training part only.

Summary

  • Text classification maps a document to a label from a fixed set: binary, multi-class or multi-label.
  • The workflow is labeled data, split, features, fit on train, predict test, compare with true labels.
  • Use train_test_split with random_state for repeatability and stratify to keep class proportions.
  • Compare every model against a majority baseline and a simple rule; small test sets give untrustworthy scores.
  • Fit everything that learns (vectorizers, scalers, models) on the training part only to avoid leakage.

Next, build your first real classifier with the Naive Bayes Text Classifier.