Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 63 ยท Text Classification

Logistic Regression and Linear SVMs for Text

Train logistic regression and linear SVM classifiers on TF-IDF features, tune regularization with C, and read the top weights to see what the model learned.

Naive Bayes is quick and surprisingly good, but it makes a strong assumption: every word contributes to the decision independently. Logistic regression and linear support vector machines (SVMs) drop that assumption. They learn one weight per word directly from the labeled examples, choosing weights that separate the classes as well as possible. With TF-IDF features they form one of the strongest baselines in text classification, and on many real problems they are still hard to beat.

Both models are linear: the score for a class is a weighted sum of word features plus a bias. That makes them fast to train on thousands of columns, and it gives them a huge practical benefit. You can open the model and read which words it relies on.

Two linear models, one idea

A linear classifier computes, for each document x and class k:

TEXT
score_k(x) = w_k . x + b_k

Here x is the TF-IDF vector, w_k has one weight per vocabulary word, and b_k is a bias. A large positive weight for goal in the sports class means "seeing goal pushes this document toward sports". The two models differ only in how they pick the weights.

Same shape, different loss

Logistic regression Turns scores into probabilities and minimizes log loss.
Linear SVM Maximizes the margin between the classes using hinge loss.

Both learn one weight per word.

1 / 2

Logistic regression gives you calibrated-looking probabilities through predict_proba. A linear SVM (LinearSVC in scikit-learn) only gives scores, but it is often a hair more accurate on sparse text and trains very fast. In practice you try both.

Training on TF-IDF features

The example builds a small synthetic corpus: each document mixes three words from its own topic, three words from random topics (noise), and a few filler words. That makes the problem realistic enough that the models can make mistakes. We split it in half, fit a TfidfVectorizer on the training half only, and train both models.

Python Logistic regression and LinearSVC on TF-IDF

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training matrix has 60 documents and 43 distinct words. Both models fit the training data perfectly (accuracy 1.0) but score a little lower on the 60 unseen documents: 0.900 for logistic regression and 0.917 for the SVM. That gap of one document is not a meaningful difference on a test set this small, but the gap between train and test is the important part: the noise words give each model something to memorize.

The last line shows predict_proba for a hand-made sentence. Classes are listed alphabetically (food, sports, tech), so the output [[0.234 0.222 0.544]] means 54% tech: server, update and battery all pushed toward tech, but the model is far from certain, because probabilities from a regularized model on 60 short documents are soft.

Always fit the vectorizer on the training data only and call transform on the test data. Fitting it on everything lets document frequencies from the test set leak into your features, which makes evaluation look better than it is. The pipelines lesson shows how to make this mistake impossible.

Reading the weights

For a linear model, the coefficients are the explanation. lr.coef_ has one row per class and one column per vocabulary word. Sorting a row shows which words most strongly indicate that class.

Python The top words per class

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each class is led by words from its own topic: recipe and fresh for food, team and goal for sports, update and chip for tech. The against list holds the most negative weights, which mostly belong to words of other topics (recipe counts against sports). One filler word, of, sneaks in for food: with only 60 training documents, chance patterns get small weights too. The last two lines summarize the split: the 30 topic words have a mean absolute weight of 0.44, while the 13 filler words average 0.14, so the model has learned to rely mostly on topic vocabulary.

Inspecting weights is also how you catch trouble. If the top word for a spam class is a user's name or a leftover HTML tag, your model has found a shortcut that will not generalize. This kind of check belongs in error analysis.

Regularization and the C parameter

With thousands of features and few documents, a model can fit the training set by giving large weights to rare words that happen to line up with the labels. Regularization penalizes large weights, trading a little training accuracy for better behavior on new data.

In scikit-learn the strength is set by C, and the direction is the one people forget: C is the inverse of regularization strength. Small C means strong regularization (small weights, simpler model); large C means weak regularization (the model can chase the training data). The default is C=1.0.

Python Sweeping C

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the logistic regression table from top to bottom. The weight size grows steadily, from 0.10 at C=0.01 to 34.91 at C=1000: weak regularization lets the weights explode. Test accuracy is best around the middle (0.900 at C=1) and falls to 0.850 at C=1000, even though training accuracy is already 1.0 everywhere. A small test set makes these differences noisy, so you should never pick C by staring at one split. The right tool is cross-validated search, introduced in scikit-learn Pipelines for Text.

Two regularization types are common. L2 (the default) shrinks all weights smoothly. L1 pushes many weights to exactly zero, which gives a sparse, shorter list of words; it needs a solver that supports it and, in recent scikit-learn versions, is set through different arguments than older tutorials show, so check the documentation of your installed version.

Practical advice

  • Start with TF-IDF and logistic regression. It trains in seconds and gives you a baseline before you try anything fancier.
  • Try LinearSVC too. It is often slightly better on sparse text and has the same one-line interface.
  • Use class_weight="balanced" when one class is rare, so errors on the rare class cost more (see Evaluating Classifiers).
  • Increase max_iter if you see a convergence warning with logistic regression.
  • Add n-grams with ngram_range=(1, 2) so that not good is its own feature; see N-grams.
  • Do not trust raw probabilities blindly. A model can be confident and wrong, and small corpora give weakly calibrated numbers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a large positive weight for the word "goal" in the "sports" row of lr.coef_ mean?

    In a linear model the score for a class is a weighted sum of features, so a large positive weight increases the score of that class whenever the word is present.
  2. 2. You set C=0.01 in LogisticRegression. What happens?

    C is the inverse of the regularization strength. A small C penalizes large weights heavily, which gives a simpler model.
  3. 3. Why should you fit TfidfVectorizer only on the training documents?

    The IDF values are learned from the data the vectorizer sees. Using test documents lets information about them influence training, so the score no longer measures performance on unseen text.
  4. 4. Which statement about logistic regression and LinearSVC is true?

    Both are linear classifiers over the same features. Logistic regression uses log loss and gives probabilities; the linear SVM uses hinge loss and maximizes the margin. Which one wins depends on the data.

Summary

  • Logistic regression and linear SVMs learn one weight per word from TF-IDF features; they are fast, strong baselines for text.
  • Logistic regression provides predict_proba; LinearSVC provides scores and is often slightly more accurate on sparse text.
  • The coefficients show which words drive each class, which helps you debug and explain the model.
  • C is the inverse of regularization strength: small C means small weights and a simpler model, large C lets the model chase the training data.
  • Fit the vectorizer on training data only, and choose C by cross-validation rather than one small split.

Next, build clean, leak-free experiments with scikit-learn Pipelines for Text.