Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 63 ยท Text Classification

Naive Bayes Text Classifier

Learn Bayes' rule for text, build a multinomial Naive Bayes classifier from scratch with Laplace smoothing and log probabilities, then use scikit-learn.

Naive Bayes is the oldest trick in text classification and still a strong first model. It trains in a single pass over the data, works with very few examples, handles thousands of features without fuss and gives probabilities you can read. The first spam filters of the 1990s used it, and it is still a sensible baseline today. The model is "naive" because of one simplifying assumption, and understanding that assumption is the whole lesson.

You will derive the classifier from Bayes' rule, write it in about 40 lines of plain Python, check it against scikit-learn's MultinomialNB, and see why Laplace smoothing and log probabilities are not optional details. It builds on bag of words, the classification workflow, and the smoothing idea from the language-model chapter.

Bayes' rule for documents

We want the probability that a document belongs to a class c given its words. Bayes' rule turns that around into things we can count:

TEXT
P(class | words) = P(words | class) * P(class) / P(words)
  • P(class) is the prior: how common the class is. If 40% of training messages are spam, the prior for spam is 0.4.
  • P(words | class) is the likelihood: how likely this exact set of words is among messages of that class.
  • P(words) is the same for every class, so when we only want to choose the best class we can ignore it.

The likelihood of a whole document is impossible to estimate directly, because almost every document is unique. Here comes the "naive" part. Assume that, given the class, words are independent of each other:

TEXT
P(w1, w2, ..., wn | class) = P(w1 | class) * P(w2 | class) * ... * P(wn | class)

This is false: "free" and "prize" appear together far more often than independence would predict. But the classifier only needs the ranking of classes to come out right, and that is surprisingly robust. The independence assumption turns an impossible estimation problem into counting how often each word appears in each class.

Naive Bayes in four steps

Count Words per class, documents per class.
Estimate P(word | class) from the counts.
Score Prior plus the log of each word.
Pick The class with the highest score wins.

The "naive" part is treating words as independent.

1 / 2

Multinomial Naive Bayes and Laplace smoothing

The multinomial version treats a document as a bag of words and uses word counts. The estimate for a word in a class is

TEXT
P(word | class) = (count of word in class + alpha) / (total words in class + alpha * V)

where V is the vocabulary size and alpha is the Laplace smoothing constant, usually 1. Without it, a word never seen in class c gets probability 0, and because the word probabilities are multiplied, one such word makes the whole score for that class zero, no matter how much other evidence there is. It is exactly the zero-probability problem from smoothing, and the same add-one fix applies.

The second practical point is log probabilities. A message with 300 words multiplies 300 numbers well below 1, and the product falls below the smallest number a float can hold. Taking logs turns the product into a sum, which is stable, and since the logarithm is monotonic the best class stays the best class:

TEXT
score(class) = log P(class) + sum over words of log P(word | class)

A quick experiment shows the problem. Suppose each word in a long message has probability 0.01 under a class.

Python Why we use logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At 100 words the product is already tiny, and by 200 words it is 0.0 (the true value, 1e-400, is below the smallest float). Both classes would score 0 and become indistinguishable, while the sum of logs is a harmless number such as -921.0.

Naive Bayes from scratch

The class below follows the recipe exactly: fit counts documents per class and words per class; log_scores adds the log prior and the log likelihood of every known word. Unknown words (never seen in any class) are skipped, because they carry no information about which class is likelier. The class also turns the scores into probabilities with the "subtract the maximum" trick that keeps the exponentials stable.

Python Multinomial Naive Bayes in plain Python

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output from the top. "free prize today" is 94% spam: both "free" and "prize" appear only in spam. "team meeting tomorrow" is 92% ham. "free meeting" is a genuine tug of war, with "free" pulling to spam and "meeting" to ham, and spam narrowly wins at 72%. "hello world" contains only unknown words, so the model falls back on the prior alone and says 50/50, which is the honest answer.

The last block explains smoothing. With alpha=1 the two classes get finite scores. With alpha=0, "notes" never occurs in spam and "free" never occurs in ham, so both scores become minus infinity: the model declares every class impossible and cannot decide. Smoothing is what keeps one unseen word from vetoing a class.

alpha=1 is a default, not a law. Smaller values such as 0.1 or 0.01 often work better on large vocabularies. Choose it by measuring on held-out data or with cross-validation.

The same model in scikit-learn

In practice you let CountVectorizer build the document-term matrix and MultinomialNB do the counting. The vectorizer below is told to use the same tokens as our class ([a-z]+, which keeps one-letter words), so the two implementations should agree to the last digit. feature_log_prob_ holds log P(word | class), and subtracting the spam row from the ham row ranks the words by how strongly each points to a class.

Python scikit-learn gives the same answers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The probabilities match the hand-written class: 0.941, 0.082 for spam on the second text and 0.723. Agreement is a good way to check your understanding: scikit-learn's MultinomialNB is no more than the counting you just wrote, vectorized. The word rankings are the readable side of Naive Bayes. The words that lean hardest toward ham are the ones that occur only in ham messages and are frequent there ("the", "for", "tomorrow", "notes"), while "free", "prize", "now" and "your" lead the spam side. Because the model is a table of per-word weights, it is easy to audit, which is an advantage over many more complicated models.

Strengths, limits and variants

  • Strong on small data and short texts. With few parameters per word, it overfits less than flexible models when examples are scarce.
  • Probabilities are not well calibrated. The independence assumption makes the model overconfident: it often says 99% when 85% would be honest. Use its ranking, not its raw probabilities, or calibrate them.
  • Correlated words get double counted. "New" and "York" push together as if they were two independent pieces of evidence.
  • It is usually beaten by logistic regression with TF-IDF on larger datasets, which learns how words interact with each other instead of assuming independence.
  • Other flavors. BernoulliNB uses word presence instead of counts, and ComplementNB is often better on imbalanced text data. GaussianNB is for continuous features and is rarely used on text.

A good habit is to train Naive Bayes first. It takes seconds, gives a baseline for the more complicated models to beat, and its word weights often reveal data problems, such as a signature line that perfectly predicts the label.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the "naive" assumption in Naive Bayes?

    The model multiplies P(word | class) over the words as if they were independent given the class. That is false in real language, but it makes the likelihood estimable from simple counts.
  2. 2. Why is Laplace smoothing (alpha greater than 0) needed?

    Without smoothing, a word unseen in a class has probability 0, and because the word probabilities are multiplied, the whole class score collapses to 0 (minus infinity in log space).
  3. 3. Why do implementations add log probabilities instead of multiplying probabilities?

    A long document multiplies hundreds of values below 1, which a float cannot represent. Logs turn the product into a sum and keep the same winning class.
  4. 4. A message contains only words that never appeared in training. What does the model above predict?

    Unknown words are ignored, so only the log prior contributes to each score. With equal class sizes that gives 50/50.

Summary

  • Naive Bayes picks the class that maximizes P(class) * product of P(word | class), derived from Bayes' rule.
  • The naive assumption is that words are independent given the class; it is wrong but works well for ranking.
  • Laplace smoothing (alpha) prevents unseen words from zeroing a class, and log probabilities prevent numeric underflow.
  • CountVectorizer plus MultinomialNB reproduces the hand-written classifier; the per-word weights make the model easy to inspect.
  • It is fast and strong on small data, but its probabilities are overconfident and larger datasets favor logistic regression.

Next, let the model learn how words interact with Logistic Regression and Linear SVMs for Text.