Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 40 ยท Classification

Naive Bayes

Classify with Naive Bayes in scikit-learn: Bayes' rule and the naive assumption, GaussianNB, MultinomialNB for text counts, BernoulliNB, smoothing and a spam filter.

Naive Bayes is a family of fast, simple classifiers built on Bayes' rule. They need very little data, train in a single pass and are a classic first choice for text classification such as spam filtering. They are "naive" because they assume features are independent of each other given the class, which is rarely exactly true, yet the models often work well anyway.

The idea

Bayes' rule turns "how likely is this evidence if the email is spam?" into "how likely is spam given this evidence?":

P(class | features) โˆ P(class) ร— P(feature1 | class) ร— P(feature2 | class) ร— ...

The model counts how often each feature value appears in each class, multiplies the pieces together (the naive independence step) and picks the class with the largest score.

Python Bayes by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Seeing one word lifts the probability of spam from 20% to about 88%. Naive Bayes does this for every feature at once.

Pick the version that matches your features

Class Features it expects Typical use
GaussianNB Continuous numbers (assumed bell-shaped per class) Measurements
MultinomialNB Non-negative counts Word counts, TF-IDF
BernoulliNB Binary 0/1 Word present or absent
CategoricalNB Integer-coded categories Categorical columns
ComplementNB Counts, tuned for imbalanced text Imbalanced text

GaussianNB

For each class and feature it learns a mean and a variance.

Python GaussianNB on iris

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

No scaling is needed because each feature is modelled on its own. Probabilities from Naive Bayes tend to be overconfident (close to 0 or 1) even when accuracy is fine, so use them for ranking rather than as exact probabilities, or calibrate them.

MultinomialNB for text: a spam filter

Text must first become numbers. CountVectorizer turns each message into word counts, and MultinomialNB works on those counts. A pipeline joins them.

Python A tiny spam filter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last message mixes spam words (win, free) with a normal one (lunch), so the model leans spam (about 86%), a good example of how the word evidence is multiplied together. More on turning text into features is in Text Features.

Smoothing: the alpha parameter

If a word never appears in spam during training, its spam probability is 0, and multiplying by 0 wipes out all other evidence. Laplace smoothing adds a small count alpha to every word so nothing has exactly zero probability.

Python Why smoothing matters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without smoothing, one unseen word makes the whole product 0, whatever the other evidence says. With alpha=1 it just lowers the score a little. scikit-learn applies this for you through the alpha parameter of MultinomialNB.

The default alpha=1.0 is a sensible start; tune it by cross-validation.

BernoulliNB: present or absent

BernoulliNB models whether each word occurs at all, and it also uses the absence of words as evidence. It can do better on short texts.

Python BernoulliNB with binary features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Few samples, still sensible

Naive Bayes stays reliable with very few training rows, where flexible models struggle. It can also learn in batches with partial_fit, which helps when data does not fit in memory.

Python Twelve training rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Strengths and weaknesses

  • Very fast to train and predict; works with tiny datasets and huge sparse text matrices.
  • Few settings to tune (mostly alpha, or var_smoothing for GaussianNB).
  • Probabilities are poorly calibrated, and strongly correlated features get counted twice.
  • Often beaten by tuned boosting or logistic regression when data is plentiful, but a strong, quick baseline.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the naive assumption in Naive Bayes?

    The model multiplies per-feature probabilities as if features did not influence each other.
  2. 2. Which class fits word-count features from text?

    MultinomialNB models non-negative counts such as word frequencies.
  3. 3. What does the alpha parameter (Laplace smoothing) prevent?

    Adding a small count to everything avoids multiplying by zero.
  4. 4. Which statement about Naive Bayes probabilities is generally true?

    The independence assumption makes probabilities too extreme, even when predicted classes are good.

Summary

  • Naive Bayes applies Bayes' rule with an independence assumption: fast, simple and strong for text.
  • Choose the version that matches your features: GaussianNB, MultinomialNB, BernoulliNB, CategoricalNB, ComplementNB.
  • Use alpha smoothing, and treat its probabilities with caution.
  • A quick baseline that works with very little data.

Next: Train/Test Split and Cross-Validation