Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 111 ยท Classification

Naive Bayes

Understand Bayes' rule and the naive independence assumption, then use GaussianNB, MultinomialNB and BernoulliNB, including a small text-count example.

Naive Bayes is a family of classifiers built on one old formula, Bayes' rule, and one bold simplification: it pretends that, given the class, all features are independent of each other. The assumption is almost never true, which is why it is called "naive", yet the classifiers work remarkably well in practice, particularly for text. They train in a single pass over the data, need very little of it, handle thousands of features and make predictions instantly.

This lesson derives the idea with a tiny example, shows the three most-used scikit-learn variants (GaussianNB, MultinomialNB, BernoulliNB), walks through a text-count example and explains why the probabilities it outputs should be treated with suspicion. For a text pipeline built on these ideas, see Text Features later in the track.

Bayes' rule in plain words

Prior How common is each class before looking?
Likelihood How typical are these features for the class?
Posterior Prior times likelihood, then normalized.

Predict the class with the highest posterior.

1 / 2

Bayes' rule and the naive assumption

Bayes' rule says how to update a belief after seeing evidence:

TEXT
P(class | features) = P(features | class) * P(class) / P(features)
  • P(class) is the prior: the share of each class in the training data.
  • P(features | class) is the likelihood: how probable these feature values are for that class.
  • P(class | features) is the posterior, what we want.
  • P(features) is the same for every class, so for choosing the best class it can be ignored and the scores are simply normalized at the end.

The trouble is the likelihood of a whole feature combination, which would need huge amounts of data to estimate. The naive assumption breaks it into a product of one-feature likelihoods:

TEXT
P(x1, x2, x3 | class) = P(x1 | class) * P(x2 | class) * P(x3 | class)

Each factor is easy to estimate from the training data. Prediction then picks the class that maximizes prior * product of likelihoods. In practice scikit-learn adds up logarithms instead of multiplying tiny numbers, to avoid underflow.

Python Naive Bayes by hand: is this email spam?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the Laplace smoothing (adding 1 to every count). Without it, a word never seen in ham, like "winner", would make the ham likelihood exactly 0 and wipe out the whole product no matter what the other words say. scikit-learn exposes this as the alpha parameter.

GaussianNB for continuous features

GaussianNB assumes each feature follows a bell curve (a normal distribution) within each class, and estimates one mean and one variance per class and feature. It needs no scaling and trains instantly. Its only smoothing knob is var_smoothing, which adds a tiny amount to every variance.

Python GaussianNB on the iris flowers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The learned parameters class_prior_ and theta_ (the means) are exactly the ingredients from Bayes' rule, so the model is fully inspectable.

MultinomialNB and BernoulliNB for counts and flags

For text and other count data, the standard choices are:

  • MultinomialNB models how many times each feature occurs (word counts, or TF-IDF values). It is the usual first choice for text classification.
  • BernoulliNB models whether each feature is present or absent and, unlike the multinomial version, it also learns from absences ("this email does not contain the word 'meeting'").

Both have alpha for Laplace smoothing (default 1.0). A smaller alpha trusts the data more; a larger one smooths more.

Python A tiny text classifier with word counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last message mixes a spam word and a ham word, so the model is clearly less certain than for the other two messages (it still leans spam because "free" appears more often in the spam examples). Inspecting feature_log_prob_ shows which words push toward each class:

Python Which words signal spam?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a Naive Bayes variant

All Naive Bayes classifiers use the same decision rule: pick the class with the largest P(class) ร— the product of P(feature | class). They differ only in the model for P(feature | class), which should match what your features look like.

Variant Feature type Models Typical use
GaussianNB Continuous numbers A bell curve per class and feature Measurements, sensor data, quick baselines
MultinomialNB Non-negative counts or TF-IDF How often each feature occurs Text with word counts
BernoulliNB Binary flags Present or absent, and it also uses absences Short texts, binary indicators
CategoricalNB Integer-coded categories A frequency table per category Tabular data with categorical columns
ComplementNB Counts Statistics of the other classes Text with imbalanced classes

Two practical points. First, the variants expect different inputs: MultinomialNB and ComplementNB fail on negative numbers (so do not feed them standardized features), and CategoricalNB needs categories coded 0, 1, 2, ..., with min_categories set so that a category that is absent from the training split does not crash prediction. Second, mixed data is awkward: you either bin the continuous columns into categories, or train a separate model per feature type and combine them.

Gaussian Naive Bayes from scratch

The implementation of GaussianNB is short enough to write yourself. Training stores a prior, a mean and a variance per class and feature. Prediction adds, for each class, the log prior and the log of a normal density for every feature, and takes the largest sum. Working with logs avoids multiplying hundreds of tiny numbers, which would underflow. The second half of the example shows what the "naive" independence assumption does when it is wrong: we feed the model near-copies of one feature, so the same evidence is counted again and again.

Python GaussianNB from scratch, and the cost of repeated features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The hand-written class gives the same accuracy as scikit-learn (0.978) and exactly the same predictions on every test row. The log joint for the first flower is far from equal across classes (-560.9, -21.2, -3.4), so the third class wins by a wide margin. The independence assumption shows in the last three lines. Adding near-duplicate copies of one feature does not make the model smarter, but it makes it more confident: the average top probability climbs from 0.971 to 0.998, because the same evidence is multiplied in again. This is why Naive Bayes is a good ranker and a poor source of calibrated probabilities (see calibration).

Which variant fits which data?

To compare the variants fairly, the next example generates one problem in several shapes: word counts, presence flags, small categorical columns and continuous numbers. It scores each applicable model by 5-fold cross-validation. It also tests ComplementNB on a text-like problem where only 10% of the rows belong to the class we care about.

Python Matching the variant to the data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On word counts, MultinomialNB (0.958) and GaussianNB (0.953) are close, and BernoulliNB is lower (0.855) because it throws away how many times each word appeared. When the same data is reduced to presence flags, BernoulliNB is the best of the three (0.855), because the flags are exactly what it models and it uses the absences too. For categorical columns, CategoricalNB matches GaussianNB here, and binning continuous numbers into categories loses a little information compared with GaussianNB (0.750 against 0.787). The final two lines show why ComplementNB exists: on the imbalanced problem it finds 97% of the minority class, against 68% for the multinomial model, at the price of a lower overall accuracy (0.920 against 0.960). If missing the rare class costs more than a few false alarms, that is a good trade. Results on your own data will differ, so always use cross-validation to decide, and measure the right metric (Classification Metrics).

Strengths, weaknesses and calibration

Naive Bayes shines when you have many features and little data, when you need speed (it supports partial_fit for streaming data), and as a quick baseline for text. Its weaknesses:

  • The independence assumption double-counts correlated evidence. If two features say the same thing, the model counts it twice and becomes overconfident.
  • Unseen combinations are handled only through smoothing.
  • The predicted probabilities are poorly calibrated: they tend to be pushed toward 0 and 1. The ranking of cases is usually good (decent ROC AUC), but a "0.99" is not really a 99% chance. If you need trustworthy probabilities, calibrate them (see Probability Calibration and Uncertainty).
Python Overconfidence from duplicated evidence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data never gets more informative (the copies carry the same information), yet the model becomes more and more "certain", because it multiplies the same evidence over and over. This is the independence assumption failing in action.

Tip: Use Naive Bayes as a fast, strong baseline for text and high-dimensional count data. If you need good probabilities or the features are strongly correlated, move to logistic regression or a tree ensemble.

Next, what changes when there are more than two classes, or several labels at once: Multiclass and Multilabel Classification.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the 'naive' assumption in Naive Bayes?

    The model multiplies one-feature likelihoods as if the features did not influence each other once the class is known.
  2. 2. Which variant is the usual first choice for word counts in text classification?

    MultinomialNB models count data such as word frequencies (or TF-IDF values). GaussianNB expects continuous bell-shaped features.
  3. 3. Why is Laplace smoothing (the alpha parameter) needed?

    Without smoothing, one unseen feature value gives probability 0 for that class no matter what the other features say.
  4. 4. Naive Bayes predicts 0.999 for a case. How much should you trust it as a literal probability?

    Naive Bayes is a good ranker but a poor probability estimator; calibrate it if the probabilities matter.

Summary

  • Naive Bayes picks the class with the highest prior * product of per-feature likelihoods, assuming features are independent given the class.
  • GaussianNB for continuous features, MultinomialNB for counts, BernoulliNB for presence/absence.
  • alpha (Laplace smoothing) prevents zero probabilities; the model trains in one pass and supports partial_fit.
  • Correlated features are double counted, so probabilities are overconfident; calibrate them if you need real probabilities.
  • It is a fast, strong baseline for text and other high-dimensional data.

Next up: Multiclass and Multilabel Classification.