Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 57 ยท Probability Foundations

Bayes' Theorem

Learn Bayes' theorem with medical-test, spam and factory examples: priors, likelihoods and posteriors, the base-rate fallacy, and sequential updating in Python.

Bayes' theorem tells you how to revise a belief when new evidence arrives. It reverses conditional probabilities: from "how likely is this evidence if the hypothesis is true?" to "how likely is the hypothesis, now that I have seen the evidence?". It is behind spam filters, medical diagnosis, search-and-rescue planning and a whole school of statistics.

The formula

From the multiplication rule, P(H and E) = P(E | H) * P(H) = P(H | E) * P(E). Solve for P(H | E):

P(H | E) = P(E | H) * P(H) / P(E)

and use the law of total probability for the denominator:

P(E) = P(E | H) * P(H) + P(E | not H) * P(not H)

The terms have names:

Term Name Meaning
P(H) Prior Belief in the hypothesis before the evidence
P(E | H) Likelihood How probable the evidence is if H is true
P(E) Evidence (marginal) Overall probability of the evidence
P(H | E) Posterior Updated belief after the evidence
flowchart LR
    P["Prior<br/>P(H)"] --> X["x"]
    L["Likelihood<br/>P(E | H)"] --> X
    X --> N["divide by P(E)"]
    N --> Q["Posterior<br/>P(H | E)"]

Posterior is proportional to prior times likelihood; dividing by P(E) makes it sum to 1.

The classic medical test

A disease affects 1% of a population. A test detects it in 95% of sick people (sensitivity) and wrongly flags 5% of healthy people (false positive rate, so specificity is 95%). You test positive. What is the probability you are sick?

Most people guess about 95%. Compute it:

  • Prior: P(sick) = 0.01
  • P(positive | sick) = 0.95, P(positive | healthy) = 0.05
  • P(positive) = 0.95 * 0.01 + 0.05 * 0.99 = 0.0095 + 0.0495 = 0.059
  • Posterior: 0.0095 / 0.059 โ‰ˆ 0.161

Only about 16%. Among 10,000 people, 100 are sick and 95 of them test positive; 9,900 are healthy and 495 of them also test positive. So of 590 positives, only 95 are actually sick. The disease is so rare that false positives from the large healthy group outnumber true positives. This is the base-rate fallacy: ignoring the prior.

Python The medical test, two ways

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Thinking with natural frequencies (counts out of 10,000) often avoids the mistake.

How the prior changes the answer

The same test gives very different posteriors depending on who is tested.

Python Posterior as a function of the prior

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a high-risk group with a 30% prior, a positive result gives about 89%. Doctors choose when to screen with exactly this reasoning in mind.

Which machine made the defect?

Earlier you saw a factory where machine 1 makes 60% of the items (2% defective) and machine 2 makes 40% (5% defective). A defective item is found. Bayes gives P(machine 1 | defective) = 0.012 / 0.032 = 0.375. Even though machine 1 makes most items, it is less likely to be the culprit because its defect rate is lower.

Sequential updating

Today's posterior becomes tomorrow's prior. Take a second, independent positive result in the medical example: start with the first posterior of 0.161 as the new prior.

Python Two positive tests in a row

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After two positives the probability rises to about 78%, and after three to about 98%. Repeated independent evidence quickly overwhelms a low prior. (This assumes independent tests; the same flawed test repeated may make the same error, which breaks the assumption.)

Odds form

Bayes can be written with odds: posterior odds = prior odds * likelihood ratio. In the medical example, prior odds are 1:99, and the likelihood ratio is 0.95 / 0.05 = 19. Posterior odds are 19:99, which corresponds to a probability of 19 / 118 โ‰ˆ 0.161. This form makes sequential updating a simple multiplication.

Spam filter in one line

A naive Bayes spam filter applies the same idea per word. If 60% of spam and 5% of ham contain "winner", and 20% of mail is spam, then P(spam | "winner") = 0.6 * 0.2 / (0.6 * 0.2 + 0.05 * 0.8) = 0.12 / 0.16 = 0.75. Combining many words, assuming they are conditionally independent given the class, is the classic naive Bayes classifier.

Bayes and statistics

Frequentist statistics asks how data would behave if a hypothesis were true. Bayesian statistics treats parameters themselves as uncertain, with a prior distribution updated by data. You will explore this in the Bayesian chapter of this course.

Common mistakes

  • Reporting P(E | H) as if it were P(H | E).
  • Ignoring the base rate.
  • Using a prior of exactly 0 or 1: no evidence can then change your mind.
  • Applying repeated updates to evidence that is not independent.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In Bayes' theorem, which term is the belief before seeing evidence?

    The prior is P(H) before conditioning on the evidence.
  2. 2. A disease has prevalence 2%, sensitivity 90%, false positive rate 10%. Roughly what is P(sick | positive)?

    0.90.02 / (0.90.02 + 0.1*0.98) = 0.018 / 0.116 โ‰ˆ 0.155.
  3. 3. What is the base-rate fallacy?

    Neglecting the prior makes rare conditions look far more likely after a positive test.
  4. 4. Posterior odds equal...

    That is the odds form of Bayes' theorem.

Summary

  • Bayes' theorem: P(H | E) = P(E | H) * P(H) / P(E).
  • Prior, likelihood and posterior are the three ingredients; the denominator makes probabilities sum to 1.
  • Rare conditions give surprisingly low posteriors after a positive test, because of the base rate.
  • Posteriors become priors for the next piece of evidence.
  • Odds form: posterior odds = prior odds times likelihood ratio.

Next: Random Variables and Probability Mass Functions