Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 36 · Bayes Theorem and Priors

Bayes' Theorem

Understand Bayes' theorem from the product rule, apply it to a medical test and spam filter, use natural frequencies and odds form, and update beliefs sequentially in Python.

Bayes' theorem tells you how to turn P(data | hypothesis), which is usually easy to compute, into P(hypothesis | data), which is usually what you want. It is a direct consequence of the product rule from the probability refresher, and it is the engine of every method in this course.

Deriving the theorem

The product rule says P(H, D) = P(D | H) P(H) = P(H | D) P(D). Divide both sides by P(D):

P(H | D) = P(D | H) P(H) / P(D)

In words: posterior = likelihood × prior / evidence.

  • P(H) is the prior: belief in the hypothesis before the data.
  • P(D | H) is the likelihood: how likely the data is if the hypothesis is true.
  • P(D) is the evidence (marginal likelihood): Σ P(D | Hᵢ) P(Hᵢ) over all hypotheses, by the law of total probability. It only normalizes the answer so it sums to 1.

The classic example: a medical test

A disease affects 1% of people. A test detects the disease in 90% of sick people (sensitivity) and wrongly flags 8% of healthy people (false positive rate). You test positive. What is the probability you are sick?

Most people guess about 90%. Compute it.

Python Medical test with Bayes' theorem

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The answer is about 0.102, roughly 10%. The disease is rare, so most positive results come from the large healthy group, even with a low false-positive rate. The prior matters.

Natural frequencies

Imagine 10,000 people. About 100 are sick, and 90 of them test positive. About 9,900 are healthy, and 8% of them, 792, test positive too. Of the 882 positives, 90 are sick: 90 / 882 ≈ 0.102. The same answer, with no formulas.

Python Check it by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Simulation is a habit worth keeping: if you can simulate the data-generating story, you can check your math.

Odds form

Odds are p / (1 - p). Bayes' theorem has a neat odds version:

posterior odds = prior odds × likelihood ratio

The likelihood ratio here is P(positive | sick) / P(positive | healthy) = 0.90 / 0.08 = 11.25. Prior odds are 0.01/0.99 ≈ 0.0101. Multiply: 0.1136, which as a probability is 0.1136 / 1.1136 ≈ 0.102. The odds form is convenient for chaining updates, because each new piece of evidence multiplies the odds.

Sequential updating

Suppose you take a second, independent test and it is also positive. The old posterior becomes the new prior.

Python Two positive tests

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The probability rises from 1% to 10%, to about 56%, to about 93.5%. Evidence accumulates, and the order of the tests does not change the final result. This is the same mechanism that turns a wide posterior into a narrow one as data arrives.

flowchart LR
    P0["Prior 1%"] -->|"test 1 positive<br/>x11.25 odds"| P1["10%"]
    P1 -->|"test 2 positive"| P2["56%"]
    P2 -->|"test 3 positive"| P3["93.5%"]

Each piece of evidence multiplies the odds by its likelihood ratio.

From hypotheses to parameters

With a discrete set of hypotheses, you apply the formula to each and normalize. With a continuous parameter θ, the sum becomes an integral:

p(θ | data) = p(data | θ) p(θ) / ∫ p(data | θ) p(θ) dθ

That integral is usually hard, which is why we have conjugate priors, grids and MCMC. But because the denominator does not depend on θ, the shape of the posterior is already known from the numerator:

p(θ | data) ∝ p(data | θ) p(θ)

where ∝ means "proportional to". Nearly all Bayesian computation works with this unnormalized form.

Common mistakes

  • The base-rate fallacy: ignoring the prior probability and treating P(positive | sick) as P(sick | positive).
  • Double counting evidence: feeding the same data in twice, or treating dependent tests as independent.
  • Zero priors: a hypothesis given prior probability 0 can never recover, however much evidence appears. Avoid exact zeros unless it is truly impossible.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. In Bayes' theorem P(H | D) = P(D | H) P(H) / P(D), what is P(H)?

    P(H) is the probability of the hypothesis before seeing the data: the prior.
  2. 2. A rare disease (1%) has a test with 90% sensitivity and 8% false positives. Why is P(sick | positive) only about 10%?

    The base rate is low, so the many healthy people produce more false positives than the few sick people produce true positives.
  3. 3. Which statement is the odds form of Bayes' theorem?

    Multiplying odds by the likelihood ratio performs the update.
  4. 4. Why is it enough to know the posterior up to proportionality?

    The evidence is a constant over the parameter; normalizing at the end recovers the exact posterior.

Summary

  • Bayes' theorem: posterior = likelihood x prior / evidence, derived from the product rule.
  • The evidence is a normalizer from the law of total probability.
  • Rare events and low base rates make positive tests less conclusive than they feel.
  • The odds form multiplies prior odds by a likelihood ratio and chains easily.
  • Posteriors are proportional to likelihood times prior, which drives all later computation.

Next: The Likelihood Function