Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 36 · Getting Started

A Probability Refresher for Bayesians

Review the probability you need for Bayesian statistics: joint, marginal and conditional probability, independence, the law of total probability and distributions, with Python.

Bayes' theorem is only a few lines of algebra on top of basic probability. If joint, marginal and conditional probabilities feel solid, everything else in this course becomes easy. This lesson is a fast, practical review with code, not a full probability course.

Events, joint and marginal probability

An event is something that either happens or not. P(A) is its probability, a number between 0 and 1. The joint probability P(A, B) is the chance that both happen. The marginal probability P(A) ignores B and adds up over all the ways B can turn out.

Imagine 1,000 people. We record whether each one has a flu shot (S) and whether they got flu this winter (F). The table of counts gives every probability we need.

Python Joint, marginal, conditional

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Conditional probability

P(A | B) is the probability of A once you know B happened. You shrink the world to the cases where B is true, then ask how many of those also have A:

P(A | B) = P(A, B) / P(B)

Here P(flu | shot) = 30/500 = 0.06 while P(flu | no shot) = 120/500 = 0.24. Conditioning changes the answer. This is the basic move of Bayesian inference: the probability of a parameter after conditioning on data.

Order matters. P(flu | shot) is not P(shot | flu). Mixing them up is the most common probability mistake in real life, and Bayes' theorem is the correct bridge between the two directions (next chapter).

The product rule

Rearranging the definition gives the product rule:

P(A, B) = P(A | B) P(B) = P(B | A) P(A)

Both factorizations describe the same joint probability. Setting them equal is exactly how Bayes' theorem is derived.

Independence

Events are independent if knowing one tells you nothing about the other: P(A | B) = P(A), so P(A, B) = P(A) P(B). Independence is a modeling assumption you will make constantly. For example, "given the coin's bias θ, the flips are independent", which lets us multiply per-flip probabilities to get a likelihood.

The law of total probability

If B₁, B₂, … split the world into mutually exclusive cases that cover everything, then

P(A) = Σ P(A | Bᵢ) P(Bᵢ)

Each term asks "how likely is A in this case, weighted by how likely the case is". This formula is the denominator of Bayes' theorem. In continuous problems the sum becomes an integral over the parameter.

Python Law of total probability by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exact value is 0.005 + 0.006 + 0.010 = 0.021, and the simulation lands very close.

Random variables and distributions

A random variable assigns a number to an outcome. Two kinds appear in this course:

  • Discrete (counts): described by a probability mass function (PMF). Examples: Binomial, Poisson.
  • Continuous (measurements, and parameters such as a rate between 0 and 1): described by a probability density function (PDF). Examples: Normal, Beta, Gamma.

A density is not a probability: it can exceed 1. Probabilities are areas under the density, found with the CDF (cumulative distribution function).

Python PMF, PDF and CDF in SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expectation and variance

The expected value is the probability-weighted average, E[X] = Σ x P(x) (an integral for continuous variables). The variance Var(X) = E[(X - E[X])²] measures spread, and the standard deviation is its square root. Posterior means and standard deviations are the two most common summaries you will report.

Likelihood vs probability

Before moving on, one subtle idea. Given a fixed parameter, P(data | θ) is a probability over possible data. Given fixed observed data, the same expression viewed as a function of θ is called the likelihood. It does not sum to 1 over θ; it just compares how well different parameter values explain the data. We devote a full lesson to this (The Likelihood Function).

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. Which formula defines the conditional probability P(A | B)?

    Restrict to cases where B is true, then take the fraction of those where A is also true: P(A, B) / P(B).
  2. 2. What does the law of total probability compute?

    It adds up the contributions of every mutually exclusive case, which is the denominator in Bayes' theorem.
  3. 3. A probability density function for a continuous variable...

    Density is probability per unit; only areas are probabilities, so the height can exceed 1.
  4. 4. Why is P(flu | shot) different from P(shot | flu)?

    Conditioning direction matters; Bayes' theorem converts one into the other.

Summary

  • Joint P(A, B), marginal P(A) and conditional P(A | B) = P(A, B)/P(B) are the core objects.
  • Product rule: P(A, B) = P(A | B) P(B); independence means P(A, B) = P(A) P(B).
  • Law of total probability sums over cases and becomes the denominator of Bayes' theorem.
  • Discrete variables have PMFs, continuous ones have PDFs; probabilities are areas.
  • Likelihood is the data's probability viewed as a function of the parameter.

Next: The Bayesian Workflow