Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท Model Checking and Comparison

Marginal Likelihood and Bayes Factors

Compute the marginal likelihood of a model, form Bayes factors to compare hypotheses, and see why Bayes factors depend strongly on the width of the prior.

Bayes' theorem divides by a constant we usually ignore: the marginal likelihood p(y | M), also called the evidence. It is the probability the model assigned to the data before seeing it, averaged over the prior:

TEXT
p(y | M) = integral of p(y | theta, M) * p(theta | M) d theta

When you compare two models, this quantity becomes the main character. The ratio of two marginal likelihoods is the Bayes factor:

TEXT
BF_12 = p(y | M1) / p(y | M2)

Combined with prior model odds, it gives posterior model odds: posterior odds = BF * prior odds. A BF of 10 means the data are ten times more probable under M1 than under M2.

A closed-form example

A coin lands heads k times in n flips. Compare:

  • M1: the coin is fair, theta = 0.5. Then p(y | M1) = C(n, k) * 0.5^n.
  • M2: theta ~ Beta(a, b). Then p(y | M2) = C(n, k) * B(k + a, n - k + b) / B(a, b), where B is the Beta function.

The binomial coefficient cancels in the Bayes factor.

Python Fair coin versus unknown bias

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With exactly 50 heads in 100 flips the fair coin is favored by roughly a factor of 8: the Bayes factor rewards a model for making a sharp prediction that comes true. At 60 heads the evidence is already about even, and at 65 or 70 it clearly favors a biased coin. A model with a wide prior spreads its probability over many datasets, so it pays for the ones that did not happen. This is the built-in Occam's razor of the marginal likelihood.

Bayes factors depend on the prior

The marginal likelihood averages over the prior, so a vague prior wastes probability on parameter values that explain the data badly, and the Bayes factor punishes that. The consequence is that Bayes factors are sensitive to prior width in a way that posterior distributions are not.

Python Widen the prior, watch the evidence change

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The posterior mean barely moves across these priors (it stays near 0.61), yet the Bayes factor moves by a factor of more than three (about 1.1 for a = 0.5, about 0.32 for a = 10, about 0.69 for a = 200) and changes which model it favors, because a prior concentrated near 0.5 makes the biased model predict nearly the same as the fair model. If you use Bayes factors, justify the prior carefully and report a sensitivity analysis. Improper (flat, unnormalized) priors make Bayes factors meaningless because the normalizing constant is arbitrary.

Computing the marginal likelihood in general

Closed forms are rare. Options:

  • Grid or numerical integration for one or two parameters.
  • Simple Monte Carlo: draw theta from the prior and average the likelihood. Works only for very low dimensions, because most prior draws have tiny likelihood.
  • Bridge sampling, nested sampling, thermodynamic integration: dedicated methods for real models.
  • Laplace approximation: a Gaussian around the posterior mode; the basis of BIC.
Python Monte Carlo evidence versus the exact value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a single parameter the plain Monte Carlo average converges quickly. With 20 parameters it would need astronomically many draws, which is why dedicated algorithms exist.

Bayes factors versus predictive comparison

Bayes factors answer "which model was more likely to have generated this data, given my priors?" They are the right tool when you truly entertain discrete hypotheses (one gene is involved or not). When you mainly want to know which model predicts new data well, cross-validation methods such as WAIC and LOO are less sensitive to arbitrary prior widths and are usually preferred in applied work.

A rough reading scale for BF (for or against, symmetric): 1 to 3 barely worth mentioning, 3 to 10 moderate, 10 to 30 strong, above 100 decisive. These are conventions, not laws.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the marginal likelihood p(y | M)?

    It integrates likelihood times prior over all parameter values.
  2. 2. A Bayes factor of 20 in favor of M1 over M2 means...

    A Bayes factor is a ratio of marginal likelihoods; multiply it by prior odds to get posterior odds.
  3. 3. Why is the Bayes factor sensitive to the width of the prior?

    The marginal likelihood averages the likelihood over the prior, so diffuse priors lower it.
  4. 4. Why should you avoid improper priors in Bayes factors?

    An improper prior has no proper normalization, so the marginal likelihood is defined only up to an arbitrary constant.

Summary

  • The marginal likelihood is the prior-averaged probability of the data; ratios of it are Bayes factors.
  • Posterior model odds equal the Bayes factor times prior model odds.
  • Bayes factors reward sharp, correct predictions, which acts as automatic Occam's razor.
  • They depend strongly on the prior width: always run a sensitivity analysis.
  • Computing evidence in many dimensions needs special algorithms; for predictive accuracy prefer WAIC or LOO.

Next: WAIC and Leave-One-Out Cross-Validation