Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 36 ยท Model Checking and Comparison

WAIC and Leave-One-Out Cross-Validation

Compare Bayesian models by predictive accuracy: compute the log pointwise predictive density, WAIC and exact leave-one-out scores for polynomial regressions in NumPy.

A model can explain the data it has seen perfectly and still predict new data badly. This is overfitting. To compare models fairly, Bayesian statisticians measure how well each one predicts data it has not been fitted to. Two standard tools are WAIC (a fast approximation) and LOO-CV (leave-one-out cross-validation, the gold standard).

Both use the log pointwise predictive density. For observation i and posterior draws theta^(s), s = 1..S:

TEXT
lppd_i = log( mean over s of p(y_i | theta^(s)) )

Summing over i gives lppd, the in-sample score. It always improves with more flexible models, so it must be corrected for the effective number of parameters.

WAIC

WAIC subtracts a penalty p_waic, the sum over observations of the variance (across draws) of the log likelihood:

TEXT
p_waic = sum_i var_s( log p(y_i | theta^(s)) )
elpd_waic = lppd - p_waic        (higher is better)
WAIC = -2 * elpd_waic            (lower is better)

An experiment with polynomials

We generate 30 points from a smooth curve with noise, then fit polynomial regressions of degree 1 to 8. To keep everything in closed form, we use a known noise standard deviation and a Normal(0, 10^2) prior on each coefficient, so the posterior is exactly Gaussian and we can draw from it directly.

Python WAIC for polynomial degrees

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the lppd column: it improves from degree 1 to 3 and then never gets worse, because flexible curves can only fit the sample better. p_waic keeps rising (the model "uses" more parameters), so elpd_waic peaks at degree 3 and then declines. The best WAIC is at degree 3, which matches the smooth curve that generated the data. p_waic is the effective number of parameters, and it is usually smaller than the raw count because the prior regularizes.

Exact leave-one-out

LOO answers a concrete question: "if I leave out point i, how well does the model fitted to the rest predict it?" Here refitting 30 times is cheap and exact. For each i, we compute the posterior from the other 29 points and evaluate the predictive density of y_i. With a Gaussian posterior and Gaussian noise, the predictive density is also Gaussian: mean x_i^T m, variance sigma^2 + x_i^T C x_i.

Python Exact LOO by refitting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The LOO ranking agrees with WAIC: degree 3 is best and higher degrees get steadily worse. In real models you cannot refit hundreds of times, so software uses Pareto-smoothed importance sampling (PSIS-LOO), which reweights a single set of posterior draws to mimic leaving each point out and reports diagnostics (the Pareto k) for points where this fails. ArviZ's az.loo and az.compare implement it; see PyMC-style workflows.

Comparing models properly

  • Compare elpd differences together with their standard error. A difference smaller than about twice its standard error is not convincing.
  • Information criteria compare models fitted to the same data.
  • High-leverage or influential points show up as large pointwise elpd differences. Inspect them.
  • If several models predict almost equally well, consider averaging their predictions (stacking) instead of picking one winner.
  • WAIC is unreliable with very weak priors or small samples; LOO is more robust.

AIC and BIC are cousins: AIC uses a maximum-likelihood point estimate and a count of parameters, BIC approximates the marginal likelihood. WAIC and LOO use the full posterior, so they work for hierarchical and other singular models where parameter counts are meaningless.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the in-sample lppd always improve with model flexibility?

    lppd is measured on the data used for fitting, so overfitting looks like improvement. WAIC and LOO correct for that.
  2. 2. What is p_waic?

    It measures how much each observation's log likelihood varies across posterior draws and serves as the penalty.
  3. 3. What does LOO-CV evaluate?

    Each observation is predicted from a model that never saw it.
  4. 4. Two models differ in elpd by 1.2 with a standard error of 2.5. What should you conclude?

    A difference well within its uncertainty is not convincing evidence of a better model.

Summary

  • Compare models by out-of-sample predictive accuracy, not by in-sample fit.
  • WAIC = lppd minus an effective-parameter penalty; LOO leaves each point out in turn.
  • On small models you can compute exact LOO by refitting; in practice PSIS-LOO approximates it.
  • Always report differences with standard errors and check pointwise outliers.
  • WAIC and LOO use the full posterior and also work for hierarchical models.

Next: Prior Sensitivity and Simulation-Based Calibration