Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 36 ยท Model Checking and Comparison

Posterior Predictive Checks

Learn how to simulate replicated data from the posterior, compare it with the real data using test statistics, and spot overdispersion and other model failures.

Fitting a Bayesian model is not the end of the analysis. The posterior is only as good as the model that produced it, and a wrong model gives confident nonsense. The main tool for catching this is the posterior predictive check (PPC): if the model is a good description of the data, then data simulated from the fitted model should look like the data you actually observed.

The recipe has four steps:

  1. Draw parameters theta^(s) from the posterior.
  2. For each draw, simulate a replicated dataset y_rep^(s) from the likelihood.
  3. Compute a test statistic T (a mean, a maximum, a count of zeros, a variance) on the real data and on every replicate.
  4. Compare T(y) with the distribution of T(y_rep).
flowchart LR
    A["Posterior draws"] --> B["Simulate y_rep"]
    B --> C["Compute T on each y_rep"]
    D["Observed data y"] --> E["Compute T(y)"]
    C --> F["Compare: is T(y) typical?"]
    E --> F

If the observed statistic lands in the far tail of the replicated ones, the model misses something.

A model that is too simple

We have 60 counts of customer complaints per day. A natural first model is Poisson with a Gamma prior on the rate, which is conjugate: with a Gamma(a, b) prior and n counts summing to S, the posterior is Gamma(a + S, b + n).

The data were secretly generated from a negative binomial, so they are overdispersed: the variance is bigger than the mean. A Poisson cannot represent that, and a PPC on the variance should reveal it.

Python PPC for a Poisson model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The posterior predictive p-value is the fraction of replicates whose statistic is at least as extreme as the observed one. A value near 0 or 1 signals misfit. Here it is essentially zero: replicated data from the Poisson model never have as much spread as the real data. Unlike a classical p-value, it is not a formal test with a fixed error rate; treat it as a diagnostic.

Fix the model, check again

An overdispersed count model fixes the problem. A negative binomial can be written as a Gamma-Poisson mixture: each day has its own rate lambda_i ~ Gamma(r, r / mu). For the check we use the marginal negative binomial likelihood and a simple grid posterior over (mu, r).

Python PPC for a negative binomial model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Now all three statistics land comfortably inside the replicated range. Using several statistics matters: the model could reproduce the variance and still fail on the maximum or on the number of zeros.

Choosing test statistics

A good statistic targets a feature that you care about and that the model is not built to fit exactly. Statistics the model fits automatically (the mean in a model with a free mean) tell you nothing. Useful choices include:

  • Variance or index of dispersion for counts.
  • Maximum and minimum, to probe tails and outliers.
  • Proportion of zeros, for zero-inflation.
  • Autocorrelation, for time dependence.
  • Residual summaries by subgroup, to find where the model breaks.

Visual checks

Plots are often more informative than a single number: overlay histograms of a dozen replicated datasets on the observed histogram, or plot the observed empirical CDF inside a band of replicated CDFs. If the real data stand out from the cloud of replicates, investigate.

Limits of PPC

  • The data are used twice, once to fit and once to check, so the check is conservative: it can miss problems, especially with flexible models.
  • Passing a check does not prove the model true; it only says this aspect of the data is reproduced.
  • For small samples a check has little power. Combine it with prior sensitivity analysis and model comparison.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a posterior predictive check compare?

    You simulate y_rep from the fitted model and see whether T(y) is typical among T(y_rep).
  2. 2. The posterior predictive p-value of the variance is 0.001. This suggests...

    Observed variance is larger than almost all replicated variances, so the model cannot generate data that spread out.
  3. 3. Which statistic is a poor choice for a model with a free mean parameter?

    The model is fitted to match the mean, so the check tells you little.
  4. 4. Why are PPCs conservative?

    The fitted model has already adapted to the data, so replicated data tend to resemble it.

Summary

  • Simulate replicated data from the posterior predictive distribution and compare test statistics.
  • A posterior predictive p-value near 0 or 1 flags a feature the model cannot reproduce.
  • Overdispersed counts break the Poisson model; a negative binomial fixes it.
  • Use several statistics aimed at different features, and also look at plots.
  • PPCs are diagnostics, not proofs of model truth.

Next: Marginal Likelihood and Bayes Factors