Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 36 ยท Model Checking and Comparison

Prior Sensitivity and Simulation-Based Calibration

Test how much your conclusions depend on the prior and verify that your inference code is correct with simulation-based calibration and rank histograms.

Two questions should follow every Bayesian analysis. First, would a different reasonable prior change my conclusions? Second, is my inference algorithm actually correct? The first is a prior sensitivity analysis. The second can be answered by simulation-based calibration (SBC), a test that works for any model you can simulate from.

Prior sensitivity analysis

Pick several plausible priors (skeptical, vague, optimistic) and refit. If the posterior summary you care about barely moves, the data dominate and the prior is not driving the result. If it moves a lot, say so; the honest statement is "the data are not strong enough to settle this".

We take an experiment with 12 successes in 40 trials and examine how the conclusion "the rate exceeds 25 percent" changes with the prior.

Python Same data, five priors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The flat and Jeffreys priors agree. The skeptical Beta(2, 18) pulls the mean from 0.31 to 0.23 and cuts the probability of exceeding 25 percent from about 0.8 to about 0.36, and the optimistic prior pushes it to 0.92. The strong skeptic (worth 100 pseudo-observations, more than twice the 40 real ones) gives 0.003. With only 40 trials the conclusion depends on the prior, and reporting this table is far more convincing than defending one prior.

Prior predictive checks

You can also test a prior before seeing data: simulate datasets from the prior alone. If a prior on the log-odds Normal(0, 10^2) implies that success rates are almost always 0 or 1, it is not "uninformative"; it is a strong claim.

Python What does a vague prior on log-odds imply?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A prior with standard deviation 10 on the log-odds puts most of its mass on near-certain success or failure. Weakly informative priors such as Normal(0, 1.5) give a much more sensible spread of probabilities.

Simulation-based calibration

Suppose you wrote your own sampler, or you are unsure about a complicated model. SBC tests the whole pipeline with this procedure:

  1. Draw a "true" parameter theta* from the prior.
  2. Simulate data y from the model given theta*.
  3. Run your inference to get posterior draws for theta.
  4. Record the rank of theta* among the posterior draws.
  5. Repeat many times.

If everything is correct, the posterior is calibrated on average, so the ranks are uniformly distributed. A rank histogram that is U-shaped means the posterior is too narrow (overconfident); a hump means too wide; a lean to one side means bias.

flowchart LR
    A["theta* from prior"] --> B["simulate y"]
    B --> C["fit: posterior draws"]
    C --> D["rank of theta*"]
    D --> E["repeat: ranks uniform?"]

The loop shows one SBC replication; uniform ranks across many replications are the pass condition.

We test two samplers for the beta-binomial model. One is the exact posterior. The other has a deliberate bug: it adds the number of trials n instead of the number of failures n - k to the second Beta parameter. We use 3000 replications and 19 posterior draws per replication, giving 20 possible ranks.

Python SBC for a correct and a buggy sampler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 20 rank bins, a statistic near 19 is what a uniform histogram produces by chance. The buggy sampler's posterior sits too low, so the true value usually ranks near the top: its histogram is piled up in the high ranks and its chi-square value is far above 19. The same idea scales to MCMC code: if your SBC ranks are not uniform, either the sampler or the data-generating code has a bug.

Practical notes

  • With MCMC, thin or run long enough that draws are nearly independent; otherwise ranks inherit autocorrelation and look non-uniform.
  • Test the prior you actually use. SBC checks consistency between the model's prior, its likelihood and your inference code.
  • SBC does not tell you whether the model fits real data; that is the job of posterior predictive checks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the goal of a prior sensitivity analysis?

    If conclusions change with reasonable priors, the data alone are not decisive and you should say so.
  2. 2. In SBC, if everything is correct, how are the ranks of the true parameter distributed?

    A calibrated posterior places the true value at a uniformly random position among its draws.
  3. 3. A U-shaped rank histogram suggests the posterior is...

    The truth often falls in the tails of an overconfident posterior, producing many extreme ranks.
  4. 4. A Normal(0, 10^2) prior on the log-odds of a probability is...

    On the probability scale it piles mass at the extremes. Prior predictive simulation reveals this.

Summary

  • Refit with several reasonable priors and report how the summary changes.
  • Simulate from the prior (prior predictive checks) to see what it really implies.
  • SBC draws truths from the prior, simulates data, fits, and checks that the ranks of the truths are uniform.
  • Non-uniform ranks reveal bugs, bad samplers or mismatched priors.
  • SBC checks the computation, PPCs check the model against real data.

Next: Bayesian Linear Regression