Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท PyMC-style Workflows

A Full Bayesian Workflow: From Prior Predictive to Checks

Follow a complete Bayesian workflow on response-time data: prior predictive checks, a grid fit, fake-data recovery tests, posterior predictive checks and sensitivity analysis.

Experienced Bayesian analysts do not fit a model once and read off the answer. They follow a loop: build a simple model, check the prior, fit, test whether the fit can be trusted, check against the data, then improve and repeat. This lesson walks through the loop on a small problem you can run completely.

flowchart TD
    A["1. Choose a model and priors"] --> B["2. Prior predictive check"]
    B --> C["3. Test on fake data: recover known parameters"]
    C --> D["4. Fit the real data"]
    D --> E["5. Posterior predictive check"]
    E --> F["6. Sensitivity to priors"]
    F -->|"problems found"| A
    F -->|"fine"| G["Report with uncertainty"]

The workflow is a loop, not a line: failed checks send you back to the model.

The problem

You measured the response time (in milliseconds) of a web endpoint 50 times. Times are positive and right-skewed, so we model their logarithm as normal:

TEXT
log(y_i) ~ Normal(mu, sigma)
mu ~ Normal(m0, s0),   sigma ~ HalfNormal(h)

With two parameters we can compute the posterior on a grid: evaluate prior times likelihood at every point of a fine grid over (mu, sigma) and normalize. No sampler to worry about.

Steps 1 and 2: choose priors and check them

Suppose you know typical response times are somewhere between 20 ms and a few seconds. We compare two priors by simulating response times from each and asking how often they produce absurd values (over 10 seconds, or under 1 ms).

Python Prior predictive check on two priors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The vague prior says "response times could be 1e-10 ms or 1e20 ms", and most of its simulated values are absurd. The second prior, centered on exp(5) ms (about 150 ms) with a modest spread, keeps nearly everything in a plausible range. We continue with the second prior. Choosing it before looking at the data is the whole point of the check.

Step 3: can the procedure recover known truths?

Before trusting the fit on real data, test it on data where you know the answer. We simulate datasets from known (mu, sigma), compute the 90% credible interval from the grid, and count how often it covers the truth. For a calibrated analysis, coverage should be close to 90% when the true values come from the prior.

Python Fake-data recovery and interval coverage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Coverage should come out near 0.90 for both parameters, up to sampling noise from only 300 trials. (A first version of this check used a coarse grid that could not resolve very small sigma, and coverage dropped to about 0.8: exactly the kind of bug this test exists to catch. The fix was a finer grid and a prior truncated to a realistic range, used both to draw the truths and to fit.) If it did not (say 0.6), there would be a bug in the likelihood or the grid, and you would fix it before touching real data. This is the cheap cousin of simulation-based calibration.

Steps 4 to 6: fit, check, probe

Now fit the "real" data (we simulate them with a hidden tail the model does not know about: 5 percent of requests hit a slow path). We fit the lognormal model, then run a posterior predictive check on the maximum, and finally repeat the fit with a different prior.

Python Fit, posterior predictive check, sensitivity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The posterior predictive p-value for the maximum is small (about 0.02): the model cannot produce values as slow as the real ones, because a lognormal with a single sigma is not built for a rare slow path. The sensitivity run shows that the problem is not the prior (the answers barely change). The model is wrong in one respect, so the next iteration of the loop would try a mixture or a Student-t on the log scale. Without the check, the single-lognormal summary would have looked perfectly plausible.

A checklist

  • Prior predictive check passed before looking at data.
  • Recovery on fake data (coverage near the nominal level).
  • Sampler diagnostics (R-hat near 1, adequate ESS, no divergences) when using MCMC.
  • Posterior predictive checks on several statistics.
  • Sensitivity to at least one alternative prior.
  • Report the intervals, not only point estimates, and what the model cannot explain.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do a prior predictive check before looking at the data?

    It exposes priors that imply absurd data while you can still change them without bias from the real data.
  2. 2. What does fake-data recovery test?

    If 90% intervals cover the known truth about 90% of the time, the procedure works; otherwise there is a bug.
  3. 3. A posterior predictive check on the maximum fails while the prior sensitivity run changes nothing. What is wrong?

    The data conflict with the model, not with a particular prior, so the model must change.
  4. 4. Which describes the Bayesian workflow?

    Failed checks send you back to model building.

Summary

  • Run prior predictive checks, recovery tests, posterior predictive checks and sensitivity analyses.
  • Grid approximation solves small problems exactly without a sampler.
  • Fake-data tests catch coding errors before they touch real data.
  • Posterior predictive checks reveal missing structure, such as rare slow outliers.
  • Treat modeling as an iterative loop and report uncertainty.

Next: PyMC and ArviZ: The Real Tools