Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 36 ยท PyMC-style Workflows

Writing a Bayesian Model as Code

Build a tiny probabilistic-programming layer in NumPy and SciPy: declare priors and a likelihood, simulate from the prior, and sample the posterior with Metropolis.

Libraries such as PyMC, Stan and NumPyro let you write a Bayesian model almost the way you would write it on paper: a list of random variables, each with a distribution, and a likelihood tying them to the data. The library then builds the log posterior and runs a sampler. In this chapter you learn that workflow, and to understand what the libraries do for you, you build a tiny version yourself.

A note on what runs here. PyMC and Stan are not available in this site's in-browser Python runner, so the examples use a small stand-in written with NumPy and SciPy. It is labelled as a stand-in throughout, and the last lesson of the chapter shows the real PyMC code as read-only snippets.

What a model object has to do

Every probabilistic-programming tool offers roughly the same few capabilities:

  1. Declare parameters with priors.
  2. Declare a likelihood given the data.
  3. Evaluate the log posterior at any parameter vector (prior plus likelihood).
  4. Simulate from the prior, and from the prior predictive.
  5. Sample the posterior with MCMC.
flowchart LR
    A["Priors"] --> D["log posterior"]
    B["Likelihood + data"] --> D
    D --> E["Sampler"]
    E --> F["Posterior draws"]
    A --> G["Prior predictive"]

The log posterior is the one function everything else uses.

A minimal model class

Our stand-in treats every parameter as a real number internally. Parameters that must be positive (a standard deviation) are sampled on the log scale, with the Jacobian term added automatically. That is exactly what real libraries do with "transforms".

Python A tiny probabilistic-programming stand-in

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what you wrote: two priors, one likelihood line and the flag that sigma is positive. Everything else (the Metropolis loop, the transform) lives in the library, which is exactly the point of probabilistic programming: separate what the model is from how to fit it.

Prior predictive simulation

Because the model knows its priors, simulating data from them is a few lines. This is the first check in any workflow: do the datasets the priors imply look remotely plausible?

Python Prior predictive draws from the model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If your data are reaction times in seconds, a prior that predicts negative values or values near 40 is at odds with what you already know. Then tighten the priors before looking at the real data.

What real libraries add

Our toy sampler is a random walk with a fixed step. Real tools improve on it in important ways:

  • Gradient-based samplers (Hamiltonian Monte Carlo, NUTS) via automatic differentiation: efficient even with hundreds of parameters.
  • Automatic tuning of step size and mass matrix during warm-up.
  • Rich transforms for bounded, simplex and correlation-matrix parameters.
  • Diagnostics: divergences, R-hat, ESS, trace and rank plots.
  • Post-processing: posterior predictive sampling, pointwise log likelihoods for WAIC and LOO.

None of this changes how you think about the model: priors, likelihood, data, posterior.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are the two ingredients of the log posterior in a probabilistic program?

    Up to a constant, log posterior = log prior + log likelihood.
  2. 2. Why does the stand-in sample a positive parameter on the log scale?

    An unconstrained sampler works with any real number; exp() maps it back to a positive value, with a Jacobian correction.
  3. 3. What is a prior predictive simulation?

    It shows what data the model considers plausible before seeing the real data.
  4. 4. What is the main advantage of probabilistic programming languages?

    Separating model specification from fitting lets you change the model without rewriting the sampler.

Summary

  • A probabilistic model is priors plus a likelihood; the log posterior is their sum.
  • Positive parameters are sampled on the log scale with a Jacobian correction.
  • The same model object supports prior simulation and posterior sampling.
  • This stand-in mirrors what PyMC or Stan do, minus gradients, tuning and diagnostics.

Next: From Prior Predictive to Posterior: A Full Workflow