Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 36 ยท Hierarchical Models

Gibbs Sampling for a Hierarchical Model

Write a Gibbs sampler for the hierarchical normal model with full conditionals, run it in NumPy, summarize group effects and the between-group spread tau.

When the hierarchy has an unknown mu and tau, the posterior has no closed form, but it has something almost as good: every parameter has a simple full conditional distribution given all the others. A Gibbs sampler cycles through those conditionals and draws from each in turn. The resulting chain visits (theta_1..theta_J, mu, tau) in proportion to their joint posterior. If Gibbs sampling is new to you, review it in MCMC and sampling.

The model and its conditionals

TEXT
y_j | theta_j     ~ Normal(theta_j, s_j^2)     s_j known
theta_j | mu, tau ~ Normal(mu, tau^2)
mu                ~ flat
tau^2             ~ Inverse-Gamma(a0, b0)

The three conditional updates are:

  1. Each theta_j: a precision-weighted normal. Its mean is (y_j/s_j^2 + mu/tau^2) / (1/s_j^2 + 1/tau^2) and its variance is 1 / (1/s_j^2 + 1/tau^2).
  2. mu: Normal(mean(theta), tau^2 / J).
  3. tau^2: Inverse-Gamma(a0 + J/2, b0 + sum((theta_j - mu)^2) / 2).

An inverse-gamma draw is 1 / Gamma(shape, scale = 1/rate), which NumPy can do.

flowchart LR
    A["theta_j given mu, tau, y"] --> B["mu given theta, tau"]
    B --> C["tau squared given theta, mu"]
    C --> A

One Gibbs sweep: update the groups, then mu, then tau, then repeat.

The sampler

Python Gibbs sampler for six clinics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the last two lines. The posterior means are pulled toward mu, and the clinic with the huge raw value of 28 is pulled the most. These numbers come from random draws, so they agree with a grid answer only up to Monte Carlo error; always run enough iterations that this error is small.

Using the draws

Posterior draws let you ask questions without any algebra. What is the probability that clinic 1 is truly better than clinic 2? What is the probability that clinic 1 is the best of all six? You simply count.

Python Questions answered by counting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how cautious the answers are. Clinic 1 has the highest raw mean, yet the probability it is truly the best is far from 1, because its standard error is large and the hierarchy pulls it back.

Diagnosing the chain

Gibbs samplers can be slow when parameters are strongly correlated, and hierarchical models have a well-known trouble spot: when tau is small, the theta_j and tau lock each other in place. Signs of trouble:

  • The trace of tau has long stretches near zero.
  • The autocorrelation of tau decays slowly.
  • Different starting points give different answers (check with several chains, see convergence diagnostics).

The next lesson, non-centered parameterization, shows why this happens and a standard fix.

Choosing a prior for tau

The Inverse-Gamma(1, 1) on tau^2 used above is a convenience for Gibbs, not a recommendation. Compare: the grid posterior in the first lesson of this chapter had a mean tau near 8, while this sampler gives a median near 1. The prior puts almost all its mass on small tau^2 values, so it pulls every clinic strongly toward mu. This is a clear case of a prior that matters. A flat prior on tau^2 or an Inverse-Gamma(0.001, 0.001) can behave badly when there are few groups. Half-normal or half-Cauchy priors on tau are the usual weakly informative choice, but they are not conjugate, so they need Metropolis steps or a different sampler. Always report how the answer changes with the prior.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a full conditional distribution?

    Gibbs sampling draws each parameter from its full conditional in turn.
  2. 2. In the hierarchical normal Gibbs sampler, mu given theta and tau^2 is drawn from...

    With a flat prior on mu, its conditional is centered at the average of the thetas with variance tau^2 / J.
  3. 3. How do you compute P(clinic 1 is the best) from posterior draws?

    Any event probability is the fraction of draws in which the event happens.
  4. 4. Why can Gibbs mix slowly when tau is near zero?

    When tau is small the thetas must be close to mu, and when the thetas are close, tau stays small, so the chain moves slowly.

Summary

  • A hierarchical normal model has simple full conditionals for each theta_j, mu and tau^2.
  • A Gibbs sampler cycles through them; discard a burn-in, then summarize the rest.
  • Posterior draws turn hard questions (best group, differences, thresholds) into counting.
  • The prior on tau matters when groups are few; check sensitivity.
  • Small tau slows Gibbs down, which motivates a reparameterization.

Next: Non-Centered Parameterization