Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 36 · MCMC and Sampling

Gibbs Sampling

Learn Gibbs sampling: update one parameter at a time from its full conditional distribution, with a bivariate normal example and a normal model with unknown mean and variance.

Metropolis proposes a move and then decides whether to accept it. Gibbs sampling skips the decision: it updates one parameter at a time by drawing directly from that parameter's distribution given the current values of all the others. Every draw is accepted, which is wonderful whenever those conditional distributions are easy to sample.

The idea

Take two parameters (θ₁, θ₂). Start at some values, then repeat:

  1. Draw θ₁ from p(θ₁ | θ₂, data).
  2. Draw θ₂ from p(θ₂ | θ₁, data).

Each distribution p(θᵢ | everything else) is called the full conditional. Cycling through them produces a Markov chain whose stationary distribution is the joint posterior. With conjugate priors the full conditionals are often standard distributions, which is why Gibbs sampling powered classic software such as BUGS and JAGS.

flowchart LR
    A["θ₁, θ₂ (current)"] --> B["Draw θ₁ from p(θ₁ | θ₂)"]
    B --> C["Draw θ₂ from p(θ₂ | new θ₁)"]
    C --> D["Record (θ₁, θ₂)"]
    D --> A

Always use the freshest value of the other parameters.

Example 1: a correlated pair

Suppose the target is a bivariate normal with correlation ρ: both marginals are standard normal. The full conditionals are normal:

θ₁ | θ₂ ~ Normal(ρ θ₂, 1 − ρ²) and θ₂ | θ₁ ~ Normal(ρ θ₁, 1 − ρ²)

Python Gibbs sampler for a bivariate normal

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sampler recovers the correlation without ever evaluating a density.

When parameters are strongly correlated

Gibbs moves along the coordinate axes, one at a time. If the target is a thin diagonal ridge, each axis-aligned step can only move a little before leaving the ridge, so mixing slows down badly.

Python High correlation slows Gibbs down

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With ρ = 0.99 successive draws are nearly identical, and even lag-10 draws stay highly correlated. This is the main practical weakness of Gibbs sampling; reparameterizing so parameters are less correlated, or using Hamiltonian Monte Carlo, helps.

Example 2: a normal model with unknown mean and variance

For normal data with unknown μ and σ², use the priors μ ~ Normal(m₀, s₀²) and σ² ~ Inverse-Gamma(a₀, b₀). The full conditionals are conjugate:

  • μ | σ², y ~ Normal(μₙ, τₙ²) where 1/τₙ² = 1/s₀² + n/σ² and μₙ = τₙ² (m₀/s₀² + Σy/σ²): the Normal-Normal update from earlier.
  • σ² | μ, y ~ Inverse-Gamma(a₀ + n/2, b₀ + Σ(yᵢ − μ)²/2).

An Inverse-Gamma draw is the reciprocal of a Gamma draw with the same shape and rate b.

Python Gibbs for (mu, sigma^2)

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

No accept/reject step, no tuning of step sizes. The marginal posterior for μ is close to the Student-t answer from the Normal-Normal lesson (interval roughly 4.6 to 5.9, with this vague prior).

Metropolis within Gibbs

Often some conditionals are standard and others are not. You can mix the two: draw from the standard ones directly, and use a Metropolis step for the awkward parameter, still one coordinate at a time. This hybrid is common in practice and is valid.

Pros and cons

Gibbs Metropolis
Tuning None Step size
Acceptance Always 100% Must tune
Needs Easy full conditionals Only the density
Weakness Slow with correlated parameters Slow in high dimensions

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. In Gibbs sampling, what is drawn at each step?

    Each parameter is updated from p(parameter | all other parameters, data).
  2. 2. What is the acceptance rate of Gibbs sampling?

    There is no accept/reject step: each conditional draw is kept.
  3. 3. Why does Gibbs sampling mix slowly when two parameters are strongly correlated?

    Each coordinate update stays close to the current value, so exploring a narrow ridge takes many small steps.
  4. 4. How do you draw from an Inverse-Gamma(a, b) distribution with NumPy?

    If X is Gamma(a, rate b), then 1/X is Inverse-Gamma(a, b); NumPy's scale parameter is 1/b.

Summary

  • Gibbs sampling cycles through parameters, drawing each from its full conditional given the rest.
  • Every draw is accepted and no step size needs tuning.
  • Conjugate priors give standard full conditionals (Normal for the mean, Inverse-Gamma for the variance).
  • Strong correlations between parameters slow it down.
  • Metropolis-within-Gibbs combines both when only some conditionals are standard.

Next: MCMC Diagnostics