Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 38 ยท Markov Chain Monte Carlo

Gibbs Sampling

Sample a joint distribution by cycling through full conditionals. Build a Gibbs sampler for a bivariate normal and see why correlation slows it.

Metropolis-Hastings proposes a move and sometimes rejects it. Gibbs sampling never rejects. Instead of moving all coordinates at once, it updates one coordinate at a time by drawing it from its full conditional distribution: the distribution of that variable given the current values of all the others. Each draw comes straight from the exact conditional, so every update is accepted.

Gibbs sampling is ideal when the joint distribution p(x, y) is hard to sample from, but each conditional p(x | y) and p(y | x) is a familiar distribution such as a normal, gamma or beta. It is popular in Bayesian statistics for exactly that reason.

The algorithm

For two variables (x, y):

  1. Start with any (xโ‚€, yโ‚€).
  2. Draw xโ‚œโ‚Šโ‚ ~ p(x | y = yโ‚œ).
  3. Draw yโ‚œโ‚Šโ‚ ~ p(y | x = xโ‚œโ‚Šโ‚).
  4. Repeat. The pairs (xโ‚œ, yโ‚œ) form a Markov chain with stationary distribution p(x, y).

With d variables you cycle through all d conditionals. Gibbs is a special case of Metropolis-Hastings in which the acceptance probability is always exactly 1.

flowchart LR
  A["x0, y0"] --> B["draw x from p(x | y0)"]
  B --> C["draw y from p(y | x1)"]
  C --> D["draw x from p(x | y1)"]
  D --> E["... repeat"]

Each coordinate is refreshed in turn, always using the newest values of the others.

Example: a correlated normal

For a standard bivariate normal with correlation ฯ, both conditionals are normal:

Text
x | y ~ Normal( ฯยทy,  1 โˆ’ ฯยฒ )
y | x ~ Normal( ฯยทx,  1 โˆ’ ฯยฒ )

The first conditional has mean ฯยทy and variance 1 โˆ’ ฯยฒ. We take ฯ = 0.8, run 20,000 iterations, and check the sample moments against the truth: means 0, variances 1, correlation 0.8.

Python Gibbs sampler for a bivariate normal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scatter forms a tilted ellipse, and the printed numbers should be close to the truth. The bad starting point (5, โˆ’5) is forgotten within a few steps, so the burn-in of 500 is generous.

Why strong correlation slows Gibbs down

Gibbs moves only parallel to the axes. If x and y are strongly correlated, the probability mass lies along a thin diagonal strip, and axis-parallel steps must be tiny to stay inside it. The chain zigzags and mixes slowly.

For this bivariate normal there is a neat exact result: the lag-1 autocorrelation of the x chain equals ฯยฒ. At ฯ = 0.8 that is 0.64; at ฯ = 0.99 it is 0.98, so successive samples are almost copies of each other.

Python Autocorrelation grows with correlation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where Gibbs shines, and tricks

  • Bayesian hierarchical models. Many models have conjugate pieces: a normal likelihood with a normal prior gives a normal conditional for the mean, and a gamma prior on the precision gives a gamma conditional. Gibbs strings these together.
  • Blocking. Update correlated variables together as a block from their joint conditional; this cures slow mixing.
  • Reparametrization. Centering or rescaling variables can reduce correlation.
  • Scan order. Cycling in fixed order or in random order both work.
  • Mixing with Metropolis. If one conditional is not recognizable, use a Metropolis step for that coordinate and Gibbs for the rest ("Metropolis within Gibbs").

Gibbs sampling is named after the physicist Josiah Willard Gibbs, because its first use (by Stuart and Donald Geman in 1984, for image restoration) drew on ideas from statistical mechanics.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In Gibbs sampling, each update draws a coordinate from:

    Each variable is refreshed from p(variable | all the others), using their current values.
  2. 2. The acceptance probability of a Gibbs update is:

    Gibbs is Metropolis-Hastings with the conditional as proposal, which gives acceptance probability exactly 1.
  3. 3. For a bivariate normal with correlation 0.95, Gibbs sampling mixes:

    The lag-1 autocorrelation of x is rho squared, about 0.90, so samples are highly dependent.
  4. 4. How can slow mixing from correlated variables be reduced?

    Drawing correlated variables jointly lets the chain move along the diagonal.

Summary

  • Gibbs sampling updates one variable at a time from its full conditional; every draw is accepted.
  • It shines when conditionals are standard distributions, as in conjugate Bayesian models.
  • For a bivariate normal the conditionals are N(ฯy, 1 โˆ’ ฯยฒ) and N(ฯx, 1 โˆ’ ฯยฒ).
  • Strong correlation slows it (lag-1 autocorrelation ฯยฒ here); blocking and reparametrization help.

Next: MCMC Diagnostics