Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 36 ยท Bayesian Regression

Metropolis Sampling for Regression

Write a random-walk Metropolis sampler for a regression with a Student-t likelihood, run several chains, check R-hat and effective sample size, and read the posterior.

Conjugate formulas disappear the moment you change a modeling choice, such as swapping the normal likelihood for a Student-t to resist outliers. The general-purpose answer is MCMC. In this lesson you write a complete random-walk Metropolis sampler for a regression, run several chains and judge whether they can be trusted.

The model

A robust regression with a heavy-tailed likelihood:

TEXT
y_i ~ StudentT(nu = 4, mu = a + b x_i, scale = sigma)
a ~ Normal(0, 10^2)
b ~ Normal(0, 10^2)
sigma ~ HalfNormal(5)        (sample log sigma to keep it positive)

The unnormalized log posterior is the sum of the log likelihood and the log priors. We sample (a, b, log sigma); remember the Jacobian: a change of variables s = log sigma adds + s to the log density.

The sampler

Each iteration proposes a new parameter vector by adding Gaussian noise, and accepts it with probability min(1, posterior ratio).

Python Random-walk Metropolis with outliers in the data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The acceptance rate should be in a healthy range, roughly 0.2 to 0.4 for a 3-parameter problem; if it is near 0 or 1 the step size needs tuning. The true line is a = 1.0, b = 0.8. The robust posterior means are about 1.00 and 0.76, while least squares, dragged by the four outliers, gives about 1.55 and 0.58. The Student-t likelihood treats the outliers as plausible tail events instead of letting them shift the line.

Check that the chains agree

One chain can look perfect and still be wrong, for example if it is stuck in a mode. Run several chains from different, spread-out starting points and compare them. The R-hat statistic (potential scale reduction) compares between-chain and within-chain variance; values close to 1 (below about 1.01) indicate agreement. The effective sample size (ESS) estimates how many independent draws the chain is worth after accounting for autocorrelation.

Python Four chains, R-hat and effective sample size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

R-hat should be close to 1 for all three parameters, meaning four chains that started in very different places have found the same distribution. The ESS is much smaller than the number of draws, which is normal for random-walk Metropolis: neighboring draws are strongly correlated. Better samplers (Hamiltonian Monte Carlo) get a much larger ESS per draw, but the diagnostics are read the same way.

Practical advice

  • Warm-up: discard the first part of each chain; a good rule is to look at trace plots and keep only the stationary part.
  • Step size: tune during warm-up so acceptance is near 0.25 to 0.4. Do not adapt after warm-up, because adaptation breaks the Markov property.
  • Parameter scale: sample log sigma, not sigma, to avoid proposals below zero.
  • Correlated predictors: center and scale x; strongly correlated parameters make a random walk crawl.
  • Always run multiple chains and report R-hat and ESS alongside the results.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a Student-t likelihood used in the example?

    Heavy tails make outliers less surprising, so they pull the line much less.
  2. 2. When sampling log sigma instead of sigma, what must you add to the log posterior?

    Changing variables from sigma to s = log sigma multiplies the density by sigma, which adds s on the log scale.
  3. 3. An R-hat of 1.4 means...

    R-hat well above 1.01 indicates between-chain differences that exceed within-chain variation.
  4. 4. What does the effective sample size measure?

    Autocorrelation reduces the information per draw; ESS accounts for that.

Summary

  • Random-walk Metropolis needs only the unnormalized log posterior.
  • A Student-t likelihood gives a regression that resists outliers.
  • Sample positive parameters on the log scale and include the Jacobian.
  • Run several chains from dispersed starts; require R-hat near 1 and a sufficient ESS.
  • Tune step sizes during warm-up only.

Next: Bayesian Logistic Regression