Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Bayesian Regression

Bayesian Linear Regression

Fit a Bayesian linear regression with a conjugate normal-inverse-gamma prior, get posterior draws for coefficients and noise, and make predictive intervals.

Ordinary least squares gives one line. Bayesian linear regression gives a whole distribution of lines: every combination of intercept, slope and noise level that is consistent with the data and the prior, weighted by plausibility. From that distribution you read credible intervals for each coefficient and, more usefully, predictions that carry honest uncertainty.

The model is the familiar one:

TEXT
y = X beta + noise,    noise ~ Normal(0, sigma^2)

The Bayesian part is that beta and sigma^2 get priors.

A conjugate prior

With the normal-inverse-gamma prior the posterior is available in closed form, so no sampler is needed:

TEXT
sigma^2      ~ Inverse-Gamma(a0, b0)
beta | sigma^2 ~ Normal(m0, sigma^2 * V0)

The posterior has the same form, with updated parameters:

TEXT
Vn = inverse( inverse(V0) + X'X )
mn = Vn * ( inverse(V0) m0 + X'y )
an = a0 + n / 2
bn = b0 + 0.5 * ( y'y + m0' inverse(V0) m0 - mn' inverse(Vn) mn )

To draw from the joint posterior: draw sigma^2 from Inverse-Gamma(an, bn), then draw beta from Normal(mn, sigma^2 * Vn).

flowchart LR
    A["Prior: m0, V0, a0, b0"] --> C["Update with X and y"]
    B["Data X, y"] --> C
    C --> D["Posterior: mn, Vn, an, bn"]
    D --> E["Draw sigma squared, then beta"]

Conjugacy means the update is just arithmetic on a few matrices.

Fit and compare with OLS

We simulate 40 points from y = 1.5 + 2x + noise with noise standard deviation 1.2, and choose a weak prior: m0 = 0, V0 = 100 * I, a0 = b0 = 0.01-ish. With a weak prior, the posterior mean should sit close to the least-squares estimate.

Python Conjugate posterior versus OLS

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The posterior mean of beta is nearly identical to OLS, and the credible intervals resemble classical confidence intervals. That is expected under a weak prior; the Bayesian reading, "given the data and prior, the slope lies in this interval with 95% probability", is the one most people actually want.

Predictive distributions

A prediction for a new x* has two kinds of uncertainty: uncertainty about the line (beta) and the irreducible noise around it. The posterior predictive draws include both: for each posterior draw (beta, sigma), simulate y* = x*' beta + Normal(0, sigma).

Python Mean uncertainty versus prediction uncertainty

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two patterns show up. The interval for a new observation is much wider than the interval for the mean line, because it adds the noise. And at x* = 6, well outside the observed range, the interval for the line is wider than in the middle of the data: uncertainty about the slope grows as you extrapolate. Classical software shows this too, but with posterior draws it takes no special formulas.

The role of the prior

The prior acts like extra data. A covariance V0 = tau^2 I with small tau pulls coefficients toward m0 (usually 0), which is a form of regularization; see shrinkage priors. Useful habits:

  • Standardize predictors so one prior scale fits all coefficients.
  • Pick a prior scale that reflects plausible effect sizes (an effect of 100 standard deviations is implausible).
  • Run a prior predictive check: do regression lines drawn from the prior look reasonable?
  • For small data sets, report results under at least two priors.

When conjugacy ends

The closed-form trick works only for a Gaussian likelihood with a Gaussian-inverse-gamma prior. If you want a heavy-tailed likelihood (Student-t), a half-Cauchy prior on sigma, or a non-Gaussian response, you need MCMC, covered next in Metropolis for regression.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a Bayesian linear regression return that OLS does not?

    It gives a distribution of plausible regression lines, from which credible and predictive intervals follow.
  2. 2. Which draw order samples the joint posterior in the normal-inverse-gamma model?

    The marginal posterior of sigma^2 is inverse-gamma, and beta given sigma^2 is normal.
  3. 3. Why is a predictive interval for a new observation wider than the interval for the mean line?

    The new value equals the line plus noise, so both sources of uncertainty enter.
  4. 4. A weak prior on beta makes the posterior mean close to...

    When the prior carries little information, the data dominate and the posterior mean approaches least squares.

Summary

  • A normal-inverse-gamma prior gives a closed-form posterior for linear regression.
  • Draw sigma^2 first, then beta, to get joint posterior samples.
  • With weak priors the posterior mean matches OLS, but intervals have a direct probability meaning.
  • Predictive intervals include both parameter and noise uncertainty, and widen when extrapolating.
  • Standardize predictors and choose prior scales deliberately.

Next: Metropolis for Regression