Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 ยท Bayesian Regression

Shrinkage Priors: Ridge and Lasso as Bayesian Models

See ridge regression as a Gaussian prior and lasso as a Laplace prior, choose the prior scale with cross-validation, and compare posterior means with maximum likelihood.

When a regression has many predictors and not much data, ordinary least squares overfits: coefficient estimates become huge and unstable. A Bayesian puts a prior on the coefficients that says "most effects are modest", and the posterior then shrinks the estimates toward zero. Two famous penalized methods fall out as special cases.

Ridge is a Gaussian prior

With a Gaussian likelihood and the prior beta_j ~ Normal(0, tau^2), the log posterior is, up to constants:

TEXT
-(1 / (2 sigma^2)) * sum (y - X beta)^2  -  (1 / (2 tau^2)) * sum beta_j^2

Maximizing it is exactly ridge regression with penalty lambda = sigma^2 / tau^2. Because the prior is Gaussian, the whole posterior is Gaussian, and the posterior mean equals the ridge estimate: (X'X + lambda I)^-1 X'y. Ridge is not just a point estimate; the Bayesian view also gives uncertainty.

Lasso is a Laplace prior

Use the double-exponential prior p(beta_j) proportional to exp(-|beta_j| / b) instead. The mode of the posterior is the lasso estimate, and because the Laplace density has a sharp peak at zero, the mode often sits at exactly zero for weak predictors. A caution: the posterior mean under a Laplace prior is not exactly sparse, only the mode is. If you want sparse posterior behavior, look at horseshoe-type priors.

flowchart LR
    A["Gaussian prior"] --> B["Ridge: shrink smoothly"]
    C["Laplace prior"] --> D["Lasso: mode can be exactly 0"]
    E["Horseshoe prior"] --> F["Shrink noise hard, keep large effects"]

Different prior shapes encode different beliefs about how many effects are large.

Ridge as a posterior mean

We create 30 observations with 20 predictors, only 4 of which matter. Then we compare OLS and the Bayesian (ridge) posterior mean on fresh test data across several prior scales.

Python Ridge: prior scale versus test error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

OLS with 20 predictors and 30 points fits noise: its test RMSE is about 1.38 although the true noise level is 1.0. Moderate shrinkage (tau around 0.5 to 1) lowers it to about 1.27. Too much shrinkage (tau = 0.1) squeezes the real effects and the error climbs to about 2.3. This U-shape is why the prior scale matters, and why it should be chosen from data (cross-validation, empirical Bayes) or from a hyperprior: a fully Bayesian model gives tau its own prior and learns it.

Lasso: a Laplace prior and its mode

The lasso solution has no closed form, but a short algorithm does it: ISTA (iterative soft-thresholding). Each step moves along the gradient of the squared-error part, then applies the soft-threshold sign(z) * max(|z| - t, 0), which zeros small values.

Python Lasso (Laplace prior mode) with soft-thresholding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As lambda grows, more coefficients are set exactly to zero. Good settings keep the four true effects and drop most of the 16 noise predictors. This is the "automatic variable selection" people like about the lasso. Remember what the Bayesian reading is: it is the posterior mode under a Laplace prior.

Beyond ridge and lasso

  • Horseshoe and regularized horseshoe: a heavy-tailed prior with a spike at zero; it shrinks noise strongly while leaving large coefficients almost untouched. Needs MCMC.
  • Spike-and-slab: each coefficient is exactly zero with some probability, otherwise drawn from a wide prior; gives posterior inclusion probabilities.
  • Hierarchical shrinkage: let a shared tau learn how strong shrinkage should be, as in the hierarchical models of the previous chapter. In fact, shrinking regression coefficients toward zero is the same idea as shrinking group means toward a population mean.
  • Always standardize predictors; shrinkage is not scale invariant.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Ridge regression corresponds to which prior on the coefficients?

    A Gaussian prior makes the log posterior a squared-error term plus a squared penalty on the coefficients.
  2. 2. The penalty lambda in ridge relates to the Bayesian prior as...

    A smaller prior standard deviation tau means a stronger penalty.
  3. 3. Why can the lasso set coefficients exactly to zero?

    The absolute-value penalty creates a corner at zero where the optimum can sit.
  4. 4. What happens to test error as the ridge prior becomes extremely tight?

    Too much shrinkage introduces large bias, so the error follows a U-shaped curve.

Summary

  • A Gaussian prior on coefficients gives ridge regression; its posterior mean equals the ridge estimate.
  • A Laplace prior gives the lasso as the posterior mode, which can be exactly sparse.
  • The prior scale controls the bias-variance trade-off; choose or learn it from data.
  • Horseshoe and spike-and-slab priors are more flexible shrinkage priors, fitted with MCMC.
  • Standardize predictors before applying shrinkage priors.

Next: Writing a Model as Code