Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Bayesian Regression

Bayesian Logistic Regression

Fit a Bayesian logistic regression with a Gaussian prior, approximate the posterior with the Laplace method, check it against MCMC, and get calibrated predicted probabilities.

When the outcome is yes or no (clicked, defaulted, recovered), the standard model is logistic regression. The probability of success is a sigmoid of a linear predictor:

TEXT
y_i ~ Bernoulli(p_i),    logit(p_i) = X_i beta
beta ~ Normal(0, tau^2 I)

Unlike the normal linear model, there is no conjugate prior for the logistic likelihood, so the posterior has no closed form. We use two standard tools: the Laplace approximation (fast, a Gaussian around the mode) and MCMC (slower, exact in the limit). Comparing them is instructive.

Laplace approximation

Find the posterior mode beta_hat by maximizing the log posterior (Newton's method works well here), then approximate the posterior as Normal(beta_hat, H^-1) where H is the negative Hessian of the log posterior at the mode.

For logistic regression with a Normal(0, tau^2) prior:

TEXT
gradient = X'(y - p) - beta / tau^2
H        = X' W X + I / tau^2,      W = diag(p (1 - p))
Newton step: beta <- beta + inverse(H) * gradient
Python Laplace approximation by Newton's method

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mode and standard deviations give you intervals in milliseconds. The approximation is excellent when the data set is large and the posterior is nearly Gaussian, and weaker when data are sparse or the classes are nearly separable.

Checking against MCMC

To see how good the Laplace shortcut is, we sample the exact posterior with Metropolis and compare. We reuse the Laplace covariance as a proposal shape (scaled by 2.38 / sqrt(d)), which makes the sampler efficient.

Python Laplace versus Metropolis

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For this sample size the two methods agree closely, so the Laplace approximation is a legitimate shortcut. With fewer observations, or with skewed posteriors, the MCMC answer is the one to trust.

Predicted probabilities with uncertainty

The best reason to go Bayesian is prediction. For a new case x*, each posterior draw gives a probability sigmoid(x*' beta); the spread of these probabilities shows how sure the model is. Averaging them gives the posterior predictive probability, which is different from plugging in the mode: the average is pulled toward 0.5 when the parameter uncertainty is large.

Python Plug-in versus posterior predictive probability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the strong-signal case the plug-in probability is 0.99, but the posterior mean is about 0.98 with an interval reaching down to about 0.91: the model admits it is not certain about the coefficients. In the extreme case everything is saturated near 1 and the difference vanishes. If predictions drive decisions (loans, medical alerts), this humility is valuable.

Practical notes

  • Standardize predictors and use priors like Normal(0, 1) to Normal(0, 2.5) on coefficients. This is a mild regularizer that also prevents infinite coefficients under perfect separation, where maximum likelihood fails.
  • Interpret coefficients on the log-odds scale: exp(beta) is an odds ratio.
  • Check calibration: among cases predicted at 0.3, about 30% should be positives.
  • Run posterior predictive checks on subgroup rates; see posterior predictive checks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is there no closed-form posterior for logistic regression?

    Unlike the Gaussian linear model, the Bernoulli-logit likelihood has no conjugate form, so we approximate or sample.
  2. 2. The Laplace approximation uses...

    It approximates the log posterior by a quadratic around the mode.
  3. 3. Which situation is hardest for maximum likelihood but easy with a weakly informative prior?

    Under separation the MLE diverges; the prior keeps coefficients finite.
  4. 4. Why can the posterior mean probability differ from the plug-in probability?

    The sigmoid is nonlinear, so averaging over parameter uncertainty moderates extreme predictions.

Summary

  • Bayesian logistic regression puts a Gaussian prior on log-odds coefficients; the posterior is not closed-form.
  • The Laplace approximation (Newton's method plus the Hessian) is fast and often accurate for moderate data.
  • MCMC is the reference answer; use a Laplace-shaped proposal to make Metropolis efficient.
  • Posterior predictive probabilities carry parameter uncertainty and are less overconfident.
  • Standardize inputs and use weak priors to avoid separation problems.

Next: Shrinkage Priors: Ridge and Lasso