Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 28 ยท Linear Regression in Depth

Regularized Regression

Shrink coefficients with ridge, lasso and elastic net in Statsmodels using fit_regularized. See how the penalty changes estimates and why inference needs care.

When you have many predictors, or predictors that move together, OLS coefficients become unstable: tiny changes in the data swing them wildly. Regularization adds a penalty for large coefficients. You accept a little bias in exchange for much lower variance, and the model often predicts new data better.

Statsmodels provides fit_regularized on OLS (and on GLMs). The penalty is controlled by two numbers:

  • alpha: the strength. 0 means plain OLS; bigger shrinks more.
  • L1_wt: the mix between the two classic penalties. 0 is ridge (squared size of coefficients), 1 is lasso (absolute size), and values between give elastic net.
Penalty L1_wt Effect
Ridge 0 Shrinks all coefficients smoothly; keeps every variable
Lasso 1 Shrinks and sets some coefficients exactly to zero
Elastic net between A compromise, good with correlated predictors
Python Ridge versus lasso on many predictors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lasso zeroes several of the useless predictors and shrinks the rest to tiny values, doing variable selection as part of fitting. Ridge keeps them all but makes them small. Both pull the real coefficients a bit toward zero.

Standardize first

Penalties treat all coefficients equally, so a variable measured in millimeters (huge numbers, tiny coefficient) is penalized unfairly compared with one measured in kilometers. Standardize predictors (mean 0, standard deviation 1) before using any penalty, and never penalize the intercept (Statsmodels does not penalize the constant column when you add it with add_constant; for formulas it handles it too).

Python Choose alpha by looking at a path

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As alpha rises, the useless predictors drop to zero first and the real ones shrink gradually: this is the regularization path. You pick alpha by cross-validation: split the data, fit on one part, score on the rest, and choose the value with the best out-of-sample error. Statsmodels has no built-in cross-validation helper, so you write a loop (below) or use scikit-learn's RidgeCV/LassoCV for tuning.

Python Cross-validate alpha by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error is usually lowest at a small positive alpha: some shrinkage helps, too much hurts.

Caveat: no standard errors

fit_regularized returns a light results object with params but no standard errors, p-values or summary table. Penalized estimates are biased, so classic formulas do not apply. Two common uses:

  1. Use the penalty for prediction or variable selection.
  2. Re-fit plain OLS on the selected variables to get inference. This "post-lasso" refit is easy, but remember the p-values ignore the selection step and are somewhat optimistic.

If predictors are highly correlated, also read Multicollinearity and VIF. Ridge is a standard remedy for that problem.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does L1_wt=1.0 give you in fit_regularized?

    L1_wt=1 means a pure absolute-value (L1) penalty, which is lasso. L1_wt=0 is ridge.
  2. 2. Why standardize predictors before penalizing?

    Without standardizing, variables on small scales receive large coefficients and are penalized unfairly.
  3. 3. What is a feature of lasso that ridge lacks?

    The L1 penalty produces sparse solutions, so lasso performs variable selection.
  4. 4. How should alpha usually be chosen?

    alpha trades bias against variance; cross-validation estimates which value predicts new data best.

Summary

  • Penalties shrink coefficients: ridge (L1_wt=0), lasso (1), elastic net (between).
  • Standardize predictors first; tune alpha with cross-validation.
  • Lasso selects variables; ridge stabilizes correlated ones.
  • fit_regularized gives parameters only, no classic inference.

Next: Residual Diagnostics