Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 40 ยท Regression

Ridge, Lasso and Elastic Net

Control overfitting in linear models with Ridge, Lasso and ElasticNet: how the alpha penalty works, why scaling matters, and how RidgeCV and LassoCV choose alpha.

Plain linear regression can overfit: with many features, or features that are nearly copies of each other, it chases noise and its coefficients become huge and unstable. Regularization adds a penalty for large coefficients, trading a little bias for a big reduction in variance. scikit-learn offers three versions, all with the same interface as LinearRegression.

Model Penalty Effect
Ridge sum of squared coefficients (L2) Shrinks all coefficients smoothly
Lasso sum of absolute coefficients (L1) Shrinks some coefficients exactly to zero (feature selection)
ElasticNet a mix of L1 and L2 Both effects; copes with correlated features

The strength of the penalty is the hyperparameter alpha: alpha=0 is ordinary regression and larger values mean smaller coefficients.

Overfitting in a picture of numbers

We create 60 samples (30 for training) and 25 features, where only 3 features really matter. Plain regression fits the training set almost perfectly but fails on new data.

Python Linear regression versus Ridge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The big gap between train and test Rยฒ for plain regression is the classic sign of overfitting. Ridge narrows it. See Overfitting and Underfitting for the general idea.

Ridge shrinks every coefficient

Python Coefficients shrink as alpha grows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ridge never makes a coefficient exactly zero; it just squeezes them all toward zero.

Lasso selects features

Python Lasso zeroes out useless features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lasso fits a model and picks the useful columns at the same time, which also makes the model easier to explain. Its weakness: among a group of correlated features, it tends to keep one arbitrarily.

ElasticNet: the compromise

ElasticNet(alpha=..., l1_ratio=...) blends the two. l1_ratio=1 is Lasso, l1_ratio=0 is Ridge, and values between mix them.

Python ElasticNet with correlated features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lasso piles the weight onto one of the copies, while ElasticNet spreads it across the group.

Always scale first

The penalty treats all coefficients alike, so a feature measured in large units gets a small coefficient and is penalised unfairly, while a feature in tiny units is penalised hard. Standardise before using Ridge, Lasso or ElasticNet. A pipeline keeps this tidy:

Python Scaling plus Ridge in a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing alpha with cross-validation

Do not guess alpha. RidgeCV, LassoCV and ElasticNetCV try many values using cross-validation and keep the best, all inside one fit.

Python RidgeCV and LassoCV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Which should you use?

Start with Ridge as a safe default. Use Lasso when you suspect many features are irrelevant and want a sparse, interpretable model. Use ElasticNet when features are correlated and you still want sparsity.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What happens to the coefficients as alpha increases in Ridge?

    A larger penalty pushes coefficients toward zero, though Ridge rarely reaches exactly zero.
  2. 2. Which model can set coefficients exactly to zero?

    The L1 penalty in Lasso produces sparse solutions.
  3. 3. Why scale features before Ridge or Lasso?

    Without scaling, features in small units need big coefficients and are over-penalised.
  4. 4. What does ElasticNet(l1_ratio=1) equal?

    l1_ratio=1 means the penalty is purely L1, which is Lasso.
  5. 5. How can you choose alpha automatically?

    The CV variants cross-validate a range of alpha values during fit.

Summary

  • Regularization penalises large coefficients to reduce overfitting.
  • Ridge shrinks, Lasso shrinks and selects, ElasticNet mixes both.
  • alpha sets the strength; pick it with RidgeCV, LassoCV or ElasticNetCV.
  • Scale features first.

Next: Polynomial Regression