Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 111 ยท Regression

Ridge, Lasso and Elastic Net

Stop overfitting with penalized regression: how Ridge and Lasso shrink coefficients, how alpha works, why Lasso selects features, and how to tune with RidgeCV and LassoCV.

Plain least squares has one goal: make the training error as small as possible. With many features, few examples or correlated columns, that goal produces huge, unstable coefficients that fit the noise. Regularization adds a second goal: keep the coefficients small. The model must now balance fitting the data against staying simple, and that trade almost always improves predictions on new data.

This lesson covers the three standard penalized linear models in scikit-learn: Ridge (L2 penalty), Lasso (L1 penalty) and Elastic Net (a mix). You will see how the strength alpha shrinks the weights, why Lasso sets some of them to exactly zero and therefore selects features, why you must scale features first, and how RidgeCV and LassoCV choose alpha for you.

Fit the data, but stay modest

Data term Squared error on the training set.
Penalty term A cost on large coefficients.
Alpha How much the penalty counts.

loss = squared error + alpha * penalty(weights)

1 / 3

The idea: penalize big weights

Ordinary least squares minimizes the sum of squared errors. The penalized versions add a term based on the weights w (not the intercept):

TEXT
Ridge:        sum of squared errors + alpha * sum(w ** 2)
Lasso:        mean squared error / 2 + alpha * sum(abs(w))
Elastic Net:  both penalties, mixed by l1_ratio

alpha is a hyperparameter you choose: zero gives ordinary least squares, and larger values mean heavier shrinkage until all the weights collapse toward zero and the model predicts roughly the mean. (scikit-learn scales the data term differently for Lasso and Elastic Net than for Ridge, so equal alpha values are not comparable across the families.) Shrinking trades a little extra bias for a large drop in variance, an idea developed in Overfitting, Underfitting and Bias-Variance.

Ridge: shrink everything

Ridge squares the weights in the penalty, so large weights are punished heavily and small ones barely at all. The result is that all coefficients get smaller and correlated features share the credit instead of getting huge opposite weights, which fixes the instability described in Multiple Linear Regression.

Python Ridge: coefficients shrink as alpha grows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At a tiny alpha, the two near-duplicate features (w0 and w1) split their shared effect very unevenly (about -0.09 and 2.70). As alpha grows they converge to similar values (1.16 and 1.25 at alpha = 10) and everything shrinks smoothly toward zero without ever reaching it. The plot of coefficients against alpha is called a regularization path.

Lasso: shrink and select

Lasso uses the absolute value of weights as the penalty. Geometrically this has a sharp corner at zero, and the optimization tends to land exactly on it: Lasso sets some coefficients to exactly zero, which removes those features from the model. It performs feature selection as part of training, which makes the final model smaller and easier to read.

Python Lasso zeroes out useless features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ordinary least squares keeps all 20 weights, including 17 that are pure noise, and has a test MSE of about 3.3. As alpha rises, Lasso drops noise features one by one; at alpha = 0.5 it keeps exactly the three informative columns (0, 1 and 2) with a test MSE of about 2.5. Push alpha to 3 and it removes real signal too, and the error explodes to about 21: the usual U-shaped tradeoff.

Note: If two features are strongly correlated, Lasso tends to keep one and drop the other, and which one it keeps can change with small changes in the data. For stable selection among correlated features, use Elastic Net.

Elastic Net: both penalties

Elastic Net adds both penalties, with l1_ratio setting the mix: 1.0 is pure Lasso, values near 0 behave like Ridge. It keeps the sparsity of Lasso while sharing weight between correlated features like Ridge, so it is a safe choice when you have many correlated features and want some selection too.

Python Ridge, Lasso and Elastic Net on correlated features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three share the effect across the four correlated columns, but they treat the noise columns differently. Ridge leaves small nonzero weights on all four noise columns. Lasso sets all four to exactly zero. Elastic Net sits in between, zeroing some and leaving tiny values on others. With stronger correlation or a larger alpha, Lasso would start keeping one column of the group and dropping the rest, which Elastic Net softens.

Scaling is required

The penalty is applied to the raw coefficient values, so it treats a weight on "distance in meters" and a weight on "distance in kilometers" very differently: the feature measured in small units needs a large weight and gets punished more. Regularized models therefore must be fitted on standardized features. Put a StandardScaler in a pipeline so that it is fitted on training data only (Pipelines). The intercept is never penalized.

Python Why unscaled features break the penalty

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both features have the same real effect: one standard deviation of either moves the target by roughly 2.7 to 3 units. Yet with alpha = 1000 and no scaling, Ridge keeps all of the meters effect and only about 39 percent of the kilos effect, simply because kilos needs a numerically large weight and meters a tiny one. With scaling, both features are shrunk to the same 25 percent, as they should be. At a huge alpha the unscaled model has effectively deleted kilos while leaving meters untouched.

Choosing alpha with cross-validation

Do not guess alpha. RidgeCV, LassoCV and ElasticNetCV try a range of values with cross-validation and keep the best one (Cross-Validation explains the method). RidgeCV takes an explicit list of alphas; LassoCV builds its own grid automatically.

Python RidgeCV and LassoCV pick alpha for you

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

LassoCV beats plain least squares here (about 12.7 against 14.1) and also reports a shorter list of features, though with this much noise it keeps 16 of 30 rather than only the 4 real ones. RidgeCV is only on par with OLS in this example, a reminder that regularization helps most when the data is scarce or noisy relative to the number of features. Remember that the alpha picked by cross-validation was chosen using the training data only; the test set was used once, for the final comparison.

Choosing between them

Situation Reasonable first choice
Many features that all matter a little, some correlated Ridge
Many features, you believe only a few matter, want a smaller model Lasso
Correlated groups plus a wish for sparsity Elastic Net
More features than rows Ridge or Elastic Net (OLS is not even unique)

Regularization is not only for linear regression: logistic regression and linear SVMs use the same L1 and L2 penalties (their strength is the inverse parameter C, see Logistic Regression). Lasso-based selection also appears as SelectFromModel in Feature Selection.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a larger alpha do in Ridge regression?

    alpha multiplies the penalty on the weights. A bigger alpha forces smaller weights, trading some bias for lower variance.
  2. 2. Which model can set coefficients to exactly zero?

    The L1 penalty has a corner at zero, so Lasso solutions often land exactly on it. Ridge only shrinks weights toward zero.
  3. 3. Why should features be standardized before Ridge or Lasso?

    A feature measured in tiny units needs a large weight and would be penalized unfairly. Standardizing puts all weights on the same footing.
  4. 4. You have several strongly correlated features and want some feature selection. Which is a sensible choice?

    Elastic Net shares weight among correlated features like Ridge but still zeroes unneeded ones like Lasso. Ridge does not select, and alpha = 0 is just ordinary least squares.
  5. 5. How should alpha be chosen?

    Tuning on the test set leaks information. Cross-validation inside the training data picks alpha fairly and leaves the test set for one final check.

Summary

  • Regularization adds a penalty on coefficient size, trading a little bias for much lower variance.
  • Ridge (L2) shrinks all weights and splits credit among correlated features; Lasso (L1) can zero weights out and so selects features; Elastic Net mixes both.
  • alpha sets the strength; zero means ordinary least squares, and very large values flatten the model toward predicting the mean.
  • Always standardize features first, inside a pipeline.
  • Pick alpha with RidgeCV, LassoCV or ElasticNetCV, and keep the test set for a final, single check.

Next, move from predicting numbers to predicting classes with Logistic Regression.