Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 40 ยท Regression

Polynomial Regression

Fit curves with linear models: combine PolynomialFeatures and LinearRegression in a pipeline, choose the degree, see overfitting, and tame it with Ridge.

A straight line cannot follow a curve. But you do not need a new algorithm to fit one: if you add columns such as xยฒ and xยณ, an ordinary linear model can combine them into a curve. The model is still linear in its coefficients, which is why it is called polynomial regression and why it uses the same LinearRegression class.

flowchart LR
    A["x"] --> B["PolynomialFeatures<br/>1, x, xยฒ, xยณ"]
    B --> C["LinearRegression"]
    C --> D["curve"]

Expand the features, then fit a line in the expanded space.

A curve a line cannot fit

Python Line versus quadratic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The true curve was 0.5xยฒ - x + 2, and the quadratic model recovers it closely. include_bias=False drops the constant column, because LinearRegression already has an intercept.

See the fit

Python Plot degrees 1, 2 and 10

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Degree 1 underfits (too stiff), degree 2 matches the shape, and degree 10 wiggles through every point, a sign of overfitting.

Choosing the degree

Training error always falls as you add degrees, so you must judge on data the model has not seen. Compare train and test (or cross-validation) scores over a range of degrees.

Python Train versus test score by degree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Train Rยฒ keeps rising, but test Rยฒ stops improving after degree 2 and collapses at degree 12. Choose the simplest degree where test performance levels off. Validation curves automate this comparison.

Keep high degrees in check with Ridge

High-degree features are huge and highly correlated, so their coefficients explode. Add scaling and a Ridge penalty to stabilise them.

Python Degree 12 with and without regularization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The regularized model is much steadier on unseen data. Read more in Ridge, Lasso and Elastic Net.

Interactions between features

With several features, PolynomialFeatures also creates interaction terms such as x1*x2, which let the effect of one feature depend on another.

Python An interaction a plain linear model misses

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical warnings

  • Features multiply quickly: 10 columns at degree 3 produce 285 features.
  • Polynomials behave wildly outside the range of the training data, so never trust them to extrapolate.
  • Scale features before expanding, to keep numbers manageable.
  • If you need very flexible curves, tree ensembles or splines (SplineTransformer) are often better than high degrees.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is polynomial regression still called a linear model?

    The curve comes from features like x squared, but the model still adds weighted features.
  2. 2. As the polynomial degree grows, training R-squared will typically:

    More flexible models fit the training data ever more closely, which is why test data is needed.
  3. 3. Why pass include_bias=False in the pipeline before LinearRegression?

    The constant column would duplicate the intercept.
  4. 4. Which technique helps stabilise a degree-12 polynomial fit?

    A penalty keeps the huge, correlated polynomial coefficients under control.

Summary

  • PolynomialFeatures plus LinearRegression in a pipeline fits curves.
  • Pick the degree by held-out performance, not training score.
  • Too high a degree overfits; add scaling and Ridge.
  • Interaction terms capture "feature A matters more when B is large".
  • Avoid extrapolating polynomials beyond the training range.

Next: Decision Tree and k-NN Regression