Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 111 ยท Regression

Polynomial Regression and Nonlinearity

Fit curves with linear models: PolynomialFeatures, choosing the degree, a first look at underfitting and overfitting, and interaction terms between features.

A straight line is a strong assumption, and plenty of real patterns break it: stopping distance grows faster than speed, plant growth levels off, ticket demand rises and then falls with price. The good news is that you do not need a new kind of model to capture a curve. You can keep linear regression and give it better inputs: add x squared, x cubed and so on as extra columns. The model is still linear in its coefficients, so everything from the previous lessons still applies, but the fitted shape is a curve.

This lesson shows how PolynomialFeatures builds those columns, how the degree controls flexibility, and why too much flexibility ruins predictions on new data (a first look at overfitting, which gets its own lesson later). It finishes with interaction terms, which let the effect of one feature depend on another.

Curves from a linear model

Add columns x, x squared, x cubed become separate features.
Fit as usual Least squares finds one weight per column.
Get a curve The weighted sum bends with x.
1 / 2

Polynomial features

For one input x and degree 3, the model is prediction = b + w1*x + w2*x**2 + w3*x**3. From the model's point of view that is just three features. PolynomialFeatures is a transformer (it has fit and transform, not predict; see The scikit-learn API) that creates those columns for you.

Python What PolynomialFeatures produces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With two features and degree 2, you get a, b, a squared, a*b and b squared. The a*b column is an interaction term, covered below. include_bias=False skips the constant column of ones, because LinearRegression already has its own intercept.

The number of columns grows quickly: degree d on p features produces C(p + d, d) - 1 columns, so 10 features at degree 3 give 285. That growth is a warning to keep the degree small.

Fitting a curve

Chain the transformer and the regressor in a pipeline, so the polynomial columns are built automatically both when training and when predicting (Pipelines explains why this matters). Here the true relationship is a curve, a gently bending cubic.

Python Line versus curve on curved data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cubic recovers weights near the true 2, -1 and 0.5 and a much better R squared than the line. The line is a textbook case of underfitting: the model is too simple for the pattern, so it is wrong both on the training data and on new data.

Choosing the degree

If more flexibility fits training data better, why not use degree 15? Because a flexible model can bend to follow noise, and noise does not repeat in new data. The way to see this is to compare error on the training data with error on held-out data as the degree grows. The training error keeps falling; the test error falls, then rises.

Python Degree versus training and test error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training error keeps shrinking as the degree rises (more flexibility always fits the training points at least as well), while test error bottoms out between degrees 3 and 5 (the true degree is 3; noise nudges the minimum) and then climbs, reaching its worst values at degrees 11 and 12. The gap between the two is the signature of overfitting. We added a StandardScaler after the polynomial step because high powers of x have wildly different scales, which makes the numbers hard for the solver.

Note: Picking the degree by looking at the test set, as the last line does, is only fine in a demo. In real work, pick it with a validation set or cross-validation (Cross-Validation) and touch the test set once at the end.

Overfitting made visible

Plot a degree-12 curve on a few points and the problem is obvious: the curve wiggles between the training points, and its behavior outside the data range is absurd. Polynomials are especially dangerous when you extrapolate, because the highest power takes over.

Python A high-degree curve chasing noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The degree-12 curve is plainly overfitted: it wiggles between the points, and at x = 4, just outside the data, it predicts about 296,000 where the true pattern is near -2. Degree 1 misses the wave entirely (underfitting), degree 3 captures the shape without chasing the noise. The cure for overfit is some mix of fewer features, more data and regularization, which is the subject of Ridge, Lasso and Elastic Net.

Interaction terms

So far each feature had its own separate effect. Sometimes the effect of one feature depends on another: an extra hour of study helps more for students who also sleep well; an extra bedroom adds more rent in a big flat than in a small one. A product column x1 * x2, called an interaction term, lets a linear model capture this. PolynomialFeatures(interaction_only=True) creates only the products, without the squares.

Python An interaction a plain linear model cannot see

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Adding the product column raises the cross-validated R squared and gives the model a weight on study sleep: the effect of studying grows with sleep. Interaction terms are cheap, easy to explain and a good first step before you reach for a tree-based model that finds interactions on its own.

Where polynomial regression is used, and where it fails

Polynomial regression is a good fit when a relationship is smooth and curved, with a single bend or two, inside a range you have actually measured. Typical real-life examples:

  • Growth and decay: how a crop yield rises with fertilizer and then levels off or drops; the speed of an epidemic wave over a few weeks.
  • Engineering and physics: stopping distance against speed, the output of a sensor against temperature (a calibration curve).
  • Economics and finance: a diminishing-returns curve for advertising spend, or the relation between income and spending.
  • Environmental studies: pollutant concentration against distance from a source.

Its disadvantages are as important as its uses:

  1. It extrapolates terribly. Outside the range of the training data, a polynomial heads off to plus or minus infinity as fast as its highest power allows.
  2. High degrees overfit. The curve wiggles to catch every point (see "Overfitting made visible" above).
  3. It is sensitive to outliers, especially at the ends of the range, where high-degree terms have the most leverage.
  4. The features are highly correlated (x and xยฒ move together), which makes the coefficients unstable and hard to interpret. Centering x first (subtracting its mean) removes most of that correlation.
  5. A single global formula cannot follow a local change. Splines, SplineTransformer, trees or gradient boosting are better when the shape differs from place to place.

The experiment below fits polynomials of degree 2, 5 and 9 to noisy data in the range 0 to 10 (the true curve is a gentle saturation), and then predicts at x = 12 and 15, outside the training range.

Python Polynomial extrapolation and correlated powers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inside the range all three degrees agree with the truth. Outside it they separate: the quadratic keeps bending and drops to 1.8 at x = 15, the degree-5 curve plunges to -14, and the degree-9 curve explodes to 1,520, while the true curve stays near 10. A model that is excellent for interpolation can be useless for forecasting beyond the data. Also note that x and xยฒ are correlated at about 0.97 in the raw data, but only about -0.1 after centering, which is why scaling or centering before polynomial expansion is good practice.

Practical advice

  • Start with degree 2 or 3; a higher degree is rarely worth it.
  • Always scale after expanding, and always evaluate on held-out data.
  • With many features, the column count explodes; consider interactions only (interaction_only=True), regularization, or switch to trees and splines (SplineTransformer, in Feature Creation and Transformation).
  • Do not extrapolate with high-degree polynomials.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is polynomial regression still called linear regression?

    The prediction is a weighted sum of columns (x, x squared, ...). The weights enter linearly, so ordinary least squares still applies.
  2. 2. As the polynomial degree increases, what typically happens to training and test error?

    More flexibility always helps fit the training data, but beyond the true complexity the model fits noise, which hurts held-out performance.
  3. 3. What does PolynomialFeatures(degree=2, include_bias=False) produce for inputs a and b?

    Degree 2 includes the original features, their squares and the pairwise product. include_bias=False drops the constant column of ones.
  4. 4. When is an interaction term useful?

    A product column lets the model change the slope of one feature depending on another, which plain additive terms cannot express.

Summary

  • Adding powers of a feature turns linear regression into a curve fitter; the model is still linear in its weights.
  • PolynomialFeatures builds the columns; put it in a pipeline with LinearRegression, and scale after expanding.
  • The degree controls flexibility: too low underfits, too high overfits and extrapolates badly.
  • Choose the degree with held-out data; training error alone always favors more flexibility.
  • Interaction terms (x1 * x2) capture effects that depend on another feature.

Next: tame overfitting with penalties in Ridge, Lasso and Elastic Net.