Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 111 ยท Regression

Multiple Linear Regression

Predict from many features at once: read coefficients correctly, spot multicollinearity, see why scaling changes interpretation, and compare with statsmodels.

Real targets rarely depend on one thing. A flat's rent depends on its size, its age, its distance from the center and more. Multiple linear regression extends the straight line to many features by giving each one its own slope (a coefficient) and adding them up. The fitting method is the same least squares you met in Linear Regression; only the bookkeeping grows.

The new difficulty is not fitting but reading. A coefficient now means "the effect of this feature while the others are held fixed", which is a delicate sentence. This lesson shows how to read coefficients, what goes wrong when features overlap (multicollinearity), why feature scale changes how you compare them, and how statsmodels reports the same fit with uncertainty attached.

One slope per feature

Size Rent goes up with area.
Age Older flats may cost less.
Distance Farther from the center, cheaper.

prediction = intercept + w1size + w2age + w3*distance

1 / 2

The model with several features

With p features the model is:

TEXT
prediction = intercept + w1 * x1 + w2 * x2 + ... + wp * xp

LinearRegression stores the weights in coef_ (an array with one entry per column of X) and the intercept in intercept_. Pandas is convenient here because you can keep the column names next to the numbers. We will create a small synthetic rent table where we know the true rule, so we can check that the model finds it.

Python Fitting several features at once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The learned weights should land near the true values (4.0, -2.5 and -12.0). Read the second one as: "if two flats have the same size and the same distance, the one that is a year older is predicted to rent for about 2.5 less." The phrase same size and same distance is the part people forget.

Passing a DataFrame also records the column names in the fitted model (model.feature_names_in_), and scikit-learn will warn you if you later predict with different columns.

Why "holding the others fixed" matters

The coefficient of a feature in a multiple regression is generally not the same as its slope in a one-feature regression. Suppose bigger flats tend to be newer. Then in a simple fit of rent on age alone, age picks up part of the size effect, and looks more harmful or more helpful than it is. The multiple regression separates them. Simple and multiple coefficients agree only when the features are unrelated.

Python The same feature, alone and with company

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fitted alone, age gets a coefficient around -8 instead of -1, because it is standing in for size (older flats are smaller, and smaller flats are cheaper). With both in the model, the estimates move back to the truth. This is why a single-feature correlation is a poor guide to what a multi-feature model will do.

Multicollinearity: when features overlap

If two features carry nearly the same information (floor area in square meters and in square feet, or total rooms and bedrooms), the model cannot tell them apart. The predictions may stay fine, but the individual coefficients become unstable: tiny changes in the data can swing them wildly, even to opposite signs, as long as they cancel each other out.

This is multicollinearity. The symptom is large, erratic coefficients and widely different values when you refit on resampled data. Let's make a near-duplicate feature and watch what happens.

Python Unstable coefficients from a near-duplicate feature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The individual weights jump around from resample to resample, while their sum stays steady near 4. The model knows the total effect of "size" well but cannot decide how to split it between two near-identical columns. Predictions are fine; interpretation is not.

How to deal with it:

  • Drop or combine redundant features. Keep one of the two area measurements.
  • Check correlations between features before fitting: DataFrame.corr() is a first look. A more systematic tool is the variance inflation factor (VIF) from statsmodels, where values above roughly 5 to 10 are a warning sign.
  • Regularize. Ridge regression, covered in Ridge, Lasso and Elastic Net, shares weight across correlated columns and keeps them stable.
  • Accept it if you only need predictions, not explanations.

Scale, units and comparing coefficients

A coefficient is in units of target per unit of feature. The weight on size_m2 is rent per square meter; the weight on age_years is rent per year. You cannot decide which feature matters more by looking at which number is bigger, because a feature measured in tiny units gets a big weight and vice versa.

One fix is to standardize the features (subtract the mean, divide by the standard deviation), so every column has mean 0 and standard deviation 1. A standardized coefficient reads "change in the target for a one-standard-deviation change in the feature", which makes the sizes comparable. Predictions do not change at all, only the units of the weights. Full details of scaling are in Scaling and Normalization.

Python Raw versus standardized coefficients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By raw weight, distance looks like the most important feature (-12 per km) and size the least (4 per square meter). After standardizing, the ranking changes: size, with its wide spread of values, has the largest effect per standard deviation. Predictions are identical, which the last line confirms. Remember that "per standard deviation" is still not "importance"; real importance depends on the question you are asking, and Feature Importance and Interpretation returns to this.

A statsmodels summary alongside scikit-learn

scikit-learn is built for prediction and gives you the numbers but not the uncertainty around them. statsmodels is built for statistical inference and reports standard errors, confidence intervals and p-values for every coefficient. You add a constant column for the intercept, because statsmodels does not assume one.

Python The same fit in scikit-learn and statsmodels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two libraries agree to the last digit on the coefficients, because they solve the same problem. statsmodels adds the columns that scikit-learn omits: a coefficient with a confidence interval that includes zero is one the data cannot distinguish from "no effect". You can also call ols.summary() for the full printed report. In machine learning projects the habit is to judge a model by held-out performance rather than by p-values, but the standard errors are valuable when you must explain or defend a coefficient.

Warning: More features always raise training R squared, even useless ones. Compare models with held-out data or with adjusted R squared (see Regression Metrics), never with training R squared alone.

Practical checklist

  1. Put features in a table with clear names and make sure none is a disguised copy of the target.
  2. Look at pairwise correlations; remove or combine near-duplicates.
  3. Fit on training data, evaluate on held-out data.
  4. Standardize if you want to compare effect sizes (or when you move on to regularized models).
  5. Plot residuals against predictions to catch curvature.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A coefficient of -2.5 on age_years in a multiple regression means:

    A coefficient is a conditional statement: the change in the prediction per unit of that feature, with every other feature unchanged. It says nothing by itself about importance.
  2. 2. What is the typical symptom of multicollinearity?

    When two features carry the same information, the model can split their combined effect in many ways. The individual weights swing, but their sum and the predictions stay steady.
  3. 3. What does standardizing the features change?

    For plain least squares, predictions are identical before and after standardizing. Only the coefficients are rescaled, so they now mean 'change per one standard deviation'.
  4. 4. Why would you reach for statsmodels in addition to scikit-learn?

    Both fit the same least squares model. statsmodels adds inference output; scikit-learn focuses on prediction and pipelines.

Summary

  • Multiple linear regression gives each feature its own coefficient; coef_ holds one value per column and intercept_ the baseline.
  • A coefficient means the change in the prediction per unit of that feature while the others stay fixed, so it can differ a lot from the single-feature slope.
  • Multicollinearity makes individual coefficients erratic though predictions stay fine; drop, combine or regularize correlated features.
  • Raw coefficients depend on units; standardize to compare effects per standard deviation.
  • statsmodels gives the same coefficients plus standard errors and confidence intervals.

Next: judge how good such a model really is in Regression Metrics.