Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 57 · Regression

Multiple Linear Regression

Extend regression to several predictors: matrix least squares, adjusted R², dummy variables, interactions, multicollinearity and model checks, in Python.

One predictor rarely tells the whole story. A house price depends on size, age, location and more. Multiple linear regression models an outcome from several predictors at once:

y = β₀ + β₁x₁ + β₂x₂ + … + βₖxₖ + ε

Each coefficient βⱼ is now the average change in y for a one-unit change in xⱼ holding all the other predictors fixed. That phrase is the whole point: it lets you separate the effect of one variable from the others, to the extent the data allow.

Fitting with matrices

Stack the predictors into a matrix X (a first column of ones for the intercept) and the outcomes into a vector y. The least-squares solution is b = (XᵀX)⁻¹Xᵀy. Software uses numerically stable versions of this, such as np.linalg.lstsq.

Python Multiple regression with NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimates land close to the true values 2.5 and −1.2 we simulated, and the intercept near 30.

Adjusted R² and overfitting

Plain R² never decreases when you add a predictor, even a useless one. Adjusted R² penalizes extra predictors: 1 − (SSE/(n−k−1)) / (SST/(n−1)). It rises only when a new variable improves the fit more than chance would. Comparing models by raw R² encourages overfitting, where the model memorizes noise and predicts new data poorly. Better tools are adjusted R², AIC/BIC, and above all testing on held-out data.

Inference: t-tests and the F-test

  • Each coefficient has its own t-test of H₀: βⱼ = 0 given the other predictors in the model.
  • The overall F-test asks whether any predictor is useful (H₀: β₁ = … = βₖ = 0).

A model can have a significant F-test while individual t-tests are not, which usually signals correlated predictors (see below).

Python A full regression table with statsmodels

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The useless noise_var should show a large p-value and a confidence interval that includes 0, while size and age are clearly significant.

Categorical predictors: dummy variables

A category such as neighborhood cannot be fed in as a number. Instead create dummy (indicator) variables: for k categories use k − 1 dummies, with one category as the baseline. Each dummy's coefficient is the average difference from the baseline category, holding the other predictors fixed. Using all k dummies together with an intercept creates perfect redundancy (the dummy variable trap).

Interactions

Sometimes the effect of one predictor depends on another: an extra bedroom may add more value in the city than in the countryside. An interaction term x₁·x₂ lets the slope of x₁ vary with x₂. When you include an interaction, interpret the main effects carefully: each is the effect when the other variable equals zero.

Python Dummy variables and an interaction

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interaction model fits clearly better because the slopes really differ by area.

Multicollinearity

When predictors are strongly correlated with each other (house size and number of rooms), the model cannot tell which one deserves the credit. Symptoms: huge standard errors, coefficients that change wildly when you add or drop a variable, or even flip sign, while predictions can remain fine. The variance inflation factor VIF = 1 / (1 − R²ⱼ), where R²ⱼ comes from regressing predictor j on the others, quantifies it. VIF above about 5 to 10 is a warning. Remedies: drop or combine redundant predictors, or use regularization (ridge regression).

Python Detecting multicollinearity with VIF

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Assumption checks and influence

All LINE checks from Simple Linear Regression carry over: residuals versus fitted values, a normal Q-Q plot, and equal spread. In addition look at influential points using Cook's distance, since one unusual row can change several coefficients at once.

Build models with a purpose. For explanation, include the confounders you know matter and interpret coefficients carefully. For prediction, keep what improves error on data the model has not seen.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. In a multiple regression, what does the coefficient of x1 mean?

    Multiple regression coefficients are partial effects: the average change in y per unit of x1 with the other included predictors held constant.
  2. 2. Why prefer adjusted R squared over R squared when comparing models with different numbers of predictors?

    Adding any variable never lowers R squared. Adjusted R squared penalizes complexity and rises only for genuinely helpful additions.
  3. 3. A categorical predictor has 4 levels. How many dummy variables go into a model with an intercept?

    Use k - 1 dummies; the omitted level is the baseline. Using all four plus an intercept creates perfect collinearity.
  4. 4. Which is a typical sign of multicollinearity?

    Correlated predictors make individual coefficients poorly determined: standard errors inflate and estimates swing when the model changes.
  5. 5. What does an interaction term x1*x2 allow?

    With an interaction, the effect of x1 on y changes with x2, which a purely additive model cannot represent.

Summary

  • Multiple regression estimates partial effects: each slope holds the other predictors fixed.
  • Use adjusted R², information criteria and held-out data to avoid overfitting.
  • Categorical predictors become k − 1 dummy variables relative to a baseline.
  • Interactions let slopes differ between groups.
  • Check VIF for multicollinearity and residual plots plus Cook's distance for assumptions and influence.

Next: Logistic Regression