Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 28 ยท Diagnostics

Multicollinearity and VIF

Detect correlated predictors with the variance inflation factor in Statsmodels, see how they inflate standard errors, and fix the problem with dropping, combining or ridge.

Multicollinearity means two or more predictors carry nearly the same information. House area_m2 and area_sqft are the extreme case, but milder versions are everywhere: age and years of experience, height and weight, the number of rooms and the area.

OLS can still fit such a model and predictions may stay fine. What goes wrong is interpretation. The model cannot tell which of the overlapping variables deserves the credit, so individual coefficients become unstable, standard errors balloon, and signs can even flip. A joint F-test may be extremely significant while every single t-test is not.

See it happen

We make two nearly identical predictors and fit the model on several random samples. Notice how erratic the individual coefficients are, while their sum is steady.

Python Unstable coefficients with correlated predictors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With near-collinearity, b1 and b2 wander far from 2 (and the standard error is large), while their sum stays near 4: the model knows the combined effect but not the split.

The variance inflation factor

The VIF of a predictor measures how much its coefficient's variance is inflated by correlation with the others. Regress that predictor on all the other predictors; if they explain it very well (R^2 close to 1), the VIF is large: VIF = 1 / (1 - R^2).

VIF Reading
1 No correlation with the other predictors
1-5 Usually fine
5-10 Worth a look
above 10 Serious; the standard error is inflated by sqrt(VIF) (over 3x)

Statsmodels computes it with variance_inflation_factor(exog, i). Pass the design matrix with the constant and skip reading the constant's own row.

Python Compute VIF for every predictor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

area and rooms get large VIFs, age stays near 1. The correlation matrix tells the same story, but VIF also catches combinations, such as one variable being a sum of three others, which pairwise correlations miss.

Remedies

  • Drop one of the overlapping variables if it adds nothing (keep the one that is easier to interpret or measure).
  • Combine them: an average, a ratio or a score (for example rooms_per_area).
  • Center variables when the collinearity comes from including x and x**2 or an interaction (Categorical Variables and Interactions).
  • Use regularization: ridge regression stabilizes the estimates (Regularized Regression).
  • Do nothing if you only need predictions, or if the collinear variables are control variables you do not interpret.
Python Centering reduces polynomial collinearity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Centering removes the artificial overlap between x and x**2 without changing the fitted curve.

The condition number

The summary's Cond. No. is another detector. It is computed from the whole design matrix: values in the hundreds of thousands mean trouble, but it is also inflated by differing scales. Standardize predictors for a cleaner reading, and use VIF to find which variables are involved.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main damage done by multicollinearity?

    Correlated predictors make individual effects hard to separate; predictions can still be fine.
  2. 2. A predictor has VIF = 20. About how much larger is its standard error than without collinearity?

    Variance is inflated by VIF, so the standard error is inflated by its square root.
  3. 3. Which is a sensible remedy when only prediction matters?

    Predictions are barely affected by collinearity; regularization can still add stability.
  4. 4. How do you compute VIF for column i of the design matrix?

    The function lives in statsmodels.stats.outliers_influence and takes the exog matrix and a column index.

Summary

  • Multicollinearity inflates standard errors and makes coefficients unstable; predictions often survive.
  • VIF above about 5 to 10 flags a problem; correlation matrices miss multi-variable overlaps.
  • Fix by dropping, combining, centering or regularizing.
  • Centering helps for polynomial and interaction terms.

Next: Classical Tests and Power