Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 28 ยท Diagnostics

Influence and Outliers

Find outliers and influential points in OLS with Statsmodels: leverage, studentized residuals, Cook's distance, DFFITS and influence plots, and decide what to do with them.

A single odd observation can twist a regression line. Not every unusual point matters, though, and the difference is the heart of this lesson. Three ideas fit together:

  • Outlier: a point whose y is far from what the model predicts (a large residual).
  • Leverage: a point whose x values are far from the other points (it has the power to pull the line).
  • Influence: how much the fitted model actually changes if the point is removed. High influence usually needs both a large residual and high leverage.

A point with high leverage that sits right on the trend is harmless. An outlier in the middle of the x range moves the line only a little. A high-leverage outlier can swing the whole slope.

Outlier, leverage, influence

Outlier Far from the line in y.
Leverage Far from the others in x.
Influence Changes the fit when removed.

Influence is the dangerous combination.

1 / 2

The influence toolbox

res.get_influence() returns an object with everything you need. Its summary_frame() is a DataFrame with one row per observation.

Column Meaning Common cutoff
hat_diag Leverage above 2p / n (p = number of parameters)
student_resid Externally studentized residual absolute value above 3
cooks_d Cook's distance, overall influence above 4 / n
dffits Change in the point's own fitted value above 2 * sqrt(p / n)
Python Find the influential points

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The added point (the last row, index 40) tops the Cook's distance list. It has enormous leverage because x = 20 is far from the rest and it sits far below the trend.

See the damage

Compare the fit with and without the point:

Python Refit without the influential point

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One observation out of 41 changed the slope a lot. That is influence.

Plots

Statsmodels has ready-made charts:

Python
sm.graphics.influence_plot(res, criterion="cooks")   # bubble size = Cook's distance
sm.graphics.plot_leverage_resid2(res)                # leverage vs squared residual
Python Influence plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What to do with influential points

Never delete a point just because it is inconvenient. Work through these questions in order:

  1. Is it an error? A typo, a unit mix-up or a sensor glitch should be corrected or removed, and you should say so in your report.
  2. Is it a different kind of case? A mansion in a data set of apartments may deserve its own model or a flag variable.
  3. Is it genuine and important? Keep it. Report results with and without it, or use a robust method such as RLM (Weighted and Robust Regression).
  4. Could a transformation fix it? Heavily skewed predictors often create leverage. A log transform may bring far points in line.

Dropping points until the results look nice is a form of data dredging. Decide your rule for exclusions before you see the effect on the coefficients, and document it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What best describes leverage?

    Leverage depends only on the predictors; high-leverage points have the potential to pull the fit.
  2. 2. Which statistic measures the overall change in the fit when a point is removed?

    Cook's distance combines leverage and residual size into one influence measure.
  3. 3. A point has high leverage but lies exactly on the trend. How influential is it?

    Influence needs both leverage and a large residual; a high-leverage point on the trend even stabilizes the slope.
  4. 4. What should you do first with a flagged point?

    Understanding the cause drives the right action: correct, model separately, keep with a robust method, or remove with documentation.

Summary

  • Outliers have large residuals; leverage points have unusual predictors; influence is the effect on the fit.
  • res.get_influence().summary_frame() gives leverage, studentized residuals, Cook's distance and DFFITS.
  • Compare fits with and without flagged points to see the real impact.
  • Investigate before removing; consider robust regression or transformations.

Next: Multicollinearity and VIF