Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 28 ยท Linear Regression in Depth

Weighted and Robust Regression

Handle unequal variance and outliers in Statsmodels with WLS, GLS, robust standard errors (HC3, HAC) and RLM. Learn which tool fixes which assumption failure.

Classic OLS assumes the errors have the same spread everywhere (homoscedasticity), are independent of each other and are not dominated by a few wild points. Real data often breaks one of these. Statsmodels has a remedy for each, and this lesson shows when to reach for which.

Which problem, which tool

Unequal variance WLS or robust standard errors.
Correlated errors GLS, GLSAR or HAC errors.
Outliers RLM with a robust norm.

Fix the cause if you know it, otherwise fix the standard errors.

1 / 2

Robust standard errors: the quick fix

Often the coefficients from OLS are fine, but the standard errors are wrong because the variance changes. You can keep the OLS fit and ask for a different covariance estimate with cov_type:

  • "HC3" (or HC0-HC3): heteroscedasticity-consistent, "White" errors. HC3 is a safe default for small samples.
  • "HAC": also handles autocorrelation (Newey-West); needs cov_kwds={"maxlags": k}.
  • "cluster": errors correlated within groups, with cov_kwds={"groups": ids}.
Python Robust standard errors with cov_type

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coefficients do not change, only their standard errors, and therefore the p-values and intervals. Cheap insurance: many analysts report robust errors by default.

Weighted least squares (WLS)

If you know how the variance behaves, use it. WLS gives each observation a weight proportional to 1 / variance: precise points count more. Typical cases are averages of different group sizes (variance is sigma^2 / n) or noise that grows with a variable.

Python WLS when variance grows with x

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the weights are right, WLS gives smaller standard errors: it is the efficient estimator. When you only suspect heteroscedasticity but do not know its form, OLS with HC3 errors is safer.

GLS and autocorrelation

With time-ordered data, neighboring errors are often alike (autocorrelation). sm.GLS takes the full error covariance matrix sigma, and sm.GLSAR estimates an AR(p) error structure for you. For many projects, simply using cov_type="HAC" is enough.

Python Autocorrelated errors: HAC versus classic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Durbin-Watson number is far below 2 and the classic standard error is too small: OLS would make you overconfident. HAC and GLSAR correct it. More on time-ordered models in Time Series Basics.

Robust regression with RLM

Outliers pull the OLS line toward them because squared errors punish big gaps heavily. sm.RLM uses a robust norm such as Huber's, which counts large residuals linearly rather than quadratically, so outliers lose influence.

Python OLS versus RLM with outliers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

rlm.weights shows how much each point counted; the outliers get small weights. RLM protects the estimates from outliers, but it is not a license to ignore them: investigate why the points are odd (Influence and Outliers).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does cov_type='HC3' change?

    Robust covariance options replace only the covariance estimate of the coefficients.
  2. 2. When is WLS the best choice?

    WLS weights each point by the inverse of its variance, giving efficient estimates when the weights are correct.
  3. 3. Residuals of time-ordered data are strongly autocorrelated. What is a reasonable fix for the standard errors?

    HAC (Newey-West) errors allow for correlation across time; GLSAR is the model-based alternative.
  4. 4. How does RLM reduce the influence of outliers?

    Robust norms like Huber's treat large residuals linearly, so they pull the fit less.

Summary

  • Robust cov_type options (HC3, HAC, cluster) fix standard errors without changing coefficients.
  • WLS uses known variance weights and gains efficiency when they are right.
  • GLS, GLSAR and HAC deal with autocorrelated errors.
  • RLM downweights outliers through a robust norm; still investigate them.

Next: Regularized Regression