Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 28 ยท Pitfalls and Production

Pitfalls and Debugging

Avoid the usual Statsmodels mistakes: missing intercept, NaN and dtype errors, wrong column order, extrapolation, causal claims, p-hacking and convergence warnings, with fixes.

Statsmodels rarely crashes when you do something statistically wrong. It happily returns numbers. The dangerous errors are the quiet ones, so this lesson collects the mistakes that bite most often, how to recognize each one and how to fix it.

1. Forgetting the intercept

The array API fits exactly the columns you pass. No constant column means a line forced through the origin, and R-squared is then computed differently (uncentered), often looking misleadingly high.

Python The missing constant

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without the constant, the slope absorbs the intercept (about 3.6 instead of 0.5) and the uncentered R-squared looks excellent. Formulas add the intercept for you; the array API does not.

2. Missing values and wrong dtypes

Two classic errors: ValueError: Pandas data cast to numpy dtype of object (a text column slipped into X) and silent row loss from NaN. Formulas drop rows with missing values silently, so always compare res.nobs with len(df). In the array API, missing="drop" or "raise" makes the choice explicit.

Python Silent row loss and object columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

3. Wrong columns or order at prediction time

predict with a formula model rebuilds the matrix from raw columns, which is safe. With the array API, the new matrix must have the same columns in the same order as training, including the constant. A forgotten add_constant on new data throws a shape error. When you need robust prediction, prefer formula models, or keep the column list in one variable.

Python Formula predictions are safer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A category that was absent from training cannot be predicted; handle it explicitly (a fallback group or a rejection).

4. Misreading p-values and significance

  • Significant is not important. With huge samples, trivial effects have tiny p-values. Look at the coefficient size and its interval.
  • Not significant is not zero. A wide interval means "we do not know", not "no effect".
  • Many tests, many false alarms. Run 20 regressions and one will look significant by chance; see ANOVA and Multiple Testing.
  • p-hacking. Trying variables and subsets until something passes makes the p-values meaningless. Decide the model before looking, or validate on held-out data.
Python Chance findings in pure noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expect about 2 false positives in 40, and an R-squared that looks respectable although the adjusted value stays near or below zero.

5. Extrapolation and causation

A fitted line is only trustworthy inside the range of the data. Predicting house prices for a 1,000 m2 house from a sample of 40 to 160 m2 is guesswork. And regression shows association: unless the data come from a randomized experiment (or you have a careful causal design), do not write "X causes Y". Omitted variables can create or hide effects.

6. Convergence warnings

Non-linear fits (logit, mixed models, ARIMA) use iterative optimizers. ConvergenceWarning, HessianInversionWarning or Maximum number of iterations reached mean the answer may be unreliable. Typical remedies:

  • Rescale predictors (mean 0, standard deviation 1).
  • Remove collinear or near-constant variables (Multicollinearity and VIF).
  • Try another optimizer (fit(method="bfgs", maxiter=500)).
  • Simplify the model, for example fewer random effects or lower ARIMA orders.
  • For logit: check for perfect separation (Logistic and Probit Regression).
Python Rescaling keeps the fit well behaved

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both fits converge here, but with a variable measured in the tens of thousands the optimizer has less room for error. After scaling, the coefficient also becomes readable: it is the change in log-odds per standard deviation of income.

7. Version traps

  • Old tutorials use removed APIs (statsmodels.tsa.ARMA, .ols module pandas.stats). Use statsmodels.tsa.arima.model.ARIMA.
  • Summary output can differ between versions; do not parse it, read attributes (res.params, res.pvalues).
  • Results hold the data; res.remove_data() shrinks them before pickling (Predict, Save and Deploy).

When stuck, shrink the problem: simulate data with known coefficients (as in this course), fit your exact code, and see whether it recovers the truth. If it does, the bug is in the real data.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You use sm.OLS(y, x) with no constant. What changes?

    Without a column of ones the line passes through the origin, and R-squared is computed around zero rather than the mean.
  2. 2. How can you notice that a formula silently dropped rows with NaN?

    res.nobs is the number of rows actually used.
  3. 3. Why is a formula-based model safer for prediction than the array API?

    Patsy stores the recipe, so you pass raw data and cannot mis-order columns.
  4. 4. A convergence warning appears for a logit fit. What is a sensible first step?

    Poor scaling and separation are common causes; fixing them usually restores convergence.

Summary

  • Quiet errors are the worst: missing constant, dropped rows, wrong columns, unseen categories.
  • Significance is not importance; many tests and model search inflate false positives.
  • Do not extrapolate beyond the data range or claim causation from association.
  • Rescale, simplify and check separation when the optimizer complains.
  • Test your pipeline on simulated data with known answers.

Next: Predict, Save and Deploy