Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 28 ยท Generalized Linear Models

Count Models

Model counts such as visits, claims or defects with Poisson and negative binomial regression in Statsmodels. Handle overdispersion, exposure offsets and excess zeros.

Many outcomes are counts: calls per hour, goals per match, accidents per road, support tickets per customer. Counts are never negative, they are whole numbers, and their variability usually grows with their average. OLS on counts can predict negative numbers and ignores that structure. The standard tool is Poisson regression.

Poisson regression models the log of the expected count: log(mu) = b0 + b1*x1 + .... So exp(b1) is a rate ratio: how many times larger the expected count is for one more unit of x.

Python Poisson regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fitted rate ratio for promo is about 1.4 (the true value is exp(0.4), about 1.5): a promotion brings roughly 40-50% more visits, all else equal.

Overdispersion

A true Poisson variable has variance equal to its mean. Real counts are often more spread out (overdispersed) because of unmeasured differences between units. If you ignore it, standard errors are too small and p-values too optimistic.

Check by fitting a GLM Poisson and computing the Pearson chi-square divided by the residual degrees of freedom. Values near 1 are fine; well above 1 signal trouble.

Python Detect overdispersion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ratio is far above 1, and the variance is much larger than the mean. Two fixes are common:

  1. Quasi-Poisson style: keep the Poisson coefficients, scale the standard errors with fit(scale="X2") (a GLM option).
  2. Negative binomial: add an extra parameter alpha for the extra variance. Use smf.negativebinomial, or sm.GLM with sm.families.NegativeBinomial(alpha=...).
Python Negative binomial regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The negative binomial standard error is larger (honest) and the AIC is much lower (better fit). The parameter alpha quantifies the overdispersion; the true value here is 1/2 = 0.5.

Exposure and offsets

Counts depend on how long or how large the observation was: 12 accidents in a year on a busy road is different from 12 on a quiet road. Model the rate by including the log of the exposure as an offset, a predictor whose coefficient is fixed to 1.

Python Rates with an exposure offset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without the offset, longer-observed roads would look more dangerous simply because they were watched longer.

Excess zeros

If many more zeros appear than a Poisson or negative binomial would give (for example customers who never buy at all), use a zero-inflated model: sm.ZeroInflatedPoisson mixes a Poisson count process with a separate "always zero" process. Hurdle-type designs are another option. Compare candidate models by AIC and by comparing the predicted share of zeros with the actual share.

Poisson and negative binomial regression are part of the generalized linear model family; see GLM Families and Links for the general picture.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does exp(b) mean in a Poisson regression?

    The model uses a log link, so coefficients are multiplicative effects on the expected count (rate ratios).
  2. 2. What is overdispersion?

    Poisson assumes variance equals the mean; overdispersed data are more variable, making Poisson standard errors too small.
  3. 3. Which model adds a dispersion parameter for overdispersed counts?

    The negative binomial has an extra alpha parameter that lets variance exceed the mean.
  4. 4. Why use an offset of log(exposure)?

    The offset fixes the coefficient of log exposure at 1, so the model describes counts per unit of exposure.

Summary

  • Poisson regression models log expected counts; exponentiated coefficients are rate ratios.
  • Check overdispersion with Pearson chi-square over residual degrees of freedom; use negative binomial or scaled errors if it is high.
  • Use offset=np.log(exposure) for rates.
  • Zero-inflated models handle excess zeros.

Next: GLM Families and Links