Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 28 ยท Generalized Linear Models

Logistic and Probit Regression

Model yes/no outcomes with Logit and Probit in Statsmodels. Read log-odds, odds ratios and marginal effects, predict probabilities and judge classification accuracy.

OLS is meant for outcomes that can take any number. When the outcome is binary (bought or not, churned or stayed, passed or failed), a straight line misbehaves: it predicts probabilities below 0 and above 1, and its errors cannot be normal. Logistic regression fixes this by modeling the log-odds of the event as a linear function of the predictors.

log( p / (1 - p) ) = b0 + b1*x1 + ...

The odds are p / (1 - p). A probability of 0.8 means odds of 4 to 1. The inverse (logistic) function squeezes any linear score into the range 0 to 1. Probit uses the normal CDF as the squeeze instead; results are almost identical, but logit coefficients have the friendlier odds-ratio reading.

Fit a Logit

smf.logit takes a formula with a 0/1 outcome. Fitting is by maximum likelihood, so you see iteration messages (turn them off with disp=0).

Python Logistic regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coefficients are in log-odds. That is hard to feel, so we translate.

Odds ratios

Exponentiate a coefficient to get the odds ratio: the factor by which the odds are multiplied when the predictor rises by one unit. An odds ratio of 1 means no effect; 2 means doubled odds; 0.5 means halved.

Python Odds ratios with confidence intervals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each extra study hour multiplies the odds of passing by about 2 (exp(0.7)). Because the effect is multiplicative on the odds, the change in probability depends on where you start.

Probabilities, marginal effects and accuracy

res.predict(new_df) returns probabilities. To say "what does one more hour do to the probability, on average?", use average marginal effects: res.get_margeff().summary().

Python Predict, marginal effects, accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The marginal effect dy/dx of hours is a probability change per hour averaged over all students, a number you can explain to a non-statistician.

Accuracy depends on the 0.5 threshold and on how common the event is. When positives are rare, look at precision, recall and ROC-AUC (for example from scikit-learn) instead of accuracy alone.

Probit and the problem of perfect separation

smf.probit has the same interface, only a different link. Compare their predictions: they are nearly equal. If a predictor separates the classes perfectly, maximum likelihood does not exist (coefficients run off to infinity) and you will see a PerfectSeparationError or huge coefficients; fix it by removing or combining the offending variable, collecting more data, or adding a penalty with fit_regularized.

Python Logit versus probit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Probit coefficients are about 1.7 times smaller because the two curves use different scales, but the predicted probabilities almost coincide. Pick logit unless your field uses probit by convention. For multi-class outcomes see MNLogit, and for ordered categories OrderedModel.

Logit is a special case of the framework in GLM Families and Links.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a logistic regression model linearly?

    The logit link makes log(p/(1-p)) a linear function of the predictors.
  2. 2. A coefficient of 0.69 in a logit model corresponds to an odds ratio of about what?

    exp(0.69) is about 2, so each unit doubles the odds.
  3. 3. What does an average marginal effect give you?

    It translates log-odds coefficients into probability changes.
  4. 4. What typically happens under perfect separation?

    When a predictor perfectly splits the classes, the likelihood has no finite maximum.

Summary

  • Use smf.logit (or probit) for 0/1 outcomes; the output is on the log-odds scale.
  • np.exp(params) gives odds ratios; get_margeff() gives probability changes.
  • predict returns probabilities; choose a threshold for class labels and judge with more than accuracy.
  • Perfect separation breaks maximum likelihood; regularize or rethink the variable.

Next: Count Models