Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 57 · Regression

Logistic Regression

Model yes/no outcomes with logistic regression: probabilities, odds ratios, maximum likelihood, confusion matrices and thresholds, with Python you can run.

Linear regression predicts a number. But many outcomes are categories: will a customer buy, does an email count as spam, did the patient recover? When the outcome is binary (0 or 1), a straight line is a poor choice, because it can predict "probabilities" below 0 or above 1. Logistic regression solves this by modeling the probability of the outcome through an S-shaped curve.

From probability to odds to log-odds

Three related scales describe the same chance:

  • Probability p, between 0 and 1.
  • Odds p / (1 − p): a probability of 0.8 means odds of 4 to 1 (written 4).
  • Log-odds (logit) ln(p / (1 − p)): any real number, from −∞ to +∞.

Logistic regression is linear on the log-odds scale:

ln(p / (1 − p)) = β₀ + β₁x₁ + … + βₖxₖ

Solving for p gives the sigmoid (logistic) function: p = 1 / (1 + e^−(β₀ + β₁x₁ + …)). Whatever the linear part equals, p stays between 0 and 1.

Python The sigmoid and the odds scale

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At log-odds 0 the probability is exactly 0.5. Each step of 1 in log-odds multiplies the odds by e ≈ 2.718.

Interpreting coefficients: odds ratios

The slope β₁ is the change in log-odds per unit of x. That is awkward, so exponentiate: e^β₁ is the odds ratio, the factor by which the odds are multiplied for each one-unit increase in x (other variables fixed).

  • e^β = 1.5: odds are 50% higher per unit.
  • e^β = 1: no effect.
  • e^β = 0.7: odds are 30% lower.

Careful: an odds ratio is not a risk ratio. When the outcome is rare they are close; when it is common they differ a lot, so avoid saying "50% more likely" for an odds ratio of 1.5.

Fitting: maximum likelihood

There is no closed form. Instead the coefficients are chosen to maximize the likelihood: the probability the model assigns to the outcomes actually observed. For each case with outcome yᵢ and fitted probability pᵢ, the contribution is pᵢ if yᵢ = 1 and 1 − pᵢ if yᵢ = 0. Software iterates (Newton's method) until it converges. This links to the idea of maximum likelihood in the estimation chapter.

Python Fit a logistic regression with statsmodels

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimates should be near the true −3 and 0.8, giving an odds ratio close to e^0.8 ≈ 2.2.

From probabilities to decisions

The model outputs probabilities. To make a yes/no call you choose a threshold, commonly 0.5. Then compare predictions with reality in a confusion matrix:

Predicted 1 Predicted 0
Actual 1 true positive (TP) false negative (FN)
Actual 0 false positive (FP) true negative (TN)

Useful summaries:

  • Accuracy = (TP + TN) / total. Misleading with rare outcomes: a model that always says "no fraud" is 99.9% accurate and useless.
  • Precision = TP / (TP + FP): of those flagged, how many were right.
  • Recall (sensitivity) = TP / (TP + FN): of the real cases, how many were found.
  • Specificity = TN / (TN + FP).

Lowering the threshold finds more positives (higher recall) but raises false alarms (lower precision). The right threshold depends on the cost of each mistake: missing a disease is worse than a false alarm; the reverse might hold for blocking a customer's card. The ROC curve plots recall against the false positive rate over all thresholds, and the AUC (area under it) summarizes ranking quality: 0.5 is guessing, 1 is perfect.

Python Confusion matrix, metrics and thresholds

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how recall falls and precision rises as the threshold increases.

Assumptions and pitfalls

  • Outcomes are independent, and the log-odds are linear in the predictors (add transformations or interactions if not).
  • Avoid separation: if a predictor perfectly splits 0s from 1s, the coefficients diverge to infinity.
  • Have enough events: a common rule of thumb is at least 10 events (the rarer class) per predictor.
  • Evaluate on held-out data (or with cross-validation), not on the data used to fit.
  • With more than two categories use multinomial logistic regression; for ordered categories use ordinal logistic regression.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. Logistic regression is linear in which quantity?

    The model is ln(p/(1-p)) = linear predictor. This keeps the fitted probability between 0 and 1.
  2. 2. A coefficient is 0.4. What is the odds ratio per unit increase?

    Odds ratio = exp(beta) = exp(0.4), about 1.49: the odds rise by about 49% per unit.
  3. 3. A rare-disease screening model predicts 'healthy' for everyone and is 99.5% accurate. What is the problem?

    With rare positives, accuracy is dominated by the majority class. Recall shows the model finds none of the sick patients.
  4. 4. What happens to recall and precision when you lower the classification threshold?

    A lower threshold labels more cases positive, catching more real positives but also adding false positives.

Summary

  • Logistic regression models ln(p/(1−p)) as a linear function and returns probabilities through the sigmoid.
  • exp(β) is an odds ratio, not a risk ratio.
  • Coefficients are estimated by maximum likelihood.
  • Choose a threshold by the cost of errors; judge with a confusion matrix, precision, recall and AUC on held-out data.
  • Watch for separation, rare events and nonlinearity.

Next: Bayesian Inference with the Beta-Binomial Model