Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 76 ยท Modelling

Classification with Logistic Regression

Learn logistic regression for classification: odds and probabilities, statsmodels and scikit-learn, choosing a threshold, and handling imbalanced classes.

Many data science questions have a yes-or-no answer. Will this customer cancel? Is this transaction fraud? Will the patient be readmitted? Linear regression does not fit these well, because it can predict values below 0 or above 1. Logistic regression fixes that: it models the probability of the positive class, and the probability always stays between 0 and 1.

Despite the name, logistic regression is a classification method. It is also one of the best first models for a binary outcome: fast, hard to overfit, and its coefficients can be explained to non-experts. This lesson covers how probabilities, odds and log-odds relate, how to fit the model with statsmodels (for explanation) and scikit-learn (for prediction), why the 0.5 cutoff is only a default, and what to do when one class is rare. It builds on linear regression.

Probability, odds and log-odds

The model works through a chain of conversions. A probability p becomes odds p / (1 - p); a probability of 0.75 means odds of 3 to 1. The logarithm of the odds is the log-odds (or logit). Log-odds can be any real number, so a linear formula can safely produce them. To get back to a probability, apply the inverse function, the sigmoid: 1 / (1 + exp(-z)).

flowchart LR
    A["Inputs x"] --> B["Linear score z<br/>(b0 + b1*x1 + ...)"]
    B --> C["Sigmoid<br/>1 / (1 + e^-z)"]
    C --> D["Probability p<br/>between 0 and 1"]
    D --> E{"p >= threshold?"}
    E -->|"yes"| F["Predict 1"]
    E -->|"no"| G["Predict 0"]

A linear score is squeezed into a probability, and a threshold turns the probability into a class.

Python Probability, odds and log-odds side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the symmetry: probability 0.5 is odds 1 and log-odds 0, and probabilities of 0.25 and 0.75 are mirror images in log-odds. Near 0 and 1, a small change in probability is a large change in log-odds, which is why the curve is S-shaped.

Fitting and reading a logistic model

We simulate customers whose churn probability falls with tenure (months as a customer) and rises with the number of support tickets. We know the recipe: a coefficient of -0.10 per month and +0.45 per ticket.

Python statsmodels Logit and odds ratios

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coefficients are in log-odds, which nobody has intuition for. Exponentiate them to get odds ratios. The tenure odds ratio of 0.91 means each extra month multiplies the odds of churning by about 0.91, a 9% drop, holding tickets fixed. The ticket ratio of 1.53 means each additional ticket multiplies the odds by about 1.5. An odds ratio of 1 means no effect, and the confidence interval tells you whether 1 is plausible.

Two cautions. First, odds are not probabilities: "odds up 50%" does not mean "probability up 50%", and the change in probability depends on where you start. For a message to non-experts, convert to predicted probabilities for a few typical cases, as the last lines above do. Second, as with linear regression, a coefficient is a conditional association, not proof of cause.

Probabilities first, classes second

A logistic model's real output is the probability. A class label is a decision you make after the model, by choosing a threshold. The default of 0.5 is only convenient; it is rarely the right cost trade-off.

Think about a fraud screen. Missing a fraud (a false negative) may cost far more than a manual check on a legitimate order (a false positive). Lowering the threshold catches more fraud but raises false alarms; raising it does the opposite. The two metrics that track this are:

  • Precision: of the cases we flagged, how many were truly positive?
  • Recall (sensitivity): of the truly positive cases, how many did we flag?

Lowering the threshold raises recall and lowers precision. The right point depends on the costs, and the next lessons (evaluating models) go deeper.

Rare positives and the accuracy trap

Now we make the positive class rare, about 11%. Look at what accuracy tells us, and how the threshold changes the picture. In scikit-learn, LogisticRegression().fit(X, y) trains the model, predict_proba returns probabilities, and stratify=y keeps the class ratio equal in the train and test parts.

Python Moving the threshold on an imbalanced problem

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A model that never predicts a positive is already right 88.7% of the time, so the default model's 90.8% accuracy sounds fine. But at threshold 0.5 it finds only 28% of the real positives. Dropping to 0.2 finds 65% at the price of lower precision and lower accuracy. If catching positives is the point, accuracy is the wrong yardstick. This is the accuracy trap for imbalanced classes.

Class weights and rebalancing

Another way to make the model care about the rare class is class_weight="balanced", which makes each rare-class error count more during training. Resampling the data (oversampling the rare class or undersampling the common one) has a similar goal. Compare the effect on ranking quality, measured by AUC (the chance that a random positive gets a higher score than a random negative), and on the probabilities themselves:

Python class_weight changes decisions, not ranking

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The AUC is the same for both (about 0.88): the model ranks cases equally well. What changed is the recall/precision balance at the default cutoff, and the average predicted probability, which for the balanced model (0.33) no longer matches the real rate (0.11). That is the hidden cost: reweighting distorts the probabilities. If you need honest probabilities, for example to compute expected cost, keep the plain model and move the threshold instead. If you only need a ranking or a yes/no flag, either works.

Practical tips

  • Scale inputs for scikit-learn if they have very different ranges; the default solver uses regularization (C=1.0), which treats features unevenly if they are not on similar scales. See encoding and scaling.
  • Encode categories as dummy columns before fitting.
  • Check for separation. If one input perfectly predicts the outcome, statsmodels raises an error and the coefficients explode; combine rare categories or regularize.
  • Do not use LogisticRegression for more than two classes without checking. It does extend to many classes (multinomial), but you then read one set of coefficients per class.
  • Split before you tune the threshold, and choose the threshold on validation data, not the final test set.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A probability of 0.8 corresponds to which odds?

    Odds are p / (1 - p) = 0.8 / 0.2 = 4, so four-to-one in favour.
  2. 2. A logistic regression coefficient of -0.10 for tenure gives an odds ratio of about 0.90. How do you interpret it?

    exp(-0.10) is about 0.90. The effect is on the odds, and it changes the probability by different amounts depending on the starting point.
  3. 3. Only 3% of cases are positive and a model reaches 97% accuracy. What is the best conclusion?

    Predicting 'negative' for everything also scores 97%. With rare positives, accuracy hides the failure; look at recall, precision and the confusion matrix.
  4. 4. You use class_weight='balanced' and the average predicted probability rises well above the true positive rate. What happened?

    Giving rare cases more weight shifts the estimated probabilities upward. Ranking may stay similar, but you should not read the outputs as true probabilities.

Summary

  • Logistic regression models the probability of the positive class through a linear score and the sigmoid function; coefficients are in log-odds.
  • Exponentiate coefficients to get odds ratios; remember they describe odds, not probabilities, and are conditional associations.
  • The model outputs probabilities; the class label comes from a threshold, which you should choose from the costs of each kind of error.
  • With rare positives, accuracy misleads. Use precision, recall and the confusion matrix.
  • class_weight="balanced" or resampling changes decisions but distorts probabilities; moving the threshold keeps them honest.
  • statsmodels suits explanation, scikit-learn suits prediction and validation.

Next, move beyond straight-line scores with decision trees, random forests and boosting.