Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 111 ยท Classification

Logistic Regression

Learn how logistic regression turns a straight-line score into a probability with the sigmoid, how to read its coefficients as log-odds, and how C controls regularization.

Linear regression predicts a number. But many questions have a yes-or-no answer: will this customer cancel, is this email spam, does this scan show a tumor? Logistic regression is the standard first model for these classification problems. Despite the word "regression" in its name, it predicts a probability between 0 and 1, and a decision rule then turns that probability into a class.

It is simple, fast, hard to overfit when regularized, and its coefficients are easy to explain. Even when you end up shipping a more complex model, logistic regression is the baseline it has to beat. It builds directly on Linear Regression and Multiple Linear Regression, and it uses the regularization ideas from Ridge, Lasso and Elastic Net.

From a score to a probability

Linear score z = intercept + w1x1 + w2x2 ...
Sigmoid Squashes any score into the range 0 to 1.
Probability p = P(class 1 | features).
Threshold Usually 0.5 turns p into a class.

The model is linear in the score, not in the probability.

1 / 2

From a straight line to a probability

Suppose you want to predict whether a student passes an exam from the hours they studied. A straight line through 0/1 labels can leave the 0-to-1 range and does not describe a "probability" at all. Logistic regression keeps the linear part but feeds it through the sigmoid (or logistic) function:

TEXT
z = b + w1*x1 + w2*x2 + ...        (the linear score, any real number)
p = 1 / (1 + e^(-z))               (the probability, always between 0 and 1)

When z is 0 the probability is exactly 0.5. Large positive scores push p toward 1, large negative scores push it toward 0. The curve is S-shaped, so extreme scores change the probability only a little, while scores near 0 change it quickly.

Python The sigmoid function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the model is still linear where it matters: the score z is a weighted sum of the features, and the decision boundary (where p = 0.5, i.e. z = 0) is a straight line, plane or hyperplane.

Fitting LogisticRegression

LogisticRegression follows the same estimator API you met in The scikit-learn API: fit, predict, plus predict_proba for the probabilities. Unlike linear regression there is no closed-form answer; scikit-learn finds the weights by minimizing the log loss (cross-entropy) with an iterative solver. Log loss punishes confident wrong answers very hard, which is exactly what you want from a probability model.

Python Pass probability from hours studied

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

predict_proba returns one column per class, in the order of classes_. Each row sums to 1. predict simply picks the class with the larger probability, which for two classes means "probability of class 1 above 0.5".

Reading the coefficients as log-odds

The odds of an event are p / (1 - p): a probability of 0.75 means odds of 3 to 1. Take the natural log and you get the log-odds, and that is exactly what the linear score z is:

TEXT
log( p / (1 - p) ) = b + w1*x1 + w2*x2 + ...

So a coefficient w means: when this feature goes up by one unit, the log-odds go up by w, holding the others fixed. Exponentiating gives something easier to say out loud: e^w is the odds ratio. If e^w = 1.5, each extra unit multiplies the odds by 1.5. A positive coefficient raises the probability, a negative one lowers it, and a coefficient near 0 means the feature barely matters.

The probability itself does not change by a fixed amount, because of the S-shape. The same one-unit step moves the probability a lot near 0.5 and hardly at all near 0 or 1.

Python Coefficients, odds ratios and the 50 percent point

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tip: Odds ratios are only comparable between features when the features are on comparable scales. A coefficient for "income in dollars" and one for "age in years" are not directly comparable. Standardize the features first if you want to compare their influence.

The decision boundary in two dimensions

With two features the boundary z = 0 is a line. Everything on one side is predicted class 1, everything on the other class 0. Plotting the predicted probability over a grid makes this visible.

Python Plotting the decision boundary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The black line is where the model is exactly undecided. Moving away from it, the color deepens as the model becomes more confident. A straight boundary is both the strength (stable, interpretable) and the limit of logistic regression: if the classes are separated by a curve, you need to add features such as polynomial terms, or switch to a nonlinear model.

Regularization and the C parameter

By default LogisticRegression applies an L2 penalty, just like Ridge. The knob is C, the inverse of the regularization strength, which trips up many people:

  • Small C (like 0.01): strong penalty, small coefficients, a simpler and smoother model.
  • Large C (like 100): weak penalty, larger coefficients, the model follows the training data closely.
  • The default is C=1.0.

Because the penalty depends on the size of the coefficients, scale your features first (see Scaling and Normalization); otherwise a feature measured in large units is penalized unfairly little or a tiny-unit one unfairly much.

Python How C shrinks the coefficients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As C grows the average coefficient grows and the training accuracy creeps up, while the test accuracy stops improving and may fall: the model is starting to memorize noise. With many features and few samples this matters a great deal. You choose C with cross-validation, covered in Cross-Validation and Hyperparameter Tuning; LogisticRegressionCV does it for you in one step.

You can also ask for an L1 penalty (l1_ratio=1 together with solver="saga", or solver="liblinear" with penalty="l1"), which drives some coefficients exactly to zero, the same sparsity effect as Lasso.

Warning: Perfectly separable data makes the unregularized solution blow up, because the weights can keep growing to push probabilities toward 0 and 1. A reasonable C prevents this. If you see a ConvergenceWarning, scale the features or raise max_iter before anything else.

When to use it

Logistic regression is a great first choice when you need calibrated-ish probabilities, a model you can explain, fast training on wide or large data, or a strong baseline. It struggles with strongly nonlinear boundaries and feature interactions unless you create those features yourself. For how to judge the predictions it makes, continue with Classification Metrics.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the sigmoid function do in logistic regression?

    The sigmoid takes the unbounded linear score z and converts it to a probability p = 1 / (1 + e^-z), which always lies between 0 and 1.
  2. 2. A coefficient of 0.4 for a feature means that a one-unit increase in that feature, with the others fixed, ...

    Coefficients act on the log-odds. The probability change depends on where you are on the S-curve, so it is not a fixed amount.
  3. 3. You decrease C from 1.0 to 0.01. What happens?

    C is the inverse of the regularization strength, so a smaller C means a stronger penalty and smaller coefficients.
  4. 4. Where is the decision boundary of a two-feature logistic regression with the default 0.5 threshold?

    p = 0.5 exactly when z = 0, and z = b + w1x1 + w2x2 = 0 describes a straight line in the plane.

Summary

  • Logistic regression passes a linear score through the sigmoid to get a probability, then thresholds it (default 0.5) to get a class.
  • It is fit by minimizing log loss; predict_proba gives class probabilities, columns ordered as classes_.
  • Coefficients are changes in log-odds per unit; exp(coef) is the odds ratio. Compare features only on comparable scales.
  • The decision boundary is linear; add features (or switch models) for curved boundaries.
  • C is the inverse regularization strength: small C is simpler. Scale features and tune C by cross-validation.

Next up: Classification Metrics.