Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 40 ยท Classification

Logistic Regression

Classify with LogisticRegression: how it turns a weighted sum into a probability, predict_proba, the C parameter, thresholds, multiclass problems and scaling.

Classification means predicting a category: spam or not spam, which digit, which species. Despite its name, logistic regression is a classification model, and it is the best first one to learn. It predicts a probability, and turning that into a class is a separate, adjustable step.

From a weighted sum to a probability

Logistic regression computes the same weighted sum as linear regression, z = intercept + w1*x1 + ..., then squeezes it through the sigmoid function so the result is always between 0 and 1:

probability = 1 / (1 + e^(-z))

Large positive z gives a probability near 1, large negative near 0, and z = 0 gives exactly 0.5.

Python The sigmoid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fit a binary classifier

We use the breast cancer dataset: 30 measurements per tumour, classified as malignant or benign.

Python LogisticRegression on breast cancer data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scaler matters: logistic regression is trained by an iterative optimiser that converges better with standardised features, and regularization (below) assumes comparable scales.

Probabilities and the 0.5 threshold

predict_proba returns one column per class. predict just picks the class with the highest probability, which for two classes means a 0.5 threshold.

Python predict_proba and a custom threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The threshold is a business decision. If missing a malignant tumour is worse than a false alarm, you would require high confidence before saying "benign". Thresholds, ROC and PR curves go deeper.

Reading the coefficients

Each coefficient is the change in the log-odds of the positive class per unit of the feature. Positive pushes toward class 1. After scaling, coefficient sizes are comparable.

Python Which features push the prediction?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Regularization strength: C

LogisticRegression applies an L2 penalty by default. The parameter C is the inverse of the penalty strength: small C means strong regularization (simpler model), large C means weak.

Python Effect of C

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As C grows the coefficients grow and the model fits the training data more tightly. Pick C by cross-validation (LogisticRegressionCV does it for you). For feature selection, use penalty="l1" with solver="liblinear" or "saga".

Multiclass problems

With more than two classes, LogisticRegression fits a softmax (multinomial) model by default, giving one probability per class that sum to 1.

Python Three classes on iris

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Strengths and limits

  • Fast, stable, and interpretable; gives calibrated-ish probabilities.
  • Draws a straight (linear) boundary: it cannot separate classes that need a curve unless you add features such as polynomial terms.
  • Sensitive to scale and to strongly correlated features, both helped by regularization.
  • A strong baseline: always try it before something fancier.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the sigmoid function do in logistic regression?

    The sigmoid squeezes any real number into (0, 1) so it can be read as a probability.
  2. 2. A smaller C in LogisticRegression means:

    C is the inverse of the penalty strength.
  3. 3. What does predict do for a binary classifier by default?

    The class with the higher probability wins, which for two classes is the 0.5 cutoff.
  4. 4. Why is logistic regression unable to separate a ring-shaped class from the middle?

    Without added non-linear features, a linear boundary cannot enclose a region.

Summary

  • Logistic regression outputs probabilities via the sigmoid and classifies with a threshold.
  • Use predict_proba to see confidence and adjust the threshold to your costs.
  • C controls regularization (smaller = stronger); scale features first.
  • Multiclass is handled with softmax.
  • Linear boundary: a strong, interpretable baseline.

Next: k-Nearest Neighbors Classifier