Skip to content
Tutorials Center
My learning

Topics

Lesson 92 of 111 ยท Projects

Project: Customer Churn Classifier

Build a churn classifier on imbalanced tabular data: ColumnTransformer pipeline, cost-based decision threshold, probability calibration and feature importances.

A subscription company loses customers every month, and winning back a customer who has already left is much harder than keeping one who is thinking about leaving. The business question is therefore practical: which customers should we call with a retention offer? A churn model ranks customers by their risk of leaving, but a ranking alone does not tell you whom to call. That depends on what a call costs and what a lost customer is worth.

This project takes you through the whole path on synthetic data: an imbalanced dataset, a ColumnTransformer pipeline, two competing models, a decision threshold chosen by money instead of by habit, a calibration check, and an explanation of what drives the predictions. It builds on Classification Metrics, Thresholds, ROC and Precision-Recall Curves, Imbalanced Classes and Probability Calibration and Uncertainty.

From scores to a decision

Data Few churners, many loyal customers.
Pipeline Numbers and categories handled together.
Rank Compare models by average precision.
Decide Pick the threshold with the lowest cost.
1 / 2

Step 1: imbalanced data

Our telecom-style table has six columns: tenure_months, monthly_charge, support_calls, contract (month-to-month, one-year, two-year), internet (fiber, DSL, none) and autopay (yes/no). The target churned is 1 when the customer left. We build it from a logistic rule, so we know that short tenure, month-to-month contracts, fiber, many support calls and manual payment all raise the risk, and a longer contract lowers it.

About one customer in four churns (24%). That imbalance matters in three ways: plain accuracy will mislead, the split must be stratified so the test set keeps the same churn rate, and the decision threshold of 0.5 will be a poor choice.

Python Create the customers and check the balance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is the trap: a useless model that always says "stays" scores about 76% accuracy. From now on we judge models with average precision (the area under the precision-recall curve), which only rewards finding the churners, and later with money.

Step 2: a pipeline and two models

The preprocessing is one ColumnTransformer: numeric columns are scaled, categorical ones one-hot encoded. We compare a logistic regression (with class_weight="balanced" to pay more attention to the rare class) and a histogram gradient boosting classifier. Both sit in pipelines, and both are evaluated with stratified cross-validation on the training part only. The test set stays locked away.

Python Compare logistic regression and boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline's average precision equals the churn rate: that is "no skill". Both real models are far above it. They rank customers similarly well, because the data came from a logistic rule, so the simpler logistic regression is hard to beat. Note the Brier score column: it measures the quality of the probabilities, and class_weight="balanced" distorts them on purpose (it inflates the predicted risk of the rare class). That will matter in Step 4.

Step 3: a threshold chosen by cost

predict uses 0.5, which means nothing for a 24% churn rate. The right cutoff comes from the business:

  • Calling a customer who would have stayed anyway (a false positive) costs the offer, say $20.
  • Missing a customer who then leaves (a false negative) costs the lost customer's value, say $200.
  • A correctly flagged churner who accepts the offer is a saved customer. To keep the arithmetic simple we count the call cost for every flagged customer and assume the offer saves the customer in a fraction of cases, save_rate = 0.5.

So for each threshold we count the total cost on a set of customers: calls * 20 + (missed churners + churners we called but did not save) * 200. We need predictions that the model did not train on, so we choose the threshold on out-of-fold probabilities from the training set, then apply it, unchanged, to the test set.

Python Choose the threshold that minimizes cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cost curve has a clear valley: it is a poor idea to call nobody, and also a poor idea to call everyone with a risk above 50%, because the rare churners get missed. The best threshold sits well below 0.5 because a missed churner costs ten times as much as an unnecessary call. The test-set costs confirm it with a threshold that was fixed before the test data was touched. If your cost numbers change (cheaper calls, more valuable customers), the threshold moves; the model does not need retraining. This separation of scoring from deciding is one of the most useful habits in applied machine learning.

When the real costs are uncertain, plot the cost curve for a few plausible values and pick a threshold in the flat part of the valley. A threshold that is good across several assumptions is safer than one that is optimal for a single guess.

Step 4: calibration

A model's output is only a probability if it is calibrated: among customers scored around 0.3, about 30% should really churn. Calibration matters whenever the number itself is used (expected revenue loss, ranking offers by expected value) and not just compared with a threshold. We check it with a reliability table, bin by bin, and with the Brier score, then compare the balanced logistic regression, a plain one and a calibrated boosting model. CalibratedClassifierCV refits the model with internal cross-validation and learns a mapping from raw scores to honest probabilities.

Python Reliability table: does a 30% score mean 30%?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the two tables. With balanced class weights the scores run far above the real churn rate in every bin (a customer scored 0.5 does not churn half the time), and the Brier score is worse. The plain logistic regression is close to the diagonal. This is the trade: class weights can help a model pay attention to the minority class, but they destroy the probability interpretation. When you need both a good ranking and honest probabilities, train plain, calibrate, and handle the imbalance with the threshold, as in Step 3. Notice also how the calibrated boosting model brings its mean predicted risk into line with the real churn rate.

Step 5: what drives the predictions?

Stakeholders will ask why the model flags a customer. Linear coefficients give one answer; permutation importance gives a model-agnostic one: shuffle one feature in the test set and measure how much the score drops. A big drop means the model relied on that feature. We shuffle the original columns, so categorical features appear once rather than once per dummy column.

Python Permutation importance on the test set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the ranking critically. Importance says what the model uses, not what causes churn: a customer who has already decided to leave might phone support first, so support calls are a signal, not a lever. Also remember that correlated features share credit (see Feature Importance and Interpretation). Use the list to sanity-check the model against domain knowledge. If a feature appears that should not be known at prediction time, you have found a leak.

Wrapping up

A deployable churn solution is three separate pieces: a scoring model (plain logistic regression or calibrated boosting), a calibration check, and a decision rule (the cost-based threshold). Store the pipeline together with the threshold and the cost assumptions (see Saving Models and Deploying), monitor the churn rate and the score distribution over time, and revisit the threshold whenever the economics change.

Ideas for taking it further: add tenure-based interaction features, compare the model against the simple rule "month-to-month and tenure under 12 months", run an A/B test of the retention offer to measure the true save rate (we only assumed 50%), and check that error rates are similar across customer segments (see Fairness, Bias and Responsible ML).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use stratified splitting for the churn data?

    With a rare class, a random split can give the sets noticeably different churn rates. Stratifying keeps them equal, so the metrics are comparable.
  2. 2. A missed churner costs $200 and an unnecessary offer costs $20. Where will the best threshold be, compared with 0.5?

    When false negatives cost far more than false positives, flagging more customers is cheaper overall, so the cost-minimizing threshold falls below 0.5.
  3. 3. On which probabilities should the cost threshold be chosen?

    In-sample probabilities are too optimistic, and choosing on the test set would use it up. Out-of-fold predictions are honest and leave the test set for the final check.
  4. 4. What does class_weight='balanced' do to the predicted probabilities?

    The weights make mistakes on the rare class cost more during training, which inflates its predicted probability. Rankings may stay good, but the numbers are no longer honest probabilities.
  5. 5. Permutation importance shows 'support_calls' as important. What can you safely conclude?

    Importance describes how much the model uses a feature, not whether intervening on it changes the outcome. Causality needs experiments.

Summary

  • Churn data is imbalanced: stratify the split and judge models by average precision, not accuracy.
  • Put scaling and one-hot encoding in a ColumnTransformer inside a pipeline, and compare against a prior-based dummy baseline.
  • Separate scoring from deciding: choose the threshold that minimizes expected cost using out-of-fold probabilities, then verify once on the test set.
  • Class weights help ranking attention but distort probabilities; check calibration with a reliability table and Brier score, and calibrate when the numbers matter.
  • Permutation importance explains what the model uses, not what causes churn.

Next up: a text project in Project: Spam Filter.