Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 40 ยท Projects and Practice

Project: Predict Customer Churn

Build an imbalanced-classification project in scikit-learn: prepare a churn table, compare models with the right metrics, tune a threshold and turn predictions into a ranked call list.

Churn means a customer stops using a service. If you can predict who is likely to leave, a retention team can contact them first. That makes this a classic business classification problem with two typical difficulties: churners are a minority (imbalance) and mistakes have different costs.

You will build the full workflow: generate a realistic table, choose metrics, compare models, pick a decision threshold from costs and finish with a ranked list.

The data

The table is generated in code so the project runs anywhere. Churn probability rises with month-to-month contracts, many support calls and a short tenure, and falls with a long contract.

Python Generate the customer table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About one customer in four churns, so accuracy will not be enough: a model that says "nobody churns" would be right about 77% of the time and useless.

Churn project plan

Imbalance Use precision, recall, AUC.
Pipeline Scale numbers, encode categories.
Costs Retention offers cost money.
Output A ranked list of who to call.

The model's job is ranking; the threshold is a business choice.

1 / 2

Compare models with the right metrics

Use stratified cross-validation and look at ROC AUC and average precision, which judge the ranking and are not distorted by imbalance.

Python Baseline, logistic regression and gradient boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The logistic model should be at least as good as the boosted trees here, because the data was generated from a logistic relationship: simpler models win when the signal is simple. class_weight="balanced" changes the probabilities but barely changes the ranking, so the AUC stays about the same.

Choose a threshold from costs

Suppose a retention offer costs 15 per contacted customer, and a customer who would have left but is kept is worth 120. Contact customer i if the expected gain is positive: p * 120 - 15 > 0 (assuming the offer always works, a deliberately simple assumption). We choose the best cutoff on validation predictions.

Python Cost-based cutoff and the final test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "contact everyone" strategy wastes money on loyal customers; "contact nobody" earns nothing; the model-driven list sits in between and should beat both. The tuned threshold lands near the break-even point 15 / 120, which is what economics predicts when probabilities are calibrated (see calibration and thresholds).

Explain it and deliver a list

Python Drivers of churn and a ranked call list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is a lift check: customers in the top 10% by predicted probability churn far more often than average. Lift tables like this are how business teams judge a churn model.

Next steps

  • Try hyperparameter tuning for the boosting model and compare via average precision.
  • Add features: ratio of fee to tenure, calls per month, a flag for a recent plan change.
  • Recalibrate probabilities and re-derive the threshold when the offer cost changes.
  • Monitor in production: score distribution drift and the real churn rate among contacted and uncontacted customers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is accuracy a poor main metric for a 25% churn problem?

    With imbalanced classes a trivial model scores high accuracy. Use recall, precision, AUC or average precision.
  2. 2. Where should the decision threshold be chosen?

    Choosing it on the test set would make the test result optimistic.
  3. 3. What does a lift of 3 in the top decile mean?

    Lift compares the positive rate in a top-scored group with the overall positive rate.
  4. 4. Why might class_weight='balanced' leave the AUC almost unchanged?

    Reweighting classes moves the decision boundary and probabilities; the order of scores changes little.

Summary

  • For imbalanced business problems, judge models by AUC and average precision, not accuracy.
  • The model ranks customers; the threshold converts ranking into action using costs and benefits.
  • Choose the threshold on validation predictions and confirm profit on the untouched test set.
  • Use permutation importance and lift tables to explain and sell the result.

Next: Project: Customer Segmentation