Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 76 ยท Projects

Project: Customer Churn Analysis

A complete churn analysis on synthetic subscription data: frame the question, clean, explore, compare models with a baseline, and turn scores into recommendations.

Churn, customers leaving a subscription, is one of the most common business problems in data science, and it exercises almost everything in this course: framing, cleaning, exploration, modelling, evaluation and a recommendation someone can act on. In this project you will do the full job on a synthetic subscription company, in five steps, and finish with a short written conclusion.

The data is generated with a fixed seed, so every number in the text matches what you will see when you press Run. Because the data is synthetic, we also know the truth that created it. Real projects never do, so treat the "known answer" as a luxury for learning: you can check whether your analysis recovers what is really going on.

flowchart LR
    A["1. Frame<br/>the question"] --> B["2. Load and clean"]
    B --> C["3. Explore"]
    C --> D["4. Model and<br/>evaluate"]
    D --> E["5. Decide and<br/>recommend"]

The five steps of the project. Each step below is a self-contained runnable block.

Step 1: Frame the question

A good project starts with a decision, not a model (see Framing a Good Question). Here is the framing, written the way you would write it for your team:

  • Business question: the retention team can call about 300 customers each month. Whom should they call, and what should the company change to keep customers?
  • Data question: which customers are most likely to cancel in the next period, and which factors are associated with cancelling?
  • Target: churned (yes or no). One row per customer.
  • Success criteria: the model must beat a simple baseline (the overall churn rate, and a one-rule baseline), and the top-scored group must contain clearly more churners than a random group of the same size.
  • Metrics: ROC AUC for ranking quality, precision and recall at the chosen calling threshold, and expected profit from the calls.
  • Out of scope: proving why customers leave. A model shows associations; any claim that a change in contract type will reduce churn needs an experiment (A/B Testing).

Step 2: Load and clean

The generator below builds a messy "export" like a real system would produce: duplicates, inconsistent plan labels, impossible tenures, missing fees and a yes/no target stored as text. Clean only what is clearly wrong and document each change. Anything that depends on the data's statistics (like filling missing numbers with a median) is left for the modelling pipeline so it is learned from the training data only.

Python Step 2: build the messy export and clean it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four defects, four fixes: duplicates dropped by ID, labels normalized, impossible tenures turned into missing (they are errors, not real zero-month customers), and the target converted to 0 and 1. The 40 duplicates were real copies of the same row; before dropping duplicates in your own project, check they really are the same customer and not two events. The missing fees and tenures remain for the pipeline to handle.

Step 3: Explore

Exploration answers "what does churn look like?" before any model exists. Look at the overall rate, then split it by the variables a business can understand. Remember that you are looking for hypotheses; with several comparisons some differences will be chance (see A Systematic EDA Process).

Python Step 3: churn rate by segment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The dashed line is the overall rate. The printed rates by band give the exact numbers. Churn falls from 51% for customers in their first six months to 20% for those with more than two years, and it rises from 30% with no support calls to 60% with four or more. Month-to-month customers churn much more than annual ones (44.9% against 25.6%), and customers on autopay churn less (33.2% against 43.8%). The variables overlap (month-to-month customers also call support more), so a model is needed to separate their effects.

Step 4: Model and evaluate

Split first, then fit everything on the training part only. Compare models against baselines: predicting the overall rate for everyone (which gives an AUC of exactly 0.5) and a one-rule model. Then try logistic regression, which is easy to explain, and a gradient-boosted tree model, which can capture interactions. A single Pipeline handles imputing, scaling and encoding so that no information from the test set leaks in (see Pipelines and Hyperparameter Tuning).

Python Step 4: baselines, logistic regression and boosted trees

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the results in this order. First, the one-rule baseline already beats chance (AUC 0.618 against 0.5), which sets the bar: a complex model that only matches it is not worth deploying. Second, logistic regression (AUC 0.738) and boosted trees (0.735) finish essentially tied. That is typical for tabular data with mostly simple effects (and here we know the truth is a logistic relationship, so the linear model is well specified). When a simple, explainable model is as good as a complex one, choose the simple one.

One test set gives one number. Before declaring a winner on a difference this small, use cross-validation to see how much the score moves between splits (see Train/Test Splits and Cross-Validation).

Step 5: From scores to a recommendation

The retention team can call 300 customers a month, out of a pool of about 900 in the test set. So the useful question is not "what is the AUC" but "how many churners are in the top group, and what do the calls earn?". The block below also reads the logistic model's coefficients as odds ratios and measures feature importance by shuffling (permutation importance), so the story is not just "the black box says so".

Python Step 5: top-group lift, profit and drivers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The top 300 are 59.7% churners against 38.1% for the average customer, a lift of 1.57x, and they reach 179 of the 343 churners in the test set (52%). Under our cost assumptions the campaign earns about 5,700 against about 3,000 for 300 random calls. Random calling is profitable here only because churn is so common (38%), so the model's real gain is the difference between the two numbers. That is the result the business cares about, in its own terms.

The odds ratios and the permutation importance tell the same story at the top: tenure matters most (one standard deviation more tenure roughly halves the odds of churning, a ratio of 0.50), followed by contract type and support calls (1.40 per standard deviation); autopay protects (0.76), while late payments and fee add less. One warning about reading coefficients: plan and monthly_fee are strongly related (each plan has its own price), so their separate coefficients, such as pro looking riskier while a higher fee looks safer, cannot be interpreted one at a time. Judge such groups of related columns together. The overall ordering agrees broadly with the formula that generated the data, which is the reassurance a synthetic project can give.

Caveats you should always write down:

  • Associations, not causes. Moving a customer to an annual contract may not lower their churn; people who choose annual contracts differ from those who do not.
  • The calling economics are assumptions (cost, save value, success rate). Change them and the best group size changes. Report a range.
  • One time period. A real project validates on a later period than the training period, and monitors drift after launch (From Model to Decision).
  • Who gets called? Check whether the top group is skewed toward a group of customers that should not be treated differently (Ethics and Fairness).

The written conclusion

End every project with a short note a manager could read in a minute. For this one:

Recommendation. Call the 300 highest-scored customers each month; about six in ten of those customers churn, against four in ten overall (a lift of 1.6x), and the campaign earns nearly twice as much as calling 300 random customers under our cost assumptions. Focus retention offers on new, month-to-month customers with several support calls or late payments, and test an annual-contract incentive and an autopay incentive in a randomized trial before assuming they reduce churn. A simple logistic model performs as well as the boosted trees, so we propose the logistic model for its transparency. Next steps: validate on the following quarter's data, and monitor the score distribution monthly.

Notice what it contains: the action, the evidence in plain numbers, the model choice with its reason, an honest limit (causation) and next steps.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is the imputation of missing fees done inside the modelling pipeline instead of during cleaning?

    Statistics used to fill values are learned from data. Learning them on the full table lets test information leak into training; a pipeline fits them on the training split only.
  2. 2. A one-rule baseline (month-to-month = will churn) already ranks well. What does this tell you?

    Baselines set the bar. If a complex model barely improves on a simple rule, the simple rule may be the better choice.
  3. 3. Logistic regression and boosted trees score about the same. Which should you usually prefer?

    At comparable accuracy, the more transparent and cheaper model is easier to trust, monitor and maintain.
  4. 4. What does 'lift of 2x in the top 300' mean?

    Lift compares the rate of the outcome in the targeted group with the overall rate. Here, the group is twice as rich in churners as a random group.
  5. 5. Why shouldn't the recommendation say 'switching customers to annual contracts will cut churn'?

    Observational data cannot separate the effect of the contract from who chooses it. A randomized trial can.

Summary

  • Start from the decision (who to call, with what capacity), then choose the target, metrics and success criteria.
  • Clean only clear errors in code and document them; leave data-dependent imputation to the pipeline.
  • Explore churn by understandable segments; overlapping variables need a model to separate their effects.
  • Compare models with baselines and prefer the simpler model when performance is equal.
  • Evaluate in business terms: lift in the top group and expected profit, not just AUC.
  • Write the conclusion with action, evidence, model choice, caveats (association, assumptions) and next steps.

Next: Project: Sales Dashboard and Forecast.