Skip to content
Tutorials Center
My learning

Topics

Lesson 96 of 111 ยท Projects

Project: Customer Segmentation

Segment customers with machine learning: build RFM features, fix skew and scale, choose k with several metrics, compare k-means with a Gaussian mixture, and name the segments.

Marketing teams do not want 1,450 individual customers. They want a handful of groups they can talk to differently: a thank-you for the best customers, a welcome series for new ones, a win-back offer for those who drifted away. Finding such groups without labels is a job for clustering, and it is the one place in machine learning where nobody can tell you the right answer in advance.

In this project you will segment a synthetic customer base using RFM features (recency, frequency and monetary value), the standard trio for purchase data. You will see why skewed data needs a log transform and scaling before any distance-based method, choose the number of segments from several metrics that disagree, compare k-means with a Gaussian mixture model, check how stable the result is, and finish by turning cluster numbers into names a marketer can use.

RFM in three numbers

Recency Days since the last purchase.
Frequency How many orders in the period.
Monetary Total money spent.

Small recency is good; large frequency and monetary are good.

1 / 2

Step 1: The data, and why scaling matters

The generator below creates five hidden customer types (champions, loyal regulars, new customers, customers at risk and lost customers) with different recency, frequency and spending. We keep their true type in a column called true_group, only so we can grade our clustering at the end. A real project has no such column, so imagine it is not there while you work: the model never sees it.

RFM data has two typical problems. First, all three columns are right-skewed (a few customers spend or buy a lot), and monetary is the worst, with a skew above 2. Second, the columns live on very different scales: recency runs up to 365, frequency up to about 40, monetary up to thousands. Distance-based methods like k-means treat all columns equally, so the biggest numbers would decide everything (Scaling and Normalization). The standard fix is log1p to tame the skew, then StandardScaler to put every column on the same scale.

The last lines of the block prove the point: the same k-means with the same k, on raw versus prepared data, compared with the hidden truth using the adjusted Rand index (1.0 is a perfect match, 0 is random).

Python Step 1: build the customers and compare raw with prepared data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Raw k-means agrees with the hidden truth only to an index of about 0.33, while the prepared data reaches about 0.75, from the same algorithm, the same k and the same seed. The difference is entirely in preparation. Nothing in the raw run raised an error or looked wrong: clustering fails silently, which is why you must check your choices rather than trust the output.

In a production pipeline, the log transform and the scaler would be steps inside a pipeline, fitted on the customers you have today and then applied to new customers when you assign them to a segment.

Step 2: Choose the number of segments

How many segments are there? The data does not say. Three common signals each give a different kind of evidence (k-Means Clustering, Evaluating Clusters):

  • Inertia (sum of squared distances to the centers) always falls as k grows, so look for an elbow where the gains slow down.
  • Silhouette score compares how close a customer is to its own cluster with how close it is to the next nearest one. Higher is better.
  • BIC from a Gaussian mixture (Gaussian Mixture Models) rewards fit but penalizes extra components. Lower is better.
Python Step 2: inertia, silhouette and BIC for k = 2 to 8

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three signals do not agree. The silhouette peaks at 3 and 4 (both about 0.45) and slips only slightly at 5. Inertia falls steeply up to 5 and then flattens, so the elbow sits around 5. BIC keeps improving up to 7, because a flexible mixture can always find sub-structure in skewed data. There is no single correct k. What you do is combine the numbers with business sense: a marketing team can act on five segments, can barely tell apart seven, and gets little from two. Here we choose 5, which the elbow supports and the silhouette tolerates.

When metrics disagree, write down why you picked your k and how you would change it. A segmentation is a decision aid, and "we chose five because the elbow was there and the team can run five campaigns" is a perfectly good justification.

Step 3: Fit, compare, check stability and name the segments

With k fixed at 5 we fit both models and compare them. k-means assigns each customer to one segment and assumes round clusters of similar size. A Gaussian mixture also allows elongated clusters and gives each customer a probability for every segment, which is useful for the customers on a border.

We then profile each k-means segment with medians (not means, which skewed data pulls around) and its share of total revenue. A small rule translates each profile into a name. Finally a stability check refits k-means on five bootstrap resamples and measures how well each refit agrees with the original segmentation. If segments change wildly with a small change in the sample, they are not trustworthy.

Python Step 3: k-means vs GMM, profiles, names and stability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Gaussian mixture matches the hidden truth a bit better (about 0.84, against 0.75 for k-means), because the real groups are not perfectly round. Its soft memberships also show something k-means cannot: the first customer is 99% in one component, while a borderline customer would show split probabilities and could receive a different message.

Now read the profile. The first segment has about 280 customers who bought a median of 22 times for roughly 900, and they bring close to half of all revenue: these are the champions. The next segments follow the rules in the code: a large group of loyal regulars, new customers with few but recent orders, customers at risk who used to buy but have gone quiet, and a lost group with very high recency and low spending. Notice that the revenue share is far more unequal than the customer share. The roughly 280 champions (about a fifth of the customers) produce about half of the money, and a retention budget aimed at champions and at-risk customers is likely to be worth more than one aimed at the lost group.

The crosstab confirms that the names mostly line up with the hidden types, with a few borderline customers on the edges (for example 37 hidden loyal customers buy often enough to land in "Champions"). The bootstrap stability scores around 0.96 to 0.98 tell us the segmentation does not depend on which customers happened to be in the sample.

Real RFM data is rarely this clean. Real customers do not fall into five tidy blobs, so expect lower stability, overlapping segments and more judgment. If refits of your segmentation disagree strongly, reduce k or revisit the features before building marketing campaigns on it.

From segments to action

A segmentation is only useful when somebody acts on it. For each segment write a one-line playbook next to the profile: what is the goal, which channel, what offer, and how will you know it worked? For example, champions get early access and a thank-you; new customers get a second-purchase nudge within 30 days; at-risk customers get a win-back offer; lost customers are put on a low-cost schedule. Then measure the result with a proper test (A/B Testing), because a segment label alone proves nothing about what works.

Re-run the segmentation on a schedule and compare it with the previous one. Customers move between segments over time, and that movement (new customers becoming loyal, loyal customers becoming at risk) is often more valuable than any single snapshot.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do we apply log1p and StandardScaler before k-means on RFM data?

    k-means is built on distances. If monetary values run into thousands while frequency stays below 40, monetary decides almost every distance. The log tames the skew and scaling gives each column equal weight.
  2. 2. Inertia always decreases as k increases. How is it used to pick k?

    More clusters always fit better, so the minimum inertia is at the largest k and tells you nothing. The elbow marks diminishing returns, and it is combined with silhouette, BIC and business usefulness.
  3. 3. The silhouette peaks at k = 3, but the business can run five campaigns and the elbow is at 5. What is a sound decision?

    Unsupervised metrics are evidence, not verdicts. Choosing a k that is supported by one metric, tolerated by another, and actionable for the business is a legitimate and common decision as long as you record the reasoning.
  4. 4. What does a bootstrap stability check tell you?

    You refit on resampled data and compare the new segmentation with the original one. A high agreement score means the segments are a property of the customers, not of the sample.

Summary

  • RFM (recency, frequency, monetary value) turns purchase history into three features that describe customer value.
  • Skewed, differently scaled features must be log-transformed and standardized before distance-based clustering; on our data this lifted agreement with the hidden truth from about 0.33 to about 0.75.
  • Inertia, silhouette and BIC each answer a different question and may disagree, so choose k by combining them with business sense and document the choice.
  • A Gaussian mixture gives soft memberships and handles non-round clusters; k-means is simpler and faster.
  • Profile segments with medians and revenue shares, name them with clear rules, and check stability with bootstrap refits.
  • A segmentation proves its worth only when the segments lead to different actions, ideally tested in an experiment.

Next, build a model where mistakes have legal and ethical consequences in Project: Credit Risk Scoring.