Skip to content
Tutorials Center
My learning

Topics

Lesson 67 of 111 ยท Unsupervised Learning

Evaluating Clusters

Evaluate clusters with and without labels: silhouette, Davies-Bouldin, Calinski-Harabasz, adjusted Rand index, stability by resampling and cluster profiles.

With a classifier you can ask "how many did it get right?" With clusters there is usually no answer key. That is the central difficulty of unsupervised learning, and it is why a clustering that looks tidy can still be useless.

There are three kinds of evidence, and a careful analysis uses all of them. Internal scores use only the data and the cluster labels. External scores compare the clusters to known labels, which you have only in experiments or when some labeling exists. Stability and interpretation ask whether the clusters survive resampling and whether a person can use them. This lesson builds each one on top of k-means and DBSCAN results.

Three kinds of evidence

Internal Compact and well separated?
External Do they match known labels?
Stable Do they survive resampling?
Useful Can a person name and act on them?

No single number settles it. Combine them.

1 / 2

Internal scores

Internal scores judge a clustering from the geometry alone. All of them reward clusters that are compact (points close to their own center) and separated (centers far from each other).

  • Silhouette score (silhouette_score): for each point, (b - a) / max(a, b), where a is the mean distance to its own cluster and b is the mean distance to the nearest other cluster. Ranges from -1 to 1; higher is better. Near 0 means the point sits on a boundary, negative means it is probably in the wrong cluster.
  • Davies-Bouldin index (davies_bouldin_score): the average, over clusters, of the worst similarity ratio between a cluster and its closest neighbor. Lower is better; 0 is the best possible.
  • Calinski-Harabasz index (calinski_harabasz_score): the ratio of between-cluster to within-cluster variance, also called the variance ratio. Higher is better; it has no upper limit, so only compare values for the same data.
Python Three internal scores across k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Silhouette (0.763) and Calinski-Harabasz (about 8,650) both peak at k=4, the true number of groups, while Davies-Bouldin narrowly prefers k=3 (0.325 against 0.335). Disagreement like this is normal, and a reason to look at the clusters themselves. Remember what each one measures. They are blind to meaning, and they are biased toward convex, ball-shaped clusters.

Internal scores favor k-means-like shapes. On two interleaved crescents, a correct DBSCAN clustering can score worse on silhouette than a wrong k-means split. Do not use these scores to compare algorithms that define clusters differently.

Python Silhouette can reward the wrong answer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The silhouette prefers k-means (0.49 against 0.33), even though DBSCAN got the real structure right (see its ARI against the truth). The silhouette measures how ball-like and separated the groups are, not whether they match what the data means.

Looking inside the silhouette

The average hides detail. silhouette_samples returns one value per point, so you can see whether some clusters are weak, or whether a few points are misplaced.

Python Silhouette per cluster

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A healthy cluster is a wide bar mostly to the right of the average line. A cluster whose bar is thin, or has many points below zero, is a candidate for merging or splitting.

External scores: when you have labels

If some ground truth exists (a labeled sample, or a synthetic benchmark), compare it with the clusters. These metrics ignore the cluster numbering.

  • Adjusted Rand index (ARI) (adjusted_rand_score): counts pairs of points that are grouped the same way in both labelings, adjusted so that random labelings score about 0. 1 is a perfect match; negative values mean worse than chance.
  • Adjusted mutual information (AMI) (adjusted_mutual_info_score): an information-theoretic version, also near 0 for random labelings.
  • Homogeneity, completeness and V-measure: homogeneity asks whether each cluster contains only one class; completeness asks whether each class sits in one cluster; V-measure is their harmonic mean. They show how a clustering fails: splitting a class lowers completeness, mixing classes lowers homogeneity.
Python External scores and what they reveal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Renaming the cluster ids changes nothing, random labels score about 0, and the last two rows show the diagnostic power. Splitting a class keeps homogeneity at 1 but lowers completeness; merging classes keeps completeness at 1 but lowers homogeneity.

Stability by resampling

A real structure should not change much when the data changes a little. A simple stability test:

  1. Draw two random subsamples of the data.
  2. Cluster each one separately with the same settings.
  3. Predict cluster ids for all points with each model and compare the two labelings with ARI.
  4. Repeat many times and average.

A high average (close to 1) means the algorithm finds the same groups however it is sampled. A low average means the clusters depend on luck. This works for models with a predict method such as KMeans and GaussianMixture.

Python Which k is stable?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data has four groups. Stability stays near 1 up to k=4 (merging real groups, as k=2 and k=3 do, is also stable, so stability does not punish too few clusters) and then drops steadily: 0.81 at k=5 and 0.66 at k=7. When k is larger than the number of groups, k-means has to cut real clusters at arbitrary places, and different samples cut them differently. Use stability to rule out too many clusters, and the silhouette to rule out too few.

Interpreting clusters

A clustering is only finished when somebody can describe it. Build a profile table: for each cluster, the size and the mean of every feature, and compare with the overall mean. Features that differ most define what the cluster is. Then name the clusters and check whether the names make sense for the problem.

Python Profiling clusters with pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first table shows each group's size and averages in real units, and the second shows which features push each cluster away from the overall average. From these two tables you could name the three groups, which matter far more to a business than any score. The same workflow reappears in Project: Customer Segmentation.

Putting it together

A practical routine for a new dataset:

  1. Scale the features and try a few algorithms and values of k.
  2. Compare internal scores, but only between runs of the same kind of algorithm.
  3. Check stability under resampling.
  4. Plot the clusters in two dimensions (see PCA).
  5. Profile the clusters and decide whether they are meaningful and actionable.
  6. If you have partial labels, check ARI or AMI against them.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which metric is better when its value is LOWER?

    Davies-Bouldin measures the average similarity between each cluster and its closest neighbor, so smaller is better. Silhouette, ARI and Calinski-Harabasz are better when higher.
  2. 2. Why is the adjusted Rand index suitable for comparing clusters to true classes?

    ARI compares pairs of points, so renaming cluster ids does not matter, and the adjustment makes chance-level labelings score about 0.
  3. 3. A clustering has homogeneity 1.0 but low completeness. What is the likely problem?

    Homogeneity 1.0 means every cluster contains one class only. Low completeness means a class is spread over more than one cluster, so clusters were split.
  4. 4. Why is silhouette a poor way to choose between k-means and DBSCAN on crescent-shaped data?

    Silhouette is based on distances to cluster members, which favors convex groups. A correct non-convex clustering can score lower than a wrong convex one.
  5. 5. What does a low mean ARI between clusterings of two random subsamples suggest?

    If different samples of the same data yield different groups, the grouping depends on chance rather than on structure in the data.

Summary

  • Evaluate clusters with internal scores, external scores (if labels exist), stability and interpretation.
  • Silhouette and Calinski-Harabasz are better higher, Davies-Bouldin is better lower; all favor compact, round clusters, so compare only like with like.
  • silhouette_samples shows which clusters or points are weak.
  • ARI, AMI, homogeneity and completeness compare against labels and ignore cluster numbering.
  • Stability by resampling checks that groups are not an accident of one sample.
  • Profile every cluster: sizes and feature averages, then give each one a meaningful name.

Next, reduce many features to a few, which also helps you plot clusters: Principal Component Analysis.