Skip to content
Tutorials Center
My learning

Topics

Lesson 61 of 111 ยท Unsupervised Learning

Clustering Basics

Learn what clustering is, how similarity and feature scaling shape the groups, and how k-means, agglomerative and DBSCAN compare on blobs and moons.

Everything so far in this tutorial has been supervised: every row came with a label or a target, and the model learned to predict it. Clustering drops that safety net. You hand the algorithm a table of numbers and ask, "Which rows belong together?" Nobody tells it what the groups are, how many there are, or whether there are any at all.

That makes clustering useful for exploring customers, documents, sensor readings or genes when you do not yet know the categories. It also makes it easy to fool yourself, because there is no single right answer to check against. This lesson builds the vocabulary: what a cluster is, why distance and scaling matter so much, and how three popular algorithms behave on the same data. The next lessons go deep on each: k-Means, hierarchical clustering, density clustering and Gaussian mixtures.

What is clustering?

No labels Only features go in.
Groups come out Each row gets a cluster id.
Similarity rules Close rows land together.

The algorithm finds structure; you decide if it means anything.

1 / 2

What a cluster is

A cluster is a set of rows that are more similar to each other than to the rest. "Similar" is not built into the data. You choose it, usually by picking a distance: a number that is small for similar rows and large for different ones. The default is the Euclidean distance, the straight-line distance you learned in school, extended to as many features as you have.

Three things follow from this:

  • Clustering results depend on which features you include and how you measure distance.
  • Different algorithms define "a group" differently: a compact ball around a center, a branch in a tree, or a dense region separated by empty space.
  • The output is only a list of integers, one per row. The integer 2 has no meaning beyond "same group as the other 2s". Running the algorithm again may swap the numbers.

scikit-learn gives every clusterer the same interface you met in The scikit-learn API: create the object with its settings, call fit, and read the labels from labels_. Many also offer fit_predict, which does both in one call.

Python Your first clustering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output has one label per row, and cluster_centers_ gives the average position of each group. Notice that we did not pass y_true to the model. We only keep it here because the data is synthetic, so we can later check how well the groups were recovered. With real data you will almost never have it.

Similarity and scaling

Distance adds up differences across all features. If one feature is measured in thousands and another in fractions, the big one decides everything. This is the same reason scaling matters for k-nearest neighbors.

Imagine customers described by yearly income in dollars and the number of purchases per month. Two customers who differ by 5,000 dollars in income and 8 purchases are, in raw distance, almost purely separated by income. The purchases are noise next to it.

Python Scaling changes the clusters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On raw features the algorithm cuts along income, the large-valued axis, and mixes up the real groups. After standardizing both features the split follows purchases, which is where the structure really was. Scaling is not always the right call (if a feature's unit has real meaning, rescaling can hide it), but you should decide on purpose and not by accident.

Different algorithms, different ideas

Here are the three algorithms you will meet most often, and what each one assumes about a group.

Algorithm A cluster is... You choose Handles odd shapes?
KMeans a ball around a center number of clusters n_clusters no
AgglomerativeClustering a branch of a merge tree number of clusters or a distance cutoff, and the linkage depends on linkage
DBSCAN a dense region neighborhood radius eps and min_samples yes, and it can label noise

The honest way to compare them is to try them on shapes where you know the answer. make_blobs gives round, well-separated groups. make_moons gives two interleaved crescents, which no ball-shaped model can separate.

Python Three algorithms on blobs and moons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The adjusted Rand index (ARI) compares the found groups with the known ones: 1 means identical grouping, values near 0 mean no better than chance. On the overlapping blobs k-means does best (ARI about 0.94) and agglomerative clustering is close (0.83), while DBSCAN with this eps does worse (0.57) because it labels the sparse edges as noise (-1). On the moons k-means and agglomerative clustering cut straight through the crescents (about 0.5), while DBSCAN, which follows dense regions, recovers them perfectly (1.00). No algorithm wins everywhere, and DBSCAN needs its eps set sensibly. Evaluating Clusters covers ARI in detail.

Cluster ids are arbitrary names. A labeling that calls the left group 0 and the right group 1 is the same grouping as one that swaps them. That is why ARI, which ignores names, is used instead of plain accuracy.

Soft clusters and message passing: fuzzy c-means and affinity propagation

Two more ideas are worth knowing because they relax assumptions that k-means makes.

Fuzzy c-means gives every point a membership in every cluster instead of a single label. A point halfway between two groups may belong 45% to one and 40% to another (and 15% to a third), and memberships always add up to 1. The algorithm is the k-means loop with soft weights: compute the cluster centers as membership-weighted averages, then recompute memberships from the distances to those centers. A single parameter m (the fuzzifier, usually 2) controls the softness: close to 1 behaves like k-means, larger values spread the memberships out. Fuzzy memberships are useful when groups genuinely overlap (customers who fit two segments) or when you want to flag borderline points for a human to check. Scikit-learn has no fuzzy c-means, but it is short to write.

Affinity propagation takes a different approach. Points exchange messages with each other about how well suited each point is to be the exemplar (representative) of another. After the messages settle, the exemplars are real data points and the other points follow their favorite exemplar. You do not set the number of clusters. Instead you set a preference, a number per point that says how willing it is to be an exemplar: lower (more negative) preferences give fewer clusters. The default is the median similarity, which often gives too many clusters. It needs the full similarity matrix, so it scales poorly beyond a few thousand points.

Python Fuzzy c-means from scratch, and affinity propagation's preference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The hard version of the fuzzy result (the cluster with the highest membership) is identical to k-means (agreement 1.0, both 0.90 against the truth), so the fuzzy method adds information rather than changing the answer: 42% of the points have a clear owner (membership above 0.9), 11% are borderline (highest membership below 0.6), and the most ambiguous point splits its membership 0.39, 0.25 and 0.36 while k-means just reports one label. The fuzzifier does what it should: the average top membership falls from 0.98 to 0.83 to 0.59 as m goes from 1.2 to 3.5. Affinity propagation is very sensitive to the preference. The default found 11 clusters (ARI 0.31), a preference of -20 found the correct 3 (ARI 0.90), and a preference of -1 produced 22. In practice, scan the preference and judge the result with cluster evaluation metrics.

Choosing k: the gap statistic

You have seen two ways to choose the number of clusters for k-means: the elbow of the inertia curve, and the silhouette score. The gap statistic is a third, more principled one. For each k it compares the log of the within-cluster spread of your data with the log spread you would get from random data with no clusters covering the same range. A big gap means your data is far more clustered than noise would be. The recommended rule picks the smallest k whose gap is within one standard error of the next one's.

Python Elbow, silhouette and gap statistic side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On four generated blobs, the inertia keeps falling smoothly after the elbow, which is hard to read, while the silhouette peaks at k = 4 (0.781) and the gap statistic also peaks at k = 4 (2.010) and picks it by the rule. The second check is the important one: on uniform random points with no clusters, the gap statistic correctly answers k = 1, while the silhouette score would still return a value for any k you try. That ability to say "there are no clusters" is the gap statistic's special strength. Its cost is time, since it runs k-means on several reference data sets for every k.

The no-ground-truth problem

In the experiments above we could compute ARI because the data was synthetic. In real projects you have no labels, so you cannot say "86% correct". What you can do instead:

  • Check internal quality. Measures such as the silhouette score ask whether points are closer to their own cluster than to others. They are cheap but biased toward ball-shaped clusters.
  • Check stability. Cluster on two random halves or on resampled data. If the groups change wildly, they are probably not real.
  • Check usefulness. Profile each cluster: average feature values, sizes, examples. Can a person name the groups, and would a decision differ between them?
  • Check against a downstream task. If cluster membership improves a later prediction or a business action, it is doing something.

None of these proves the clusters are "true". Many datasets have no natural clusters at all, and k-means will still happily split a uniform cloud into any number of pieces you ask for.

Python Clusters in data with no clusters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The algorithm always returns exactly the number of groups you request, and the silhouette scores hover around 0.4 for every k, even though there is no gap anywhere to find. A moderate score on its own proves nothing. Treat a clustering as a hypothesis about the data, not a fact about it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main difference between clustering and classification?

    Classification learns from labeled examples; clustering gets only features and has to discover structure by itself. It does not guarantee the groups mean anything.
  2. 2. Why should you usually standardize features before clustering with Euclidean distance?

    Distance adds up differences across features, so a feature measured in thousands drowns out one measured in fractions unless you put them on a comparable scale.
  3. 3. KMeans labels one group 0 and another group 1. On a second run the same groups get 1 and 0. What does this tell you?

    Only the grouping matters. That is why metrics like the adjusted Rand index ignore the numbering.
  4. 4. Which algorithm from this lesson can separate two interleaved crescents (make_moons)?

    KMeans and Ward linkage prefer compact, ball-shaped groups. DBSCAN follows connected dense regions, so with a suitable eps it traces each crescent.

Summary

  • Clustering groups rows without labels; the output is one arbitrary integer per row.
  • Similarity is your choice, usually Euclidean distance, so feature selection and scaling shape the result.
  • Standardize features unless their raw units carry real meaning.
  • KMeans finds ball-shaped groups, agglomerative clustering builds a merge tree, and DBSCAN finds dense regions and marks noise.
  • Without ground truth, judge clusters by internal scores, stability and usefulness, and remember that any algorithm will return groups even in structureless data.

Next, the most widely used algorithm in detail: k-Means Clustering.