Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 40 ยท Unsupervised Learning

K-Means Clustering

Group unlabeled data with scikit-learn KMeans: how the algorithm works, choosing k with inertia and silhouette scores, scaling, k-means++ and MiniBatchKMeans.

Everything so far had a target: a label or a number to predict. Unsupervised learning has none. You hand the algorithm only the features and ask it to find structure. The most common task is clustering: grouping similar rows together, such as customers with similar habits or documents about the same topic.

K-means is the classic clustering algorithm. You choose the number of clusters k, and it finds k centers so that every point is close to the center of its own cluster. It is fast, simple, and a good first thing to try, as long as you know its assumptions.

How k-means works

The k-means loop

Start Pick k starting centers.
Assign Each point joins its nearest center.
Update Move each center to its cluster mean.
Repeat Stop when assignments stop changing.

Each round can only lower the total squared distance.

1 / 2

The quantity being minimized is inertia: the sum of squared distances from each point to its assigned center. Lower is tighter, but inertia always falls as k grows, so it cannot choose k alone.

A first model

Python KMeans on three blobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Key arguments:

  • n_clusters: the k.
  • n_init: how many times to restart from different random centers; the best run (lowest inertia) is kept. K-means can get stuck in a poor solution, so several restarts matter.
  • init="k-means++" (default) spreads the starting centers apart, which gives better and more stable results than purely random starts.
  • random_state: for reproducible cluster numbering. The numbers 0, 1, 2 are arbitrary names, not an ordering.

Choosing k: elbow and silhouette

You rarely know k. Two common aids:

  • Elbow method: plot inertia against k and look for the bend where extra clusters stop paying off.
  • Silhouette score: for each point, compare the distance to its own cluster with the distance to the nearest other cluster. It ranges from -1 to 1; higher means better separated. Take the mean over points and pick the k with the highest value.
Python Inertia and silhouette for k = 2..8

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data was built from 4 blobs, and the silhouette peak should point at k = 4, while inertia keeps falling smoothly. On real data the elbow and the peak are often vague. Treat them as hints and also ask: do the clusters make sense, and are they useful for the business question?

Scaling matters

K-means uses Euclidean distance, so a feature measured in thousands drowns one measured in fractions. Standardize first.

Python Why you scale before clustering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

adjusted_rand_score compares two groupings (1 is identical, about 0 is random). Without scaling, the clustering follows the noisy large-scale feature and misses the real structure.

Evaluating clusters

With no labels, evaluation is indirect:

  • Silhouette and Davies-Bouldin (davies_bouldin_score, lower is better) measure compactness and separation.
  • Stability: re-run with different seeds or on bootstrap samples; good clusters reappear.
  • Profiles: group the original data by cluster and look at the means. If you cannot describe a cluster in a sentence, it is probably not useful.
  • If you do have reference labels, adjusted_rand_score or adjusted_mutual_info_score compare the clusters with them.
Python Profiling clusters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The crosstab compares clusters with the species names the algorithm never saw. One species separates cleanly; the other two overlap a little, which is typical.

Big data and limits

  • MiniBatchKMeans updates the centers from small random batches. It is much faster on large datasets with a small loss in quality.
  • K-means struggles with non-round clusters (rings, crescents), very different sizes or densities, and outliers that drag a center. DBSCAN and hierarchical clustering handle some of these cases.
  • It always returns exactly k clusters, even on data with no structure.
  • High-dimensional data makes distances less meaningful; reduce dimensions first, for example with PCA.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does k-means minimize?

    This quantity is called inertia (within-cluster sum of squares).
  2. 2. Why does KMeans use n_init restarts?

    Different starting centers can converge to different solutions. The run with the lowest inertia is kept.
  3. 3. Which value of silhouette score indicates the best-separated clustering?

    Silhouette ranges from -1 to 1 and higher means points sit well inside their cluster and far from others.
  4. 4. Why should features be standardized before k-means?

    K-means uses Euclidean distance, so large-scale features dominate unless every feature is put on a comparable scale.

Summary

  • K-means alternates between assigning points to the nearest center and moving centers to cluster means.
  • Choose k with inertia (elbow), silhouette scores and common sense; use n_init restarts and k-means++.
  • Scale features first; profile clusters to judge their usefulness.
  • It assumes compact, similar-sized, round clusters; use MiniBatchKMeans for huge data.

Next: Hierarchical Clustering and DBSCAN