Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 48 ยท MLlib

MLlib Clustering: K-Means and Silhouette

Group unlabeled data with Spark MLlib KMeans: scale features, choose k with the silhouette score, read cluster centers and assign clusters to rows.

Classification and regression need a label: a known answer to learn from. Often you do not have one. You have customers and want natural segments, or sensors and want to see which behave alike. Clustering is the unsupervised task of grouping rows so that rows in a group are similar and rows in different groups are not. K-means is the workhorse: it is fast, simple and parallelizes well, which is why it is in Spark MLlib.

This lesson shows how k-means works, why features must be scaled first, how to choose the number of clusters with the silhouette score, and how to read the centers and attach cluster ids to your rows. It builds on MLlib Overview and Pipelines. Spark cannot run on this site, so the PySpark code is read-only; the runnable examples are plain-Python stand-ins.

How k-means works

You pick k. The algorithm then repeats two steps until nothing changes (or a maximum number of iterations is reached):

  1. Assign each point to its nearest center (by Euclidean distance).
  2. Update each center to the mean of the points assigned to it.

It starts from initial centers and each round moves them to better positions. The goal is to minimize the within-cluster sum of squared distances. Two facts matter in practice. The result depends on the starting centers, so a good initialization matters: Spark's default initMode="k-means||" is a parallel variant of k-means++ that spreads the starting centers out. And k-means assumes roughly round, similar-sized clusters; it does badly on long snakes or rings.

In a cluster, each iteration is one pass over the data: every partition assigns its points and computes partial sums per center, and the driver combines them. That is why k-means scales so well.

Preparing features

K-means uses distances, so a feature measured in thousands drowns a feature measured in fractions. Always scale first:

Python
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.clustering import KMeans

shoppers = spark.createDataFrame([
    (120.0, 3), (95.0, 2), (30.0, 12), (28.0, 15), (500.0, 1), (480.0, 2),
    # ... more rows
], ["annual_spend", "visits_per_month"])

assembler = VectorAssembler(inputCols=["annual_spend", "visits_per_month"], outputCol="raw")
scaler = StandardScaler(inputCol="raw", outputCol="features", withMean=True, withStd=True)

StandardScaler rescales each feature to mean 0 and standard deviation 1. By default withMean is False (to keep sparse vectors sparse), so set withMean=True for dense data like this. Categorical columns need indexing and one-hot encoding first (see MLlib Classification), and one-hot columns usually make poor k-means input, so consider whether the clusters should use only numeric behavior.

Fitting and reading the result

Python
kmeans = KMeans(featuresCol="features", k=3, seed=42, maxIter=50)
model = Pipeline(stages=[assembler, scaler, kmeans]).fit(shoppers)

clustered = model.transform(shoppers)
clustered.select("annual_spend", "visits_per_month", "prediction").show()

km = model.stages[-1]
for center in km.clusterCenters():
    print(center)                       # centers live in the SCALED feature space

The prediction column holds the cluster id (0 to k-1). Cluster numbers have no meaning and may change between runs, so describe clusters by their content: average spend and visits per cluster, found with groupBy("prediction").agg(F.avg(...)). Centers are in the scaled space, so to explain them in original units compute averages of the original columns per cluster, as above.

Always set seed for repeatable results; the same data and seed give the same clusters on the same Spark version.

Python K-means from scratch on two customer features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing k with the silhouette score

How many clusters are there? The algorithm cannot tell you; you must try several values and score each. The silhouette compares, for every point, two distances: a, the average distance to the other points in its own cluster, and b, the average distance to the points in the nearest other cluster. The point's silhouette is (b - a) / max(a, b):

  • near +1: the point sits well inside its cluster;
  • near 0: it is on the border between two clusters;
  • negative: it is probably in the wrong cluster.

The average over all points is the model's score. Spark computes it with ClusteringEvaluator:

Python
from pyspark.ml.evaluation import ClusteringEvaluator

evaluator = ClusteringEvaluator(featuresCol="features", predictionCol="prediction",
                                metricName="silhouette", distanceMeasure="squaredEuclidean")

scaled_df = Pipeline(stages=[assembler, scaler]).fit(shoppers).transform(shoppers)
for k in range(2, 8):
    model = KMeans(featuresCol="features", k=k, seed=42).fit(scaled_df)
    score = evaluator.evaluate(model.transform(scaled_df))
    print(k, round(score, 3), round(model.summary.trainingCost, 2))

Choose the k with the highest silhouette, or use the elbow method on trainingCost (the within-cluster sum of squares): the cost always falls as k grows, but at the right k the improvement suddenly flattens. Do not blindly trust either; a k that gives a slightly lower score but segments a business can act on is often the better choice. Cache the scaled DataFrame before the loop, because each fit reads it many times (see Caching and Persistence).

Python Silhouette score for k = 2 to 5

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Spark's ClusteringEvaluator computes the silhouette with an efficient distributed formula for squared Euclidean or cosine distance; it does not compare every pair of points as the stand-in does, which would be impossibly slow at scale. The stand-ins also use random starting centers (best of five) instead of k-means||. The definitions are the same.

Spark MLlib also provides BisectingKMeans (divisive, often more stable), GaussianMixture (soft, elliptical clusters with probabilities per cluster) and LDA (topic models for text). K-means is the right first choice for numeric features and large data; reach for the others when clusters are not round or you need probabilities.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why scale features before k-means?

    Euclidean distance is dominated by features with big numeric ranges. StandardScaler puts features on a comparable scale.
  2. 2. A cluster has the id 2 in one run and 0 in another run with a different seed. Is something wrong?

    The ids carry no meaning. Compare clusters by their averages or centers.
  3. 3. A silhouette score near 0 for a point means:

    Near +1 means well inside, near 0 means on the border, negative means probably misassigned.
  4. 4. Which statement about the training cost (within-cluster sum of squares) is true?

    More centers can only reduce the squared distances, so the lowest cost is always the largest k. The elbow shows diminishing returns.

Summary

  • K-means alternates assigning points to the nearest center and moving each center to the mean of its points; each pass is one parallel scan.
  • Scale features with StandardScaler (withMean=True for dense data) before clustering, and set a seed.
  • KMeans adds a prediction column with arbitrary cluster ids; describe clusters by their averages in original units.
  • Choose k by trying a range and comparing ClusteringEvaluator silhouette scores and the elbow in trainingCost.
  • Consider BisectingKMeans or GaussianMixture when clusters are not round.

Next: turn words into numbers in MLlib Overview and Pipelines, where the text feature stages are collected.