Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 63 ยท Text Classification

Clustering Documents

Group documents without labels. Run k-means on TF-IDF vectors, choose k with inertia and silhouette scores, and learn how to read clusters in scikit-learn.

Every classifier so far needed labels: someone had to mark each message as spam or not spam. Often you have the opposite situation. You have ten thousand support tickets, product reviews or news articles, nobody has sorted them, and you want to know what is in the pile. Clustering groups documents that look alike without being told the groups in advance. It is an unsupervised method: the output is a set of groups, and giving them names is your job.

The standard recipe is short. Turn each document into a TF-IDF vector, run k-means to split the vectors into k groups, then read the most characteristic words of each group. This lesson builds that recipe, shows how to choose k, and explains why you must treat the result as a hypothesis, not a fact.

How k-means works

k-means keeps k points called centroids, which are the averages of their groups. It repeats two steps until nothing changes:

The k-means loop

Start Pick k documents as starting centroids.
Assign Each document joins its nearest centroid.
Update Move each centroid to the mean of its group.
Repeat Stop when assignments stop changing.

Every pass can only lower the total distance.

1 / 2

"Nearest" means smallest Euclidean distance. That is a good fit for TF-IDF because scikit-learn's vectorizer already scales every document to length 1, so Euclidean distance between two documents moves in step with their cosine similarity. If you cluster raw counts instead, long documents sit far from the origin and drag the groups around, so normalize first.

The quantity k-means minimizes is called inertia: the sum of squared distances from each document to its centroid. Lower is tighter. Because the result depends on the starting centroids, scikit-learn runs the algorithm n_init times from different starts and keeps the run with the lowest inertia.

Clustering a small corpus

To see the whole pipeline with a known answer, the next example builds 60 short documents from three word pools (kitchen, space and football) plus a few filler words that appear everywhere. The pools are hidden from the clustering code, so we can check afterwards whether it rediscovered them. Real data is messier, but the mechanics are identical.

Python TF-IDF plus k-means

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The clusters have 20 documents each and their top words are easy to name: garlic, pasta and sugar are the kitchen; match, crowd and cup are football; orbit, comet and moon are space. The words come from cluster_centers_. A centroid is an average of TF-IDF vectors, so its largest coordinates are the words that this group's documents share and the other groups lack. Sorting a centroid's coordinates is the quickest way to label a cluster.

The label numbers (0, 1, 2) are arbitrary. Run again with another seed and the kitchen group may be called 2. Never hard-code the meaning of a cluster id.

Choosing k

We cheated slightly: we knew the answer was three. Real corpora do not announce it, so you try several values and compare. Two measures help.

  • Inertia always falls as k grows, because more centroids mean shorter distances (with k equal to the number of documents it is zero). Look for an "elbow", the point where adding clusters stops paying off much.
  • Silhouette score compares, for each document, the distance to its own cluster with the distance to the nearest other cluster. It ranges from -1 to 1. Near 1 means well separated, near 0 means the document sits on a border, negative means it is probably in the wrong cluster. The average over all documents can be compared across values of k, and the best k is where it peaks.
Python Inertia and silhouette for several k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Silhouette peaks at k = 3 (0.197). Beyond that, the groups of 20 are cut into pieces of 9 and 11 documents that are no better separated than before, so the score drops. The inertia column falls smoothly the whole way and has no sharp elbow, which is typical for text; do not expect a clean bend.

Notice how low the best silhouette is. For sparse TF-IDF vectors in thousands of dimensions, every pair of documents is far apart, and values around 0.1 to 0.3 are normal even for good clusterings. Use silhouette to compare values of k on the same data, not as an absolute grade. The final judge is always whether the clusters make sense when you read them.

When the vocabulary is large, reduce it first. Chaining TruncatedSVD (50 to 300 components) and Normalizer before k-means is called latent semantic analysis (LSA). It speeds up clustering a lot and often makes the groups cleaner, because synonyms land close together. See Embeddings from SVD.

Reading and using the clusters

A cluster is useful only when you can describe it. Three habits help:

  1. Top terms of each centroid, as above.
  2. The documents closest to each centroid, which are the most typical members. km.transform(X) returns the distance from every document to every centroid.
  3. Random members, not only the typical ones, because the edges show where the group is muddy.

When there are no hidden topics to compare against, a human reads about ten documents per cluster and decides whether the group has a clear theme. When you do have labels for a sample, the adjusted Rand index (ARI) measures agreement between two groupings: 1.0 is identical, about 0 is no better than random. It ignores how the groups are numbered.

Python Typical documents, ARI and new documents

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The most typical documents are mostly topic words, the filler words are scattered, and the ARI of 1.0 says the clusters match the hidden pools exactly. The last line shows a practical use: once fitted, predict assigns unseen documents to the nearest centroid. The vectorizer must be the same fitted object, because a new document has to be turned into the same columns.

When clustering disappoints

  • k-means assumes round, similar-sized groups. Elongated or very uneven topics get split or merged. Alternatives are hierarchical clustering (AgglomerativeClustering), which gives a tree you can cut at any level, and density-based methods such as DBSCAN, which find their own number of groups and mark outliers as noise.
  • A generic cluster appears. Short, vague documents ("thanks", "ok") often land together in a catch-all group. Filter them or accept the group.
  • Different seeds, different answers. Raise n_init, fix random_state for reproducibility and check that important findings survive a change of seed.
  • Clusters follow style, not topic. Boilerplate such as email signatures or HTML leftovers can dominate. Look at the top terms, then clean the text or add stop words.
  • Very large collections. MiniBatchKMeans fits on small random batches and scales to millions of documents with only a small quality loss.

Clustering is also close to topic modeling: LDA and NMF let a document belong to several topics at once, while k-means forces exactly one group per document.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does k-means need to be told before it starts?

    k-means is unsupervised, so it needs no labels, but it never decides the number of groups itself. You supply k and compare several values.
  2. 2. How do you usually find out what a cluster is about?

    Cluster ids are arbitrary numbers. The centroid's largest TF-IDF coordinates and the documents nearest to it tell you what the group shares.
  3. 3. Why can you not choose k by picking the value with the lowest inertia?

    More centroids always means shorter distances, so inertia keeps falling. You look for an elbow or use silhouette, which compares separation between clusters.
  4. 4. A silhouette score of 0.2 on TF-IDF vectors of real text usually means what?

    In thousands of sparse dimensions most documents are far from each other, so even a good clustering has small silhouettes. The score is for comparing options on the same data.
  5. 5. Why does scikit-learn run k-means several times with n_init?

    k-means only finds a local minimum, so different starts can end differently. Several starts and keeping the tightest result makes it more reliable.

Summary

  • Clustering groups unlabeled documents; k-means on L2-normalized TF-IDF vectors is the standard starting point.
  • k-means alternates assigning documents to the nearest centroid and moving centroids to the group mean, minimizing inertia.
  • Choose k by comparing inertia (look for an elbow) and silhouette (look for a peak) over a range, then confirm by reading the clusters.
  • Label clusters from the top terms of each centroid, the documents closest to it, and a random sample of members.
  • Cluster ids are arbitrary; use ARI to compare a clustering with known labels.
  • Reduce dimensions with SVD for big vocabularies, use MiniBatchKMeans for huge corpora, and consider hierarchical or density-based methods when groups are not round.

Next, pull the most important words and sentences out of a single document with Keyword and Summary Extraction.