Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 63 ยท Text Classification

Topic Modeling with LDA and NMF

Discover hidden themes in unlabeled text with NMF and Latent Dirichlet Allocation in scikit-learn, read topics from their top words and choose the number of topics.

Classification needs labels: someone has to tell the model which documents are about sports. But what if you have 50,000 support tickets or news articles and no labels at all? Topic modeling is an unsupervised technique that looks at which words tend to appear together and proposes a small number of topics, each one a group of related words. It also tells you how much of each topic every document contains.

Topic models are a favorite exploration tool: summarize a huge archive, find themes in survey answers, or build features for other models. Two methods dominate classical NLP and both are in scikit-learn: NMF (non-negative matrix factorization) and LDA (Latent Dirichlet Allocation). This lesson explains the shared idea, runs both on the same corpus, shows how to read the results and how to choose the number of topics.

What a topic model gives you

Topics Each topic is a weighted list of words.
Document mixes Each document is a blend of topics.

No labels needed, but you must interpret the topics yourself.

1 / 2

The idea: documents as mixtures of topics

Imagine a document-term matrix X with one row per document and one column per word, like the bag of words or TF-IDF matrices from earlier lessons. Topic modeling approximates it as a product of two smaller matrices:

TEXT
X (documents x words)  ~  W (documents x topics)  *  H (topics x words)

H says which words belong to each topic; W says how much of each topic each document contains. With 3 topics, every document is summarized by just three numbers instead of thousands of word counts. That also makes topic models a form of dimensionality reduction.

  • NMF finds W and H with all entries at least 0, by minimizing the reconstruction error. Because nothing can be negative, topics behave like parts that are added together, which often gives readable topics. It works well on TF-IDF input.
  • LDA is a generative story. Each document picks a mixture of topics; each topic is a probability distribution over words; to write a word, choose a topic from the mixture, then choose a word from that topic. Fitting inverts the story to find the topics most likely to have produced the corpus. It expects raw counts, not TF-IDF.

A fixed number of topics, n_components, is chosen by you in both cases.

Running NMF and LDA

Our corpus has 150 short documents generated from three hidden themes (sports, tech and food). The documents never contain a label: each one mostly uses words from one theme, with about a quarter of its content words drawn from random themes, plus filler words. We fit both models with three topics, print the six strongest words of every topic, and then compare the dominant topic of each document with the hidden theme we generated it from.

Python Topics from NMF and LDA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the topics as a person would: the first NMF topic (recipe bake fresh kitchen oven sauce) is clearly food, the second (network update device code app server) is tech, and the third (player coach score stadium referee goal) is sports. LDA finds the same three themes, but numbers them in a different order: its topic 0 is sports, topic 1 food and topic 2 tech. Topic numbers are arbitrary labels; you name a topic by reading its words, never by its index.

The document 0 line shows the difference in output. NMF returns non-negative weights with no fixed total, while LDA returns a probability mixture that sums to 1. Document 0 is about tech, and LDA gives that topic most of the mass. The crosstab at the end compares each document's strongest topic with the hidden theme: NMF's strongest topic disagrees with the hidden theme for only 3 of 150 documents (three food documents landed on the tech and sports topics), while LDA agrees with the hidden theme for all 150 documents on this corpus. Real text is far less clean than ours, so do not expect such results; the point is that the method recovered the structure without ever seeing a label.

Stop words matter a lot for topic models. Without removing words like the and of, they can dominate every topic. See Stop Words and When to Keep Them. With real corpora, also use min_df to drop rare typos and max_df to drop words that appear in most documents.

Choosing the number of topics

Topic modeling has no single right answer for the number of topics. Too few merges distinct themes; too many splits one theme into near-duplicates. Two practical aids:

  • Fit measures. For NMF, reconstruction_err_ always shrinks as topics are added, so you look for an elbow where the gains flatten. For LDA, perplexity measures how well the model predicts the words (lower is better); it is best computed on held-out documents, whereas our quick check uses the training counts, which flatter bigger models.
  • Human judgment. Print the top words for several values and pick the number whose topics are coherent and distinct. Research on topic coherence shows that people often disagree with perplexity, so reading the topics is not optional.

The second example sweeps the number of topics from 2 to 6 on the same corpus. It also infers topics for a brand-new document that mixes sports and food, using transform on the fitted vectorizer and model.

Python How many topics, and new documents

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The NMF error falls steadily as topics are added (10.17 for two topics, 9.36 for three, 8.53 for six), with no sharp elbow. That is typical: error alone rarely settles the question. LDA's perplexity is lowest at three topics (24.1) and gets worse with more (25.8 at four, 28.8 at six): for this corpus both measures agree that three topics, the number we generated, is a good choice, but remember that LDA's score here is computed on the training counts.

The last block fits LDA with three topics and infers the mixture of an unseen document that mixes a sports sentence with a cooking sentence. The printed lines list each topic by its three strongest words, with the share of the document assigned to it: 0.72 to the food topic (bake recipe kitchen), 0.23 to the sports topic and only 0.05 to tech. The document mixes two themes and gets mass on both, but not equally, because it has more cooking words than sports words. A one-topic document concentrates almost all its mass on one topic.

When topic models help, and when they do not

  • Good for: exploring large archives, summarizing themes over time, grouping documents for review, generating features, and finding what you did not know to look for.
  • Not for: precise categorization. If you have labels, a supervised classifier will be more accurate and controllable.
  • Beware: topics can be junk (a topic of boilerplate words), mixed, or sensitive to preprocessing. Run with different seeds and settings, and keep a human in the loop to name and validate topics.
  • Short texts such as tweets give very little co-occurrence information; classic LDA struggles there.
  • Modern alternatives cluster sentence embeddings and label the clusters with their key words; you will meet this idea in Clustering Documents.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does topic modeling need as input?

    Topic modeling is unsupervised. It finds word groups from co-occurrence patterns, and you interpret and name the topics afterward.
  2. 2. In "X ~ W * H", what does the matrix H describe?

    H has one row per topic and one column per word, so each row is a topic as a weighted list of words. W holds the per-document topic weights.
  3. 3. Which input representation is the usual choice for LDA?

    LDA is a probabilistic model of word counts, so it expects count data from CountVectorizer. NMF is commonly run on TF-IDF.
  4. 4. Two runs give the same themes, but "topic 0" is food in one run and sports in another. Is something wrong?

    The order of topics depends on initialization and the algorithm. Only the content of each topic matters.
  5. 5. Why is choosing n_components with reconstruction error alone unreliable?

    More topics can always fit the data better. Judging the result needs held-out measures, a visible elbow and, above all, human reading of the topics.

Summary

  • Topic modeling is unsupervised: it approximates the document-term matrix as documents-by-topics times topics-by-words.
  • NMF works well on TF-IDF and returns non-negative weights; LDA is a probabilistic model on raw counts and returns mixtures that sum to 1.
  • A topic is a weighted list of words; read the top words to name it, because topic numbers are arbitrary.
  • Choose the number of topics with fit measures (error elbow, held-out perplexity) and by reading the topics.
  • Remove stop words and rare terms first, and use topic models for exploration rather than precise labeling.

Next, group whole documents in Clustering Documents.