Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 63 ยท Text Classification

Keyword and Summary Extraction

Pull keywords and key sentences out of a document. Rank words with TF-IDF, write TextRank by hand, and build an extractive summarizer step by step.

Sometimes you do not want to classify or cluster a document; you want to know what it is about, quickly. A list of five keywords can tag an article, feed a search index or label a cluster. Two or three sentences can stand in for the whole text on a results page. Both jobs are forms of extraction: we select the most important pieces that are already in the text instead of writing new ones.

This lesson shows two ways to score importance. TF-IDF compares a document with a collection and finds words that are special to it. TextRank needs no collection at all: it builds a graph from one document and lets importance flow through it, like PageRank flows through web links. We use TextRank twice, once on words (keywords) and once on sentences (an extractive summary).

Two ways to find what matters

TF-IDF Words special to this document compared with others.
TextRank Words and sentences that many others point to.

TF-IDF needs a corpus. TextRank works on one text.

1 / 2

Keywords with TF-IDF

When you have a collection of documents, TF-IDF already does the work: the words with the highest weights in a document are the ones that are frequent there and rare elsewhere. Add bigrams (ngram_range=(1, 2)) so that a phrase like wild yeast can compete with single words, and sublinear_tf=True so that repetition helps less.

Python Top TF-IDF terms of each article

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first article is summed up by starter and bread, the second by bike, council, plan and streets, the third by galaxy and image. Look at the weaker slots, though. For article 0 they are bread depends, bread rise and complex, each with the same score of 0.16, and for article 3 the tail is cycling, cycling days and days, again tied and overlapping. This is the usual problem of TF-IDF keywords: with only a handful of documents, words that occur once all tie, and a phrase ranks right next to its own parts. In practice you use thousands of documents, remove overlapping candidates, and often keep only nouns and noun phrases (see part-of-speech tagging).

TF-IDF has one more limit: it needs other documents. For a single text, or a text unlike the rest of your corpus, use a method that looks only inside.

TextRank for keywords

PageRank ranks web pages by the idea that a page is important if important pages link to it. TextRank applies the same idea to text:

  1. Keep only the candidate words (drop stop words).
  2. Make each distinct word a node. Connect two words if they appear within a small window of each other (here, two words apart or fewer).
  3. Give every node a score of 1, then repeat: a node's new score is (1 - d) + d * (sum of each neighbor's score divided by that neighbor's number of neighbors). The damping factor d is 0.85.
  4. After a few dozen rounds the scores settle. The highest-scoring words are the keywords.

A word scores high when it is connected to many words, especially to words that are themselves well connected. In a text about a sourdough starter, starter appears in many contexts and collects the most links.

Python TextRank for keywords, written by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

starter wins by a wide margin (3.357, with 16 neighbors), followed by bread (1.551). Everything else sits near 0.95, because each remaining word appears once and is linked to about four neighbors. Compare this with TF-IDF: TextRank did not need any other document, but it can only rank words that appear more than once in a connected way. On short texts the tail of the list is nearly arbitrary, and it is common to ask only for the top few.

The textbook TextRank filters candidates by part of speech (nouns and adjectives) before building the graph, and then merges adjacent top words into phrases such as wild yeast. We skipped both to keep the code short. Libraries such as pytextrank and yake do them for you.

Extractive summaries

The same graph idea works for sentences. Now each sentence is a node, and the weight of an edge is how similar two sentences are. A sentence that resembles many other sentences, especially other central ones, is probably at the heart of the text. We measure similarity by the cosine of TF-IDF vectors, built over the sentences of this one document.

Ranking uses the same update rule, in matrix form. Normalize each row of the similarity matrix so it sums to 1 (a sentence passes its score to neighbors in proportion to similarity), then repeat rank = (1 - d)/n + d * (P.T @ rank).

Python TextRank over sentences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two highest-ranked sentences are 3 (the volunteers and the museum) and 1 (the keeper and the lamp). They share the most vocabulary with the rest of the text: keeper, lamp, tower, museum. The summary prints them in their original order, which matters, because reading them by score would scramble the story. The last lines print the lead baseline, the first two sentences, which is a strong competitor for news articles because journalists put the main facts first. In this text the opening sentence (about the building date) has the lowest rank, so TextRank and the baseline share only one of their two sentences. That is a useful reminder: always compare a clever method with the trivial one on your own data.

Strengths and limits of extraction

  • Fast, transparent and faithful. Every output sentence is real text from the source, so the summary can never invent a fact. You can also explain why a sentence was chosen.
  • Choppy output. Selected sentences may start with "He" or "This" and lose their referent, or repeat each other. A common fix is to skip a sentence whose similarity to an already chosen one is high (Maximal Marginal Relevance).
  • Length and position. Long sentences collect more shared words and win too often; a length penalty or a minimum length helps.
  • No paraphrase or compression. Only abstractive neural models can rewrite; see translation, question answering and summarization. To check a summarizer against a reference text, you will use the ROUGE metric there and again in Project: Extractive Text Summarizer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main difference between TF-IDF keywords and TextRank keywords?

    TF-IDF needs document frequencies from a corpus. TextRank builds a word graph from a single text and ranks words by how well connected they are.
  2. 2. In keyword TextRank, when do two words get an edge?

    Co-occurrence within a sliding window is the link. Words that keep appearing next to many other words become central in the graph.
  3. 3. What does a node's score depend on in TextRank?

    Like PageRank, a node is important if important nodes link to it. That is why the iteration repeats until the scores settle.
  4. 4. Why should an extractive summary print the chosen sentences in their original order?

    Rank decides which sentences are chosen, but readers need the story in the sequence the author wrote it.
  5. 5. Why is the lead-2 baseline worth checking against a summarizer?

    Cheap baselines tell you whether a complicated method really helps on your data.

Summary

  • Extraction selects the most important words or sentences already in a text, so it never invents content.
  • TF-IDF keywords are the highest-weighted terms of a document within a corpus; add bigrams and watch for overlapping candidates.
  • TextRank builds a graph (words linked by a window, or sentences linked by cosine similarity) and iterates a PageRank-style update with damping 0.85.
  • Print extractive summaries in original order, and compare against a lead baseline.
  • Limits: dangling references, redundancy, bias toward long sentences and no rewriting.

Next, move from documents to word-by-word structure with Part-of-Speech Tagging.