Skip to content
Tutorials Center
My learning

Topics

Lesson 59 of 63 ยท Projects

Project: Extractive Text Summarizer

Build an extractive text summarizer in Python: split sentences, score them by word frequency and TextRank, and check quality with a ROUGE-style metric.

Nobody has time to read everything. A summarizer reads a document and returns a short version that keeps the important points. There are two families of methods. Abstractive summarizers write new sentences, which today means a large language model. Extractive summarizers pick the most important sentences from the original and put them together. Extractive methods are simple, fast, predictable and cannot invent facts, which is why they are still widely used.

In this project you build two extractive summarizers from scratch, a word-frequency scorer and TextRank (a graph algorithm in the spirit of PageRank), and then measure them with a ROUGE-style metric against a human-written summary. You will need sentence splitting, TF-IDF and cosine similarity.

The plan

Pick the best sentences

Split Break the text into sentences.
Score Rate every sentence.
Select Keep the top few.
Reorder Show them in original order.
1 / 2

Step 1: score sentences by word frequency

The oldest idea in automatic summarization is also a good one: the words a document uses most (apart from stop words) tell you what it is about, so sentences full of those words are probably central. Score each sentence as the sum of its words' frequencies divided by its length (otherwise long sentences always win), pick the top few, and print them in their original order so the summary still reads naturally.

We also build a lead baseline: simply the first three sentences. News articles put the key facts first, so the lead is a surprisingly strong baseline and any fancy method must beat it.

Python A frequency-based summarizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The text has 10 sentences and 146 words. The most frequent content word is bees (6 times), followed by hive and flowers (3 each), so the scorer is rewarding sentences about bees, hives and flowers. The highest score, 2.20, goes to sentence 6 (nectar, flowers, honey, hive). Sentence 2, about the queen, scores lowest at 1.00: its words (queen, eggs, job) appear only once in the whole text.

The frequency summary picks sentences 1, 3 and 6. It differs from the lead baseline only in one place: sentence 6 replaces sentence 2. Notice what both miss: the waggle dance, the most distinctive idea of the text, and the decline of bee colonies, its main message. Frequency scoring finds the vocabulary that repeats, not necessarily the point. A human-written reference will help us see that.

Step 2: a way to measure, ROUGE

How do you know if a summary is good? You could read it, and for a few examples you should. To compare methods on many documents, you compare against reference summaries written by humans. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) counts overlapping n-grams between the candidate and the reference:

  • ROUGE-1 counts shared single words, ROUGE-2 shared word pairs.
  • Recall: what fraction of the reference's n-grams appear in the candidate.
  • Precision: what fraction of the candidate's n-grams appear in the reference.
  • F1: the harmonic mean of the two.

Counts are clipped: if the reference contains bees twice, the candidate gets credit for at most two. Here is a complete ROUGE-N function on tiny examples, so you can verify the arithmetic by hand.

Python ROUGE-N written by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check two rows by hand. "the cat is on the mat" shares five of its six words with the reference (the twice, cat, on, mat), so ROUGE-1 precision and recall are both 5/6 = 0.83. For word pairs, three of five bigrams match (the cat, on the, the mat), giving 0.60. The degenerate candidate "the the the the the the" shows why counts are clipped: the reference has only two the tokens, so the overlap is 2, giving precision 0.33 instead of a perfect 1.00. A short candidate such as "a cat sat" has decent precision (0.67) but low recall (0.33), which is why you look at all three numbers.

ROUGE has well-known weaknesses. It rewards word overlap, not meaning: a summary that rephrases ideas with other words scores low, and a summary that copies the right words in a nonsensical order can score high. It is a cheap proxy for comparing systems, not a replacement for reading.

Step 3: TextRank

Word frequency treats each sentence on its own. TextRank asks a different question: which sentences are central to the document? Build a graph where every sentence is a node, and connect two sentences with an edge weighted by how similar they are (here, the cosine similarity of their TF-IDF vectors). Then run the PageRank idea: a sentence is important if it is similar to many other important sentences. The scores come from repeating one update until they settle:

score[i] = (1 โˆ’ d) / N + d ร— ฮฃ over j of (similarity[j][i] / ฮฃ over k of similarity[j][k]) ร— score[j]

with a damping factor d of 0.85. In words: each sentence shares its current score among its neighbors in proportion to similarity, and a small constant keeps every node alive.

The final example puts all pieces together: both summarizers and the lead baseline, evaluated against a reference summary that a person wrote in their own words.

Python TextRank and a ROUGE comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sentence scores show the graph at work. Sentence 6 (nectar, flowers, honey, hive) is again the most central at 0.166, followed by sentence 1 (0.139) and sentence 4, the waggle dance (0.135). Sentence 2, about the queen, is almost isolated at 0.015, because it shares almost no vocabulary with the other sentences.

Against our reference summary, TextRank scores best: ROUGE-1 F of 0.259 against 0.200 for the lead baseline and 0.190 for frequency scoring. ROUGE-2 is 0.072 for TextRank and 0.000 for the other two, which share no word pair with the reference. TextRank wins mostly because it includes sentence 4 ("... performs a waggle dance"), which the reference also mentions.

Be careful about what this proves. It is one document and one reference, and the reference is a free rewrite, so every score is low. The differences here illustrate the method; they do not show that TextRank is always better. On real news data the lead baseline is famously hard to beat. To compare systems properly, average ROUGE over hundreds of documents.

Step 4: improving the summarizer

Once you have a pipeline and a metric, improvements become experiments. Some good ones to try:

  • Length control. Return sentences until a word budget is reached rather than a fixed count.
  • Redundancy removal. Skip a candidate sentence if it is too similar to one already chosen (this is the idea behind Maximal Marginal Relevance).
  • Position features. Add a bonus for the first sentence of each paragraph, or for early sentences in news.
  • Better similarity. Replace TF-IDF with sentence embeddings so that sentences with different words but the same meaning link up.
  • Better sentence splitting. Abbreviations such as "Dr." and decimal numbers break the simple regex; see Sentence Segmentation.
  • Evaluate on more than one document. One text and one reference prove very little; use several documents with a few references each.

Extractive summaries can read oddly because sentences are lifted out of context: a pronoun like "she" or "it" may lose its referent. Abstractive models fix this by rewriting, at the price of occasionally stating something the source never said. See Translation, Question Answering and Summarization for the neural approach.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the frequency summarizer divide a sentence's total word frequency by its length?

    A raw sum grows with the number of words, so long sentences would always rank first. Averaging compares how topical a sentence is per word.
  2. 2. What does ROUGE-1 recall measure?

    Recall is overlap divided by the number of n-grams in the reference. Precision divides by the number of n-grams in the candidate instead.
  3. 3. In TextRank, what makes a sentence score high?

    Like PageRank, TextRank passes score along similarity edges, so sentences that are well connected to other well connected sentences end up central.
  4. 4. Why are overlap counts clipped in ROUGE?

    Without clipping, "the the the the" would match every "the" in the reference and look perfect. Clipping caps credit at the count in the reference.
  5. 5. What is a main weakness of ROUGE as a quality measure?

    ROUGE counts shared n-grams. A faithful summary in different words scores low, while a scrambled copy of the right words can score high.

Summary

  • Extractive summarization selects the highest-scoring sentences and keeps them in original order.
  • A frequency scorer rates sentences by the average corpus frequency of their content words; a lead baseline is the thing to beat.
  • TextRank builds a sentence similarity graph and uses PageRank-style iteration to find central sentences.
  • ROUGE-N compares clipped n-gram overlap with a reference, reporting precision, recall and F1.
  • ROUGE is a proxy for word overlap, not meaning: read examples and use several documents and references.

Next, make a program that talks back: Project: Rule and Retrieval Chatbot.