Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 63 ยท Neural NLP and Transformers

Translation, Question Answering and Summarization

Learn the sequence-to-sequence idea behind translation and summarization, extractive question answering, and compute BLEU and ROUGE scores from scratch.

Classification gives you one label per text. Many of the most useful language tasks return text instead: a French sentence for an English one, a three-line summary of a long article, a short answer to a question about a passage. These are text generation tasks, and they are harder to build and much harder to evaluate, because there is no single right answer. "The cat sits on the mat" and "A cat is sitting on the mat" are both good translations of the same sentence.

In this lesson you will learn how the three tasks are framed, how encoder-decoder (sequence-to-sequence) models handle them, how extractive question answering finds an answer inside a passage, and how the standard metrics BLEU and ROUGE work. We write both metrics from scratch, so you will know exactly what the numbers in research papers measure and where they mislead.

Three tasks, one framing

Text in, text out

Translation A sentence in one language becomes another.
Question answering A question plus a passage becomes an answer.
Summarization A long text becomes a short one.
1 / 2

All three can be written as sequence-to-sequence (seq2seq) problems: map an input token sequence to an output token sequence. The classic solution is an encoder-decoder Transformer. The encoder reads the whole input and builds a contextual vector for every token. The decoder then writes the output one token at a time, as in LLM generation, but at each step it also uses cross-attention to look back at the encoder's vectors, so it can focus on the source words it is currently translating or summarizing. Models such as T5, BART and the Marian translation models follow this design; today, large decoder-only LLMs handle the same tasks from a prompt, often with similar quality.

Evaluating generated text

How do you score a translation automatically? The standard trick is to compare it with one or more human reference texts and count overlapping n-grams (see n-grams). Two metric families dominate:

  • BLEU is precision-oriented: of the n-grams the system produced, how many appear in the reference? It was designed for machine translation.
  • ROUGE is recall-oriented: of the n-grams in the reference, how many did the system capture? It was designed for summarization.

BLEU from scratch

BLEU combines three ideas. First, modified n-gram precision: count n-grams in the candidate, but clip each count to the number of times it appears in the reference, so repeating a word cannot inflate the score. Second, a geometric mean of the precisions for n = 1 to 4, so the output must match single words and longer phrases. Third, a brevity penalty, because without it a very short output made only of safe words would get perfect precision.

Python BLEU written by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An exact copy scores 1.0. The paraphrase, which a human would accept as a good translation, scores only 0.377, because sits and a do not match the reference, which is the first weakness of BLEU: it punishes valid wording differences. The word salad has a perfect BLEU-1 of 1.0 because it contains exactly the right words, but BLEU-4 is 0.0 because no 4-gram matches, so higher-order n-grams are what catch broken word order. The repeated the shows clipping at work: the reference contains the only twice, so only 2 of the 8 are credited, and BLEU-1 is 0.25. The short candidate has perfect precision, yet the brevity penalty exp(1 - 8/4) cuts the score to 0.368.

Real evaluations use a standard implementation such as sacrebleu, which fixes tokenization so scores are comparable between papers, and they are computed over a whole test set rather than on single sentences. Note that our simple version returns 0 whenever any n-gram level has zero matches; library versions offer smoothing for short sentences.

ROUGE from scratch

For summaries, we care whether the important content of the reference is present, so ROUGE reports recall (and usually precision and F1 too). ROUGE-N counts overlapping n-grams. ROUGE-L uses the longest common subsequence, which rewards words in the same order without requiring them to be adjacent.

Python ROUGE-1, ROUGE-2 and ROUGE-L

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "short" summary has a unigram precision of 1.00 (every word is in the reference) but a recall of only 0.38, so ROUGE correctly says it misses most of the content. The "long" summary has a unigram recall of 1.00 because it contains every reference word, but the precision is only 0.72 because of padding, and its ROUGE-L F1 drops to 0.45 since the words appear in a different order. The "good" summary balances both (ROUGE-L F1 0.87). The reason for reporting F1 is visible here: recall alone would reward the long summary for simply being long.

BLEU and ROUGE measure word overlap, not meaning or truth. A fluent summary with a wrong fact can score high, and an excellent paraphrase can score low. Treat them as a cheap way to compare versions of a system on the same data, and check samples by hand or with human ratings. Learned metrics (such as BERTScore) compare embeddings instead, and LLM judges are common, but they bring their own biases; see Evaluation, Bias and Responsible NLP.

Extractive question answering

In extractive QA, the answer is a span of text copied from a given passage (the setup of the SQuAD benchmark). A neural model, usually an encoder like BERT, reads the question and the passage together and predicts two numbers per token: the probability that the answer starts here and that it ends here. The answer is the best-scoring span. If the passage does not contain the answer, newer datasets teach the model to answer "no answer".

We cannot run such a model here, but the idea has two parts we can imitate with simple rules: find the sentence that best matches the question (as in retrieval), then pick a span of the right type inside it. The program below is an honest stand-in, a rule-based toy rather than a neural model. It matches sentences by shared content words, and looks for a year for "when" questions and for a capitalized name for other questions.

Python A rule-based extractive QA stand-in

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three questions find the right evidence sentence and the right span (1867, Paris, 1911). The rules are brittle: ask "Who shared the prize with Pierre Curie?" and the name-picking rule would return several names with no way to choose. A trained model learns this selection from tens of thousands of labeled questions and generalizes far better, using the attention between question and passage to find the span.

Using ready-made models

With the Hugging Face transformers library, all three tasks are a few lines. The snippets are read-only here, because they download model weights of hundreds of megabytes.

Python
from transformers import pipeline

# Translation (an encoder-decoder model)
translate = pipeline("translation_en_to_fr", model="Helsinki-NLP/opus-mt-en-fr")
print(translate("The meeting has been moved to Thursday."))

# Extractive question answering (an encoder with start and end predictions)
qa = pipeline("question-answering")
print(qa(question="Where did she move?",
         context="She moved to Paris in 1891 to study physics."))

# Abstractive summarization (an encoder-decoder model)
summarize = pipeline("summarization", model="facebook/bart-large-cnn")
print(summarize(long_article_text, max_length=60, min_length=20, do_sample=False))

Evaluate such systems on your own data with the metrics above, using a library implementation (sacrebleu for BLEU, the rouge-score package for ROUGE), and keep humans in the loop for anything user-facing. For summaries, check faithfulness: abstractive models can invent details that are not in the source. If faithfulness is critical, an extractive summarizer that only selects existing sentences is the safer choice.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the role of cross-attention in an encoder-decoder model?

    At each decoding step, cross-attention weights the encoder vectors so the decoder can focus on the part of the source that matters for the next output token.
  2. 2. Why does BLEU clip n-gram counts to the number of times they appear in the reference?

    Without clipping, a candidate made of the word "the" repeated would get a perfect unigram precision. Clipping credits each n-gram at most as often as the reference contains it.
  3. 3. ROUGE is mostly used for summarization because it emphasizes what?

    A summary should capture the important content of the reference, so ROUGE counts how much of the reference n-grams (or subsequence) appears in the candidate.
  4. 4. What does an extractive QA model predict?

    Extractive models score every token as a possible start and end of the answer, then return the best span copied from the passage.
  5. 5. A summary has a very high BLEU or ROUGE score. What can you conclude?

    These metrics only count overlapping words. They do not verify facts or meaning, so samples still need human or task-specific checks.

Summary

  • Translation, question answering and summarization are text-output tasks, usually framed as sequence-to-sequence; encoder-decoder models use cross-attention, and decoder-only LLMs can do the same from a prompt.
  • Output can be extractive (copy spans or sentences) or abstractive (write new text); abstractive output can hallucinate.
  • BLEU is clipped n-gram precision with a geometric mean and a brevity penalty; ROUGE measures recall (ROUGE-N for n-grams, ROUGE-L for the longest common subsequence).
  • Overlap metrics reward matching words, not meaning, so use them to compare systems and still inspect samples.
  • Extractive QA predicts start and end tokens of an answer span in a given passage.

Next, step back and look at measuring and improving whole systems in Evaluation, Bias and Responsible NLP.