Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 63 Β· Neural NLP and Transformers

Machine Translation in Depth

From word alignment to Transformers: learn how machine translation works, with IBM Model 1 by EM, corpus-level BLEU and a beam search demo in plain Python.

Machine translation (MT) is the oldest big goal of the field: in the 1950s researchers hoped to translate Russian into English with a few hundred rules. The task turned out to be much harder than that. Languages differ in word order (la casa roja is literally the house red), in gender and agreement, in how many words a concept needs and in what each word means. Today, translation systems handle dozens of languages in one model, and they are also a good way to understand almost every idea in this chapter.

The modern NLP tasks lesson introduced translation in one paragraph and wrote BLEU for a single sentence. Here we go deeper into three things that explain how MT works. First, how a system can learn which word translates which from nothing but pairs of sentences (word alignment, with the classic IBM Model 1). Second, how to evaluate a system properly on a whole test set (corpus BLEU with several references and smoothing). Third, how a decoder turns word probabilities into one good sentence (beam search). In between, you get the history from rules to Transformers.

Four eras of machine translation

From rules to neural nets

Rule-based Dictionaries and hand-written grammar rules.
Statistical Phrase tables learned from parallel text.
Neural seq2seq One network reads and writes, with attention.
Transformers and LLMs Multilingual models, prompted or fine-tuned.

Each era replaced hand-made parts with learned ones.

1 / 2

Rule-based systems (1950s to 1980s) combined a bilingual dictionary with grammar rules written by linguists. They were precise on the covered cases and brittle everywhere else. Statistical MT (1990s to 2010s) learned from parallel corpora: millions of sentence pairs, such as parliament proceedings published in several languages. It estimated which words and phrases translate each other, and used an n-gram language model to choose fluent output. Neural MT (since about 2014) replaced the phrase tables with a single recurrent encoder-decoder network, and from 2017 the Transformer took over. Today, strong translation also comes from large multilingual models trained on text in many languages, as described in pretrained models, and from LLMs given an instruction and an example.

Learning word alignments: IBM Model 1

Imagine a collection of English-Spanish sentence pairs and no dictionary. Which Spanish word means house? In the pair (the house, la casa) you cannot tell whether casa goes with the or with house. But casa also appears with a house and the red house, and always together with house. Statistical MT turned that observation into an algorithm: the IBM models (1990s) assign a probability t(f | e) that English word e is translated as foreign word f, and learn these numbers with the EM algorithm (expectation-maximization):

  1. Start with uniform probabilities.
  2. E-step: for every sentence pair, split each foreign word's "credit" among the English words in that pair, in proportion to the current probabilities.
  3. M-step: re-estimate each t(f | e) from the accumulated credit.
  4. Repeat. Words that keep appearing together pull each other's probabilities up.

IBM Model 1 ignores word order completely: every foreign word may come from any English word in the same sentence pair. It is far too simple to translate with, but it is the seed of every later alignment model, and it fits in 30 lines. The corpus below is a handful of pairs that we wrote ourselves.

Python IBM Model 1: learning a dictionary with EM

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

From ten tiny pairs the program discovered that libro is book (probability 1.00) and perro is dog (1.00). It also found the order-free alignment el to the, perro to dog, rojo to red, although the Spanish word order is different from English. The rest of the output shows what Model 1 cannot do. house splits its probability, 0.80 for casa and 0.20 for la, because la always appears next to casa in this corpus. the goes 0.80 to el and 0.20 to la, and a is split evenly between un and una. Spanish articles and adjectives change with grammatical gender, and red is half roja and half rojo for the same reason. A model that translates words one by one has no way to know that casa is feminine and that therefore the article must be la and the adjective roja: that needs context, which is exactly what the later phrase-based and neural models provide.

From seq2seq to Transformers

In neural machine translation (NMT), one network reads the source sentence and writes the target one. The first version, sequence-to-sequence (seq2seq), squeezed the whole source into one fixed-size vector, and the quality dropped quickly for long sentences. Attention (2014) fixed this: at every output step the decoder looks back at all the encoder states and decides which source words matter right now. This also gave back something like an alignment, as a soft, learned matrix.

flowchart LR
    S["Source tokens: the red house"] --> E["Encoder: contextual vector per token"]
    E --> X["Cross-attention"]
    P["Words written so far: la casa"] --> D["Decoder"]
    X --> D
    D --> N["Probabilities for the next word"]
    N --> W["roja"]

A Transformer translation step: the decoder reads what it has written so far, attends to the encoder's vectors, and scores every word of the target vocabulary.

The Transformer replaced the recurrent layers with self-attention, which made training parallel and much faster, and with it MT quality jumped. Three practical details are worth knowing:

  • Subword tokens. Vocabularies are built with byte-pair encoding or similar methods, so rare words and names are split into pieces instead of becoming unknown.
  • Multilingual models. One model can serve many language pairs. A language tag in the input chooses the target language, and low-resource languages benefit from related, high-resource ones.
  • Decoding. The model gives probabilities, not sentences. Turning them into one output is the job of the decoder search that we implement below.

Evaluating at corpus level: BLEU with several references

The BLEU written in the earlier lesson scores one candidate against one reference. In practice, BLEU is a corpus-level metric: you add up the clipped n-gram matches and the n-gram counts over the whole test set, and only then divide, so a few short sentences do not dominate. Two other refinements are standard: multiple references (a candidate n-gram is clipped by its maximum count in any reference, because there are several valid translations), and smoothing, because a short sentence with zero 4-gram matches would otherwise score exactly 0.

Python Corpus BLEU with several references and smoothing

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

System A copies the references in three of four sentences and changes one word, so its BLEU is 0.865, with precisions of 0.96 for single words falling to 0.82 for 4-grams. System B paraphrases and drops words (please shut the door leaves out quietly), and scores 0.500: the precision falls from 0.80 for words to 0.31 for 4-grams, because higher-order matches break easily when the wording changes. The last line is the reason for smoothing. the meeting begins at nine against the meeting starts at nine has no matching 4-gram, so plain BLEU is exactly 0, while smoothed BLEU gives 0.359. Libraries such as sacrebleu implement standard smoothing and fixed tokenization, so scores are comparable between papers. Use sacrebleu's chrF (a character n-gram F-score) for languages where tokenization is unclear, and learned metrics such as COMET or BERTScore, which compare meaning with neural models, for finer comparison. Also remember the limits discussed in the earlier lesson: BLEU says nothing about whether a fluent translation is true to the source. Human evaluation is still the final judge, especially for medical, legal and safety text.

Decoding: how the sentence is chosen

At each step, an NMT model gives a probability for every word in the target vocabulary. Greedy decoding takes the most likely word every time and never looks back. Beam search keeps the k best partial translations (the beam) at each step, extends each of them with every possible next word, and keeps the best k again; it is described in detail, with length penalties, in beam search and repetition control. Beam search needs far more work than greedy search, but translation systems typically use a beam of 4 to 5 because the greedy path often misses a better full sentence.

The demo below uses a small hand-made table instead of a neural network: the probability of the next word given the words so far, for the source the red house. It is built so that the most probable first word, la, leads to a trap: the model prefers to write roja next and stop, which leaves out casa.

Python Greedy decoding vs beam search for a translation

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Greedy writes la roja (the red) with probability 0.150: it picked la, then the likeliest follow-up roja, then stopped, so the house is missing. Beam 1 is the same as greedy search. With a beam of 2 the search also keeps la casa alive, even though casa is slightly less likely than roja at step 2, and later finds la casa roja with probability 0.196, a better translation of the whole sentence. A beam of 3 also surfaces casa roja (0.162), which is correct apart from the missing article. Notice that the right answer dropped out only because a locally best choice was wrong, the classic reason for a beam. Real systems add a length normalization (otherwise short outputs are favored, since every extra word multiplies by a number below 1) and a coverage penalty that discourages leaving source words untranslated, a problem that our toy table has exactly.

Using pretrained translation models

Training your own system needs millions of sentence pairs and GPUs, so in practice you start from a pretrained model. This code needs a model download, so it is read-only:

Python
from transformers import pipeline

translate = pipeline("translation_en_to_es", model="Helsinki-NLP/opus-mt-en-es")
result = translate("the red house is on the hill", num_beams=5, max_length=60)
print(result[0]["translation_text"])

# Score a batch with sacrebleu (pip install sacrebleu)
import sacrebleu
hypotheses = ["la casa roja esta en la colina"]
references = [["la casa roja estΓ‘ en la colina"]]       # one list per reference set
print(sacrebleu.corpus_bleu(hypotheses, references).score)
print(sacrebleu.corpus_chrf(hypotheses, references).score)

Common problems in real deployments: named entities and numbers (copy them, do not translate them), domain terms (a glossary or fine-tuning on in-domain pairs helps), gender and honorifics when the source language does not mark them, hallucinated output for noisy or very short input, and bias, for example when a gender-neutral pronoun is translated into a gendered one by stereotype. Check these on your own data, as described in evaluation, bias and responsible NLP.

METEOR, limitations and uses of machine translation

METEOR is a metric that fixes two weaknesses of BLEU. BLEU is based on n-gram precision only, while METEOR combines unigram precision and recall in a weighted harmonic mean that favors recall, and it multiplies the result by a fragmentation penalty that punishes a candidate whose matched words are scattered in the wrong order. The full metric also matches stems and synonyms; the version below uses exact matches only to keep the mechanics visible.

Python A METEOR-like score with a fragmentation penalty

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The perfect candidate scores 0.998, not 1.0, because even one chunk gets a tiny penalty. The reordered candidate has all six words right but they form three separate chunks, so it drops to 0.711, which an unordered word count would not show. The short candidate has perfect precision but poor recall (3 of 6 reference words), giving 0.517, and the unrelated one scores zero. Automatic scores of this kind still only approximate human judgment, so important systems are also checked by people.

Limits of machine translation

  • Domain-specific errors: a model trained on news can mistranslate legal, medical or technical terms.
  • Cultural differences: idioms, humor, politeness levels and names often have no literal equivalent.
  • Low-resource languages: with little parallel text the quality drops sharply.
  • Fluent but wrong: neural systems can produce smooth output that omits or invents content, and bias in the training text can appear in the output (for example gendered pronouns).

So can human translators be replaced? For gist and high volume, machines are often enough; for contracts, literature or medical text, people still review and edit the output (post-editing).

Where machine translation is used

Machine translation is scalable, available 24/7 and cost-effective compared with human work, which explains its spread: business communication (support tickets, e-commerce listings, internal documents), travel (phone apps and live translation of signs and menus), education (access to material in other languages) and media (subtitles and news). The usual pattern is machine first, human check where the stakes are high.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. What does IBM Model 1 learn?

    Model 1 estimates t(f | e) from parallel text. It ignores word order and only models which words tend to translate each other.
  2. 2. Why did the Model 1 demo split "house" between "casa" and "la"?

    Words that always appear together share credit. Resolving them needs different contexts (in this corpus, "la" and "casa" never occur apart) or a model that sees word order.
  3. 3. Why is corpus-level BLEU preferred over averaging sentence BLEU?

    Corpus BLEU pools clipped matches and totals over all sentences before dividing, which makes the 4-gram precision far more stable.
  4. 4. In the decoding demo, why can a beam of 2 find a better translation than greedy search?

    Greedy search commits to the locally best word. A beam keeps several candidates, so a prefix that looked worse at step two can win once the whole sentence is scored.
  5. 5. What problem did attention solve in neural machine translation?

    With attention, the decoder looks at all the encoder states at each step, so no single vector must hold the whole sentence.

Summary

  • MT moved from rules to statistical models learned from parallel text, then to neural encoder-decoders with attention, Transformers and multilingual or prompted LLMs.
  • IBM Model 1 learns word translation probabilities with EM, without any word order, and cannot handle agreement such as gender; context-aware models can.
  • Corpus BLEU pools n-gram counts over the test set, supports several references, and needs smoothing for short texts; chrF and learned metrics add character-level and meaning-level views, and humans have the last word.
  • Decoding chooses one sentence from word probabilities; beam search keeps several candidates, usually with length normalization and coverage handling.
  • Start from pretrained models, evaluate on your own domain, and watch for entities, numbers, gender and hallucinations.

Next, see how language technology meets documents that arrive as images in OCR and Text from Documents.