Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 63 ยท Language Models Before Neural Nets

Smoothing and Perplexity

Fix zero probabilities with add-one smoothing, interpolation and backoff, then measure language models with perplexity on held-out text in Python.

The n-gram model from the last lesson has a fatal flaw. If a sentence contains a single word pair the model never saw during training, its probability is exactly zero, no matter how reasonable the rest of the sentence is. Training text is always a small sample of a language, so unseen but perfectly valid pairs are the rule, not the exception.

Smoothing moves a little probability from the events we have seen to the events we have not. This lesson covers the three classic recipes (add-one, backoff and interpolation) and then introduces perplexity, the standard score for deciding which language model is better.

Why zero is a problem

Imagine the model never saw "mouse chased". The maximum likelihood estimate says P(chased | mouse) = 0, so any sentence containing those two words gets probability 0. When you compare models, one zero makes the whole test set impossible, and when you use the model inside a speech recognizer, the correct sentence is rejected for good. Zero should mean "I have no evidence", not "this can never happen".

Three ways to handle unseen pairs

Add-one Pretend every pair was seen once more.
Backoff Fall back to a shorter context.
Interpolation Always mix long and short contexts.

All three keep every probability above zero.

1 / 2

Add-one (Laplace) smoothing

The simplest fix is to add one to every count, including the counts of pairs that never occurred. To keep the probabilities summing to 1, add the vocabulary size V to the denominator:

TEXT
P(word | prev) = (count(prev word) + 1) / (count(prev) + V)

A more general form adds a small number k instead of 1, so (count + k) / (count(prev) + k * V). The example uses the five-sentence corpus from the previous lesson and prints both estimates side by side.

Python Add-one smoothing next to the raw estimate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unseen pair mat sat now gets 0.083 instead of 0, and P(cat | the) drops from 0.333 to 0.200. That drop is the cost: with a vocabulary of 11 words and only nine observations after "the", add-one hands a large share of the probability to events that never happened. The last lines show that a smaller k (0.1 or 0.01) is much gentler and stays close to the raw estimate. Add-one is rarely the best choice for n-gram language models, but it is the simplest, and it is useful for Naive Bayes, where the counts behave differently.

Backoff and interpolation

Add-one treats all unseen pairs alike: it does not know that mouse chased is more plausible than mouse rug. A better idea is to use shorter contexts as evidence. We may never have seen "mouse chased", but we have seen "chased" many times, and that unigram tells us something.

  • Backoff uses the bigram estimate when the bigram was seen, and otherwise falls back to the unigram (scaled down). A very simple version called stupid backoff multiplies the unigram probability by a constant such as 0.4. It does not form a true probability distribution, but it works for ranking, and it is cheap on huge corpora.
  • Interpolation always mixes the two estimates with weights that add up to 1:
TEXT
P(word | prev) = lambda * P_bigram(word | prev) + (1 - lambda) * P_unigram(word)

With lambda = 0.7, the bigram counts get 70% of the vote, and the unigram picks up the rest. Unseen pairs now get some probability from the unigram term. In practice you pick lambda by testing values on held-out data, never on the training data.

flowchart TD
    A["Need P(word | prev)"] --> B{"Was the pair<br/>seen in training?"}
    B -- "yes" --> C["Backoff: use the bigram estimate"]
    B -- "no" --> D["Backoff: use 0.4 x unigram estimate"]
    A --> E["Interpolation: always blend<br/>lambda x bigram + (1 - lambda) x unigram"]

Backoff chooses one source per event; interpolation always blends both.

Perplexity

How do we say that one language model is better than another? Train on one set of sentences and measure how surprised the model is by a held-out set it never saw. The measure is perplexity, the inverse probability of the test text, normalized by its length:

TEXT
perplexity = 2 ^ ( - (1 / N) * sum of log2 P(word_i | context_i) )

where N is the number of predicted words. Lower is better. A helpful way to read it is as an average branching factor: a model with perplexity 10 is as uncertain as if it had to pick between 10 equally likely words at every step. A model that assigns probability 1 to every word has perplexity 1; a model that assigns probability zero to anything makes perplexity infinite.

Three rules keep the comparison honest:

  • Test on unseen text. Perplexity on the training data flatters the model; an unsmoothed bigram looks excellent on the text it memorized.
  • Use the same vocabulary and the same tokenization. Perplexity values are only comparable if the models predict the same units.
  • Treat unknown words explicitly. Real systems replace rare training words by a special <unk> token so that the model can score words it has not seen.

The next example trains on 18 short sentences and measures five test sentences. One test sentence ("the mouse chased the dog") contains the pair mouse chased, which never appears in training.

Python Perplexity of unigram, bigram, smoothed and interpolated models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from the top. The unigram model (no context) has perplexity 11.11. The raw bigram model is infinite on the test set because of the single unseen pair. Add-one fixes that but gives away so much mass that it scores 6.01, while add-0.1 reaches 3.56. Interpolation improves as lambda grows from 0.3 (5.93) to 0.7 (3.90) and 0.9 (3.44), and stupid backoff, which trusts a seen bigram completely, reaches 3.08. The pattern to remember: a little context helps a lot, an unseen event must never be allowed to reach zero, and the amount of smoothing must be tuned on held-out data. Setting lambda = 1.0 removes the unigram term and the model is back to infinity. Finally, the last line shows why training perplexity is misleading: the raw bigram scores 2.52 on text it has already seen.

Perplexity as a branching factor

To build the intuition, compute the perplexity of a model that always picks among k equally likely words. Each word gets probability 1/k, so the perplexity should come out as exactly k.

Python A uniform model has perplexity k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first three lines print 2.0, 10.0 and 50.0. The last model is confident about eight words (probability 0.9) and unsure about two (probability 0.1), giving a perplexity of 1.72: low, because perplexity averages the log probabilities and the confident words dominate. Remember too that perplexity is not an end goal. A lower value means better predictions of text, but a system that translates or answers questions still needs to be tested on that task itself.

Compare perplexities only between models on the same test set, tokenization and vocabulary. A character-level model and a word-level model can have wildly different perplexities on the same text without either being "better".

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main purpose of smoothing in an n-gram language model?

    Smoothing moves a bit of probability mass from seen events to unseen ones, so one missing n-gram does not set the probability of a whole sentence to zero.
  2. 2. In add-one smoothing, why is the vocabulary size V added to the denominator?

    Adding 1 to the count of each of the V possible next words adds V to the total, so the denominator must grow by V for the distribution to still sum to 1.
  3. 3. How do backoff and interpolation differ?

    Backoff picks one source per event, the longer context if it was seen and the shorter one otherwise. Interpolation always combines the estimates with weights that sum to 1.
  4. 4. A language model has perplexity 20 on a test set. How can you read that?

    Perplexity is the exponential of the average negative log probability, which acts like an average branching factor. Lower is better.
  5. 5. Why should you not tune lambda or compare models using the training text?

    A model fitted to its training text scores unrealistically well on it. Held-out text shows how it behaves on sentences it has never seen, which is what matters in use.

Summary

  • Unsmoothed n-gram models give probability zero to unseen n-grams, which can zero out a whole sentence.
  • Add-one (or add-k) smoothing adds a pseudo-count to every pair; it is simple but can take too much mass from seen events.
  • Backoff falls back to shorter contexts when the longer one is unseen; interpolation always blends several orders with weights that sum to 1.
  • Perplexity is the exponentiated average negative log probability on held-out text: lower is better, and it reads as a branching factor.
  • Tune smoothing settings on held-out data, and compare models only on the same test set and vocabulary.

Next, use the same counting ideas to fix typos in Spelling Correction.