Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

Next-Token Prediction

Build n-gram language models in Python, measure them with probability and perplexity, and see why context length and data sparsity led to neural LLMs.

Every LLM answers one question over and over: "given these tokens, how likely is each possible next token?" Before neural networks, researchers answered it with simple counting models called n-grams. They are worth building because they make the core vocabulary concrete: context, probability, likelihood, perplexity, and sparsity. Each of those returns later when we train a real model.

This lesson uses characters as tokens so the examples stay small. The ideas are identical for subword tokens, which we cover in Tokenization Basics.

Context and the n-gram idea

An n-gram model predicts the next token using only the previous n-1 tokens. A bigram looks one token back, a trigram two, and so on. The model is a table: for each context, how often each next token followed it. Dividing by the total gives probabilities.

Python Character n-gram model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A longer context sharpens the prediction. After a lone h the model is unsure, but after th it knows the vowel or r or i that follows in this text. More context means more information, which is the whole reason Transformers were designed to look at thousands of tokens.

Scoring text: likelihood and perplexity

A language model can score any text by multiplying the probability it gave each actual token. Because products of small numbers underflow, we add log probabilities instead. The average negative log probability per token is the cross-entropy, and its exponential is perplexity: roughly "how many equally likely choices the model felt it had at each step". Lower is better.

Python Perplexity of a bigram model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The alpha term is smoothing: it gives every unseen pair a small probability so the log never hits zero. Without it, one unseen pair makes the whole text impossible (infinite perplexity). The line printing len(vocab) shows a useful reference point: a model that guesses uniformly over V tokens has perplexity exactly V. Anything lower means the model learned something.

The wall: sparsity and the curse of context

Counting works until the context grows. With a vocabulary of V tokens, there are V^(n-1) possible contexts. For words with V = 50,000 and a three-word context that is more than 10^14 possible contexts, and almost none appear even once in any training set. The model has no data to estimate them from. Smoothing and back-off (falling back to shorter contexts) help, but they cannot create information that is not there.

Python How fast contexts become unseen

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even in this tiny world the fraction of unseen test contexts explodes as n grows. Real languages are far richer.

What neural networks add

A neural language model replaces the lookup table with a function. Instead of storing a separate count for every context, it maps a context to a vector and computes probabilities from that vector. Similar contexts get similar vectors, so what the model learned about "the cat sat" helps it with "a dog sat". This generalization is the breakthrough: the model shares statistical strength across contexts it has never seen.

Counting versus learning

N-gram table One entry per context. No sharing.
Neural network One function for all contexts.
1 / 2

The autoregressive view

The factorization P(t1...tn) = product of P(ti | t1...ti-1) is the same whether the conditional comes from a table or a Transformer. Models that generate left to right using it are called autoregressive. The difference between an n-gram model and GPT-style models is only how the conditional is computed and how much context it can use: five tokens versus hundreds of thousands.

Two practical consequences follow. First, generation is sequential: a token cannot be produced before the ones that precede it, which makes inference slow in a specific way we analyze in the inference chapter. Second, training can be parallel: since the real text is known, the model can predict every position at once and be scored on all of them in one pass.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a trigram model use to predict the next token?

    An n-gram model looks back n-1 tokens, so a trigram uses two.
  2. 2. A model guesses uniformly among 30 tokens. What is its perplexity?

    Uniform guessing over V tokens has cross-entropy log V, so perplexity is V = 30.
  3. 3. Why is smoothing needed in a count-based model?

    A single zero makes the whole text impossible. Smoothing reserves a little probability for unseen events.
  4. 4. Why do neural language models handle unseen contexts better than count tables?

    A neural model computes predictions from learned vectors, so similar contexts get similar predictions.

Summary

  • A language model estimates the probability of the next token given a context.
  • N-gram models count; longer context helps but makes most contexts unseen.
  • Perplexity is the exponential of the average negative log probability; lower is better, and uniform guessing over V tokens gives V.
  • Smoothing prevents zero probabilities.
  • Neural networks generalize across similar contexts, which is why they replaced count tables.

Next: A Short History of LLMs