Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 45 ยท Projects

Project: A Character-Level Language Model

Build a character-level language model: count a bigram table, train the same model with gradients, add context, and sample text with temperature.

A language model assigns a probability to the next token given what came before. Everything from the autocomplete on your phone to the largest chatbots is built on that one idea. In this project you build the smallest honest version: a model that reads a list of names one character at a time and learns to invent new, name-like words. Because the vocabulary is only the alphabet, the whole thing is small enough to understand completely, yet it contains the three ideas every large language model uses: a loss that is the negative log-likelihood of the next token, a softmax over the vocabulary, and sampling with a temperature.

PyTorch cannot run on this site, so the PyTorch models here (a bigram and a small MLP with context) are read-only. The live editors use NumPy: a count-based bigram model with a sampler, the same model trained by gradient descent (which shows that training and counting arrive at the same answer), and an experiment with temperature and context length. You should know Activation and Loss Functions (softmax and cross-entropy) before starting.

The language-model recipe

Characters The alphabet plus a boundary mark.
Next-char task Given the context, predict what comes next.
Softmax A probability for every character.
Sampling Draw, append, repeat until the end mark.
1 / 2

The task

We use 79 invented-looking first names. Each name is wrapped in a boundary character .: .amira. The dot at the start means "a name begins" and the dot at the end means "the name is over", so the model learns when to stop. The training examples are the adjacent pairs: after . comes a, after a comes m, after m comes i, and so on, ending with a followed by .. For 79 names that gives 511 pairs over an alphabet of 26 symbols (25 letters plus the dot).

The bigram model by counting

The simplest model looks only at the current character. Count how often each character follows each other one, and turn every row of counts into probabilities. That table is the model. We use a little smoothing (adding a half to every count), so that a pair never seen in training still has a small non-zero probability rather than zero; a zero would make the loss infinite if that pair ever appeared.

The quality measure is the average negative log-likelihood per character: the mean of -ln P(correct next character). Random guessing among 26 symbols costs ln 26 = 3.26. Lower is better, and this is exactly the cross-entropy loss you use for classification, applied to each position of the text. Sampling is the reverse process: start at the boundary, draw the next character from the current row, and stop when the dot comes up.

Python A bigram model by counting, and a name sampler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model learned real structure. After a, the most likely next symbol is the end of the name (a probability of 0.34): many names in the list end in a. Names start with an l, a, t or m with similar likelihood. The loss is 2.30, well below the 3.26 of random guessing. The samples, though, look like what a model that remembers only one letter of context can produce: kian and coritcucypmyspa are almost plausible, ilenmxdnid is not. The model has no idea how long a name should be, nor which letters belong together over more than one step. We will fix a little of that below.

The same model by gradient descent

Counting is a special trick that works only for tiny models. The general method, which scales to billions of parameters, is to define the model as a function with parameters and adjust them with gradient descent. A bigram model written that way is a table W of logits (raw scores) with one row per current character. To predict, look up the row of the current character (this is exactly an embedding lookup) and apply the softmax. Initially W is zero, so every character is equally likely and the loss is ln 26.

The gradient of the average cross-entropy with respect to the logits is the familiar probabilities - one_hot, divided by the number of examples, and it is scattered back into the rows that were looked up. A theorem about maximum likelihood says that the best table is the one made of the raw, unsmoothed relative frequencies, so we know where training should end: at the loss of the plain count table.

Python Gradient descent finds the count table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unsmoothed count table has a loss of 2.042 and is the best any bigram model can do on this data. Gradient descent starts at 3.258 (ln 26), drops to 2.40 after 10 steps, and creeps toward the optimum: 2.13 at step 50, 2.08 at step 100, 2.05 at step 300. The learned probabilities differ from the counted ones by at most 0.03. Notice the shape of the curve: fast at first, then slower, which is typical. The heat map on the right is the learned table; the bright cells are the likely transitions.

Training and counting agree, which is the key insight of this project: a language model trained with cross-entropy is a machine for learning conditional probabilities. The count table is simply the closed-form answer for the simplest possible model. Once the model has hidden layers, there is no closed form, and gradient descent is the only way.

The PyTorch version

In PyTorch the bigram model is a single nn.Embedding whose rows are the logits. This code is read-only:

Python
import torch
import torch.nn.functional as F

chars = sorted(set("".join(names)))
stoi = {c: i + 1 for i, c in enumerate(chars)}
stoi["."] = 0
itos = {i: c for c, i in stoi.items()}
V = len(stoi)

xs, ys = [], []
for name in names:
    seq = "." + name + "."
    for a, b in zip(seq, seq[1:]):
        xs.append(stoi[a]); ys.append(stoi[b])
xs, ys = torch.tensor(xs), torch.tensor(ys)

torch.manual_seed(0)
W = torch.zeros(V, V, requires_grad=True)          # the whole model: a table of logits

for step in range(300):
    loss = F.cross_entropy(W[xs], ys)              # lookup rows, softmax, negative log-likelihood
    W.grad = None
    loss.backward()
    with torch.no_grad():
        W -= 30.0 * W.grad

F.cross_entropy fuses the softmax and the negative log-likelihood, and autograd produces the gradient that we scattered back by hand. Sampling uses torch.multinomial:

Python
@torch.no_grad()
def sample(temperature=1.0):
    out, cur = [], 0
    while True:
        probs = F.softmax(W[cur] / temperature, dim=0)
        cur = torch.multinomial(probs, num_samples=1).item()
        if cur == 0:
            return "".join(out)
        out.append(itos[cur])

More context: an MLP

A model that sees only one character cannot know that jb is not how a name starts. The standard neural upgrade looks at the previous k characters: embed each one (shared table), concatenate the embeddings, pass them through a hidden layer and produce logits for the next character.

Python
class CharMLP(torch.nn.Module):
    def __init__(self, V, context=3, dim=10, hidden=128):
        super().__init__()
        self.context = context
        self.emb = torch.nn.Embedding(V, dim)
        self.net = torch.nn.Sequential(
            torch.nn.Linear(context * dim, hidden), torch.nn.Tanh(),
            torch.nn.Linear(hidden, V),
        )

    def forward(self, x):                      # x: (batch, context) of character ids
        e = self.emb(x)                        # (batch, context, dim)
        return self.net(e.flatten(1))          # (batch, V) logits for the next character

Training pairs now are (three previous characters) -> next character, built by sliding a window over each padded name. Replace the fixed window by self-attention over all previous characters (with a causal mask so that position t cannot look ahead) and you have the core of a GPT-style transformer (see Building a Transformer Block). The training loop, the loss and the sampling code do not change at all, and that continuity is why this tiny project is such a good map of the big ones.

Temperature and context length

Two experiments in the next editor. First, temperature: before the softmax, the logits are divided by T. At T = 1 we sample from the model's own distribution; at T < 1 the distribution sharpens toward the most likely choice; at T > 1 it flattens toward uniform. The editor prints the next-letter distribution after a and eight sampled names at each temperature.

Second, a context experiment that mirrors the MLP idea with counting: a bigram table (context of one character) versus a table that conditions on two characters. We train on the first 63 names and measure the loss on the 16 names the model has never seen. This is the important discipline for language models too: judge them on held-out text, not on the data they were trained on.

Python Temperature sampling and context length

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the temperature rows first. After a, the model's own distribution (T = 1) gives the end-of-name mark 40% and spreads the rest over many letters (entropy 2.06 nats). At T = 0.3 the end mark takes 94%, which is why the low-temperature samples are tiny (ta, a, l): a bigram model that always picks the likeliest move stops early. At T = 3 the distribution is nearly flat (entropy 3.07), and the samples are letter soup such as wathyhxmcsppjf. In real language models the same dial trades off safe, repetitive text against creative, error-prone text; most applications use a value around 0.7 to 1.

The context experiment is a lesson in overfitting. With two characters of context, the loss on the training names drops from 2.074 to 1.605: the model looks much better. But on the 16 held-out names, the loss gets worse, from 2.662 to 2.897. With only 63 names, the larger table (147 contexts instead of 26) memorizes the particular names it saw and has too few examples per context to generalize. A longer context is only a gain when you have enough data, or when the model shares statistical strength between contexts, which is exactly what embeddings and neural networks do and what a lookup table cannot. That is the reason the PyTorch MLP generalizes better than a trigram count table at the same context length, and why you would add regularization and evaluate on held-out data in real training (see Overfitting and Regularization).

Extend the project

  • In the NumPy gradient-descent model, plot the loss on held-out names during training and find the step where it stops improving.
  • Implement top-k sampling: before drawing, zero out everything except the k most likely characters and renormalize.
  • Measure the typical name length produced at T = 1 and compare it with the training names.
  • In PyTorch, train CharMLP with a context of 3 and a train/validation split, and compare its validation loss with the bigram table's. Then replace the MLP with one causal self-attention layer.
  • Try the model on a different alphabet: a list of city names, chemical terms or file extensions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a random-guess language model over an alphabet of 26 symbols score as cross-entropy loss per character?

    A uniform prediction gives each correct character probability 1/26, so the loss is -ln(1/26) = ln 26. Any trained model should score lower.
  2. 2. What does the temperature do during sampling?

    Dividing logits by T < 1 exaggerates differences between characters (more predictable output); T > 1 shrinks them (more random output).
  3. 3. Why did the two-character-context model look better on training names but worse on held-out names?

    More contexts mean fewer examples each. The model fits the training names closely but predicts unseen ones poorly, which is overfitting.
  4. 4. Why does gradient descent on a table of logits end near the plain count table?

    Maximum likelihood for a categorical distribution is the relative frequency. Cross-entropy training minimizes the negative log-likelihood, so it converges toward those frequencies.

Summary

  • A character-level language model predicts the next character from the context and is trained with cross-entropy, the negative log-likelihood per character.
  • A bigram model can be made by counting; the same model trained with gradient descent as a table of logits converges to the same probabilities.
  • Sampling means: draw from the softmax, append, repeat until the end mark. Temperature divides the logits and controls randomness.
  • A longer context needs more data or shared parameters (embeddings and hidden layers); a plain lookup table overfits, which only held-out loss reveals.
  • In PyTorch the same project is nn.Embedding or a small MLP with F.cross_entropy and torch.multinomial; replacing the MLP with causal self-attention leads toward GPT-style models.

Continue with Project: Forecasting a Time Series.