Skip to content
Tutorials Center
My learning

Topics

Lesson 64 of 73 ยท Projects

Project: Text Generation with an LSTM

Train a character-level LSTM in NumPy to write text, track loss and perplexity, and compare greedy, temperature, top-k and top-p sampling.

A language model predicts the next piece of text from what came before. Train one on a lot of text, then let it predict, append its own prediction, and predict again, and it writes. This is how autocomplete works, and it is the same loop that powers large language models, only with a much bigger network, a much bigger corpus and a different architecture. In this project you build the oldest and simplest version: a character-level LSTM that reads text one letter at a time and learns to spell, to form words and to repeat the style of its training text.

You will implement the whole pipeline: turning text into integer sequences, a many-to-many LSTM with backpropagation through time, training with Adam and gradient clipping, and generation with a temperature control. The second half of the project examines sampling, the part of text generation that is not about the network at all but changes the output enormously.

From text to a training set

Characters to ids One integer per character of the vocabulary.
Windows Input is a chunk of text, target is the same chunk shifted by one.
Next character At every position, predict what comes next.
1 / 2

Step 1: the data

We use a small original text of nine sentences about a lighthouse keeper, 632 characters long. That is far too little for a language model, and we will see the consequences, but it trains in seconds. The vocabulary is the set of distinct characters (lowercase letters, a space and a period: 25 in total). Each character gets an integer id.

To train, we cut random windows of 16 characters from the text. The input is the window and the target is the same window shifted one position to the right. At every position the network must predict the character that follows. A window of 16 therefore provides 16 training examples at once, one per time step. This is the same many-to-many setup as in Sequence Data, and the network is the LSTM from LSTM and GRU: its output at every step goes through a softmax layer over the vocabulary, and the loss is the cross-entropy averaged over all positions.

Step 2: the LSTM with a per-step output

The forward pass loops over time and stores what the backward pass needs. The backward pass walks the loop in reverse, adding the gradient of each step's output to the gradient flowing back from the future (BPTT). Because the input is a character id, the first matrix product is a row lookup, Wx[ids], and its gradient is a one-hot matrix product that adds each row's gradient to the matching character. We initialize the forget-gate bias to 1, clip the gradient norm to 5 (see Backpropagation Through Time) and use Adam.

During training we report the loss in nats per character and its exponential, the perplexity: roughly the number of equally likely characters the model is torn between at each step. A model that knows nothing has a perplexity equal to the vocabulary size (25 here). A model that only knows letter frequencies has a perplexity of 16.0 on this text. A good model gets close to 1.

Python Training a character-level LSTM and generating text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls from 3.22 nats (perplexity 25.0, the uniform guess) to 0.29 (perplexity 1.3) in 500 steps, far below the 2.77 nats of the letter-frequency model, which means the network has learned which letters follow which. The numbers are for one random batch, so they wobble a little from step to step.

Read the generated text. At temperature 0.3 the model writes whole, correct sentences from the corpus: the keeper held the lamp steady and the harbor. in the morning the sea was quiet.... It stitches fragments together and may loop. At 0.8 the sentences begin to mix: the keeper held the beam crossed the long stairs to the water, locally fluent but not always sensible. At 1.5 it spells words approximately and then collapses: the wicrsmousted the hark waves and waites in crosnight. You are seeing a model that has memorized a tiny corpus (nine sentences leave very little to generalize from) and a dial that controls how far it wanders from what it memorized. With a corpus of megabytes instead of 632 characters the same code produces new sentences rather than recombined old ones.

Step 3: what sampling does

The network's output at each step is a list of logits, one per character. To pick the next character you convert them to probabilities with a softmax and then choose. How you choose is a decision outside the network:

  • Greedy: always take the most probable character. Deterministic, safe, and prone to repeating itself forever.
  • Temperature sampling: divide the logits by T before the softmax. T < 1 sharpens the distribution toward the top choice, T > 1 flattens it, and T = 1 samples from the model's own beliefs.
  • Top-k: keep only the k most probable characters and renormalize, which removes the long tail of nonsense options.
  • Top-p (nucleus): keep the smallest set of characters whose probabilities add up to p, so the number of choices adapts to how confident the model is.
Python Temperature, top-k and top-p on one prediction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first table is the whole story of temperature. At T = 0.2 the favorite (e) holds 99.9% of the probability and the entropy is 0.01 bits: effectively greedy. At T = 1.0 it holds 66.9% and the model is as uncertain as it says it is (1.57 bits). At T = 3.0 the distribution is nearly flat (2.77 bits, close to the maximum of 3 for eight options). The last table shows the consequence for generated text: with 3,000 draws, T = 0.2 produces only 3 distinct pairs of consecutive characters (it repeats), T = 1.0 produces 48 of the 64 possible pairs, and T = 3.0 produces every pair. Nothing in the network changed; only the way we choose.

Top-k and top-p both cut the tail: at T = 1 they leave only e, o and i and lower the entropy from 1.57 to 1.09 bits. Top-p is usually preferred because it keeps more options when the model is unsure and fewer when it is sure. Most text-generation systems combine a moderate temperature (0.7 to 1.0) with top-p (around 0.9). This is the same toolbox you will meet again in How Large Language Models Work.

Step 4: the real recipe

On a real corpus (a book, a collection of poems, source code) the data is larger and the model is trained in PyTorch with nn.LSTM. The loop is the same as yours. This code is read-only.

Python
# Read-only: character-level language model in PyTorch
import torch, torch.nn as nn

text = open("corpus.txt", encoding="utf-8").read()
chars = sorted(set(text)); stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])

class CharLSTM(nn.Module):
    def __init__(self, vocab, embed=64, hidden=256, layers=2):
        super().__init__()
        self.embed = nn.Embedding(vocab, embed)
        self.lstm = nn.LSTM(embed, hidden, layers, batch_first=True, dropout=0.2)
        self.out = nn.Linear(hidden, vocab)
    def forward(self, x, state=None):
        h, state = self.lstm(self.embed(x), state)
        return self.out(h), state                        # logits at every position, plus the state to continue from

model = CharLSTM(len(chars))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(5000):
    starts = torch.randint(0, len(data) - 129, (64,))
    x = torch.stack([data[s:s + 128] for s in starts]); y = torch.stack([data[s + 1:s + 129] for s in starts])
    logits, _ = model(x)
    loss = nn.functional.cross_entropy(logits.reshape(-1, len(chars)), y.reshape(-1))
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()

@torch.no_grad()
def generate(prompt, n=300, temperature=0.8):
    x = torch.tensor([[stoi[c] for c in prompt]]); out, state = prompt, None
    for _ in range(n):
        logits, state = model(x, state)
        p = torch.softmax(logits[0, -1] / temperature, dim=-1)
        nxt = torch.multinomial(p, 1)
        out += chars[nxt.item()]; x = nxt.view(1, 1)      # feed back only the new character; the state carries the memory
    return out

Notice the two changes compared with the NumPy version: an embedding layer replaces the one-hot lookup (it is the same row lookup, with a smaller learned vector), and generation passes the state forward, so the network does not re-read the whole text at each step.

Ideas to extend the project

  • Use a bigger corpus. A few hundred kilobytes of plain text is enough for a char-level model to produce plausible words. Compare the loss on a held-out part of the text to see overfitting.
  • Go word-level or subword-level. Predicting words (or the subword tokens from Hugging Face Transformers Pipelines) gives longer-range coherence and a shorter sequence, at the cost of a bigger vocabulary.
  • Swap the LSTM for a transformer. The Tiny Language Model lesson does this with attention.
  • Add a prompt-conditioned task, such as generating text in the style of one of several authors by feeding the author as an extra input.
  • Evaluate samples, not only loss: measure how often the model repeats itself, or how many of its n-grams appear verbatim in the training data (a memorization check).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the training windows, what is the target at each position?

    The target sequence is the input sequence shifted by one position, so the model learns to predict what comes next at every step.
  2. 2. What does a perplexity of about 1.3 mean for this model?

    Perplexity is the exponential of the average cross-entropy and can be read as the effective number of choices the model considers at each step.
  3. 3. What happens to the output as the sampling temperature is raised from 0.3 to 1.5?

    A higher temperature flattens the distribution, so unlikely characters get picked more often: more variety, more mistakes.
  4. 4. How does top-p (nucleus) sampling differ from top-k?

    Top-p adapts to the model's confidence: few options when the distribution is sharp, more when it is flat.
  5. 5. Why does gradient clipping matter when training an LSTM with BPTT?

    Backpropagating through many time steps can occasionally produce very large gradients. Clipping their norm keeps updates bounded.

Summary

  • A character-level language model trains on windows of text with the target shifted by one position, giving one prediction per time step.
  • A many-to-many LSTM with BPTT, Adam and gradient clipping reduced the loss from 3.22 to 0.29 nats per character on a 632-character corpus (perplexity 25.0 to 1.3).
  • Generation repeatedly samples from the predicted distribution and feeds the sample back in.
  • Temperature divides the logits: low values repeat and play safe, high values vary and make mistakes. Top-k and top-p remove the unlikely tail.
  • A tiny corpus is memorized, so output at low temperature is recombined training text; real generalization needs much more data.
  • The PyTorch version replaces the one-hot lookup with an embedding layer and carries the LSTM state across generation steps.

Next project: Face and Emotion Detection.