Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 73 ยท Sequences, Attention and Transformers

Training a Tiny Language Model

Build and train a character-level, decoder-only transformer in NumPy with a full backward pass, then sample text with temperature and top-k.

A language model has one job: given some text, assign a probability to every possible next piece of text. Everything else, from autocomplete to chatbots, is built on that single skill. In the last lesson you assembled a transformer block. Now you will put it to work: a complete, trainable, decoder-only language model that fits in about 80 lines of NumPy, including its hand-written backward pass.

The model is tiny (about 3,000 parameters) and its training text is a toy corpus, so do not expect poetry. What you get instead is every real ingredient of a GPT-style model in a form small enough to read, run in your browser in seconds, and verify with gradient checking.

The whole pipeline

Tokenize Turn text into integer ids.
Embed Look up a vector for each id, plus its position.
Transform Attention and MLP mix and refine the vectors.
Predict Output a probability for every next character.
1 / 2

Step 1: a character-level tokenizer

Neural networks work with numbers, so text must become integers first. Real models split text into sub-word pieces (you will build such a tokenizer in How Large Language Models Work). The simplest scheme uses one token per character. The vocabulary is just the set of distinct characters in the corpus, and the id of a character is its position in a sorted list.

Our corpus is generated from a small template: a sentence about an animal that goes somewhere, repeated 120 times with random choices. That gives the model real structure to find (spelling, spaces, the shape of a sentence) and real randomness it can never predict (which animal comes next).

Training examples come from sliding a window over the data. The target is the input shifted by one character: for every position, the correct answer is simply the character that follows. Look at how one window yields several examples.

Python Tokenizer and next-character examples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 22 characters include the space and the period; underscores in the printout stand for spaces. A window of 8 characters produces 8 training examples at once: from t predict h, from th predict e, and so on. Because of the causal mask you met in the previous lesson, a single forward pass computes all 8 predictions together. A batch is these windows stacked, so X and Y both have shape (batch, time).

Step 2: the model

The model has five parts, in this order:

Stage Shape What happens
Token ids (B, T) Integers from the tokenizer
Embedding + position (B, T, d) E[ids] + P[:T]: a learned vector per character plus a learned vector per position
Transformer block (B, T, d) Pre-norm layout: causal self-attention, then a ReLU MLP, each with a residual connection
Final layer norm (B, T, d) Steadies the scale before the output layer
Output layer (B, T, V) One score (logit) per vocabulary entry at every position

Without position information, attention would see only a bag of characters, so we add a learned positional embedding (see Self-Attention and Multi-Head Attention). To keep the hand-derived gradients short, this model uses a single attention head and a ReLU instead of GELU; the structure is otherwise the block from the last lesson. The loss is the average cross-entropy over all B x T positions (see Loss Functions).

The backward method applies the chain rule in reverse through each stage. Its trickiest parts are the layer-norm gradient and the softmax inside attention, dS = A * (dA - sum(dA * A)). Instead of trusting the algebra, the example below checks every parameter against finite differences on a miniature model. A relative error around 1e-8 or smaller means the backward pass is right.

Python The model, with a full gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At initialization the loss is 2.534 against ln(11) = 2.398 for pure guessing: a model that knows nothing should be near ln(V), and a large gap would hint at a bug. All twelve parameter groups have relative errors of order 1e-8 or 1e-10, so the gradients are correct, including the gradient flowing back into the token and position embeddings.

Step 3: training

Training is the loop you already know: sample a batch of windows, run forward, run backward, update with Adam and a cosine learning-rate schedule. The numbers are chosen to respect the browser's 30-second limit.

The in-browser NumPy has no fast BLAS library, so even small matrix multiplications are slow. This run uses d = 16, a context of 16 characters, batches of 16 and 300 steps, which takes roughly 10 to 15 seconds. Doubling the width or the batch size would push you past the limit. Real training runs use GPUs, millions of times more parameters, and weeks of compute.

Watch the loss fall from about 3.5 (worse than guessing) toward 0.3, and then read what the model writes.

Python Train the tiny model and sample from it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts at 3.497, above the random-guess level of ln(22) = 3.091, and falls to about 0.3 after 300 steps. For this corpus the lowest possible value is not zero: the choice of animal, verb and place is random, so the model can never predict those letters perfectly. It also has seen the same few thousand characters over and over, so part of the improvement is memorization. A tiny model on a toy text is a demonstration of the mechanism, not of generalization.

Step 4: sampling, temperature and top-k

To generate text, feed the model a prompt, look at the last position's logits, turn them into probabilities with a softmax, pick one character, append it, and repeat. Two knobs control how adventurous the choice is.

  • Temperature divides the logits before the softmax. Below 1, the distribution sharpens and the model plays safe; at 1 it uses its true probabilities; above 1 it flattens, so unlikely characters appear more often.
  • Top-k discards everything except the k most likely characters before sampling, which removes the long tail of nonsense choices.

Look at the samples printed by the previous example. At temperature 0.3 the model writes clean sentences, here the old bear sleeps by the lake. repeated with small changes: reliable but repetitive. At 1.0 it follows its own probabilities and starts to garble words (tsleefrog), because an undertrained model is genuinely unsure. At 1.5 the text degrades further. Top-k 3 at temperature 1 trims the worst choices but, with a model this small, cannot repair its confusion. A better-trained model would sound far more fluent at temperature 1 and would simply be more creative as the temperature rises.

The next example isolates the two knobs on a fixed set of logits so you can see what they do to the probabilities.

Python What temperature and top-k do to probabilities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At temperature 0.5 the favorite takes 79% of the probability; at 2.0 it falls to 34% and the tail grows. Top-k 2 removes all but the two best tokens and renormalizes them to 69% and 31%. With 1,000 draws, the low-temperature distribution produces five distinct tokens and the high-temperature one produces all six.

What a tiny model teaches you about big ones

Everything here scales. A real language model changes four things, not the recipe:

  • Tokens are sub-word pieces instead of characters, so a sentence is far shorter in tokens.
  • Depth and width grow: dozens of blocks, hundreds or thousands of dimensions, many attention heads per block.
  • Data grows from a few thousand characters to trillions of tokens, which is what makes the model generalize rather than memorize.
  • Hardware and code change: GPUs, mixed precision, and automatic differentiation instead of a hand-written backward (see Automatic Differentiation From Scratch).

The limits you hit here are also the real ones in miniature: a context window of 16 characters means the model cannot remember anything older, and it can only reproduce patterns that its data contains.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In next-character training, what is the target for each position in the input window?

    The model learns to predict the next character, so the target sequence is the input sequence moved one step to the left.
  2. 2. A freshly initialized model for a vocabulary of V tokens should have a loss close to which value?

    If every token has probability 1/V, the cross-entropy is ln(V). A much larger or smaller starting loss suggests a bug or a bad initialization.
  3. 3. What happens when you sample with a temperature below 1?

    Dividing the logits by a number smaller than 1 makes the differences between them larger, so the softmax concentrates probability on the top choices.
  4. 4. Why do we run a finite-difference gradient check on a hand-written backward pass?

    Finite differences estimate the gradient from the loss alone, with no calculus. If the analytic and numerical values match to a small relative error, the backward pass is almost certainly right.
  5. 5. What does top-k sampling do?

    Top-k removes the long tail of unlikely tokens before sampling. Picking only the single most likely token every time would be greedy decoding, which is the case k = 1.

Summary

  • A language model predicts the next token; training windows are the text and the same text shifted by one position.
  • A character-level tokenizer maps each distinct character to an integer id; real models use sub-word tokens.
  • The decoder-only model is embeddings plus positions, a pre-norm transformer block with causal attention, a final layer norm and an output layer of logits.
  • The backward pass can be written by hand and verified with finite differences; a fresh model starts near ln(V).
  • Generation repeats: take the last position's logits, apply temperature and optionally top-k, sample, append.
  • Real models use the same recipe with sub-word tokens, far more data, depth and width, and GPUs.

Next: How Large Language Models Work.