Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 53 ยท Projects

Project: A Tiny Character-Level Language Model

Train a character-level language model from scratch in NumPy: a bigram baseline, a small neural network, a loss curve, overfitting and sampling.

Every large language model, from the smallest open-weight model to the biggest chatbot, is trained with the same loop: show the model some text, ask it to predict the next token, measure how wrong it was, and nudge the numbers inside it to be a little less wrong. In this project you run that exact loop yourself, in about 100 lines of NumPy, on a text small enough to train in seconds.

The model is character-level: its tokens are single characters, so there is no tokenizer to build and the vocabulary has about 25 entries. It is not a transformer, and it will not write poetry. But it has the same ingredients as a real model (an embedding table, a neural network, a softmax over the vocabulary, cross-entropy loss, gradient descent, sampling with temperature), and it shows one thing real training also has to deal with: overfitting.

The training recipe

Data Turn text into numbers.
Predict Probabilities for the next character.
Loss How surprised was the model?
Update Nudge weights downhill.

Repeat thousands of times.

1 / 2

Step 1: data and a counting baseline

We use a short original text of about 330 characters. The code builds a vocabulary (every distinct character, sorted), two lookup tables (stoi: character to number, itos: number to character) and an encode / decode pair. After that the whole text is just an array of small integers. If this feels like tokenization, it is: the simplest possible tokenizer, one character per token.

Before training anything, we build a baseline. Counting how often each character follows each other character gives a table of probabilities, exactly like the word-level model in A Language Model in 20 Lines. We measure it with cross-entropy loss: the average of -log(p) where p is the probability the model gave to the character that actually came next. A model with no knowledge at all gives every character the same probability 1/V, and its loss is ln(V). Anything below that has learned something.

Python Vocabulary, encoding and a counting baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The counting model is the best any model can do if it only sees one previous character, because its table is the exact frequency of each pair in the data. Its loss is the floor for the next step. Notice how much it already knows: after a t it expects an h about half the time, because "the" is the most common word in the text.

Step 2: learn the same table with gradient descent

Counting does not scale to long contexts, so real models learn their probabilities instead. To see how, we make the bigram model learnable. Its parameters are a matrix W with one row of logits (raw scores) per input character. To predict, we take the row for the current character and turn it into probabilities with softmax, as described in Logits, Softmax and Temperature.

Training then runs this loop:

  1. Forward pass: compute probabilities and the loss.
  2. Backward pass: compute the gradient, the direction in which each weight should move to lower the loss. For softmax with cross-entropy the gradient has a lovely simple form: the predicted probabilities, minus 1 at the correct character.
  3. Update: subtract the gradient times a learning rate.

We start with W = 0, which gives every character the same probability, so the loss starts at ln(V). The plot shows it falling toward the counting floor and flattening there. It cannot go lower, and that is a useful check that your gradient code is right: a learned model that beats its own mathematical floor on the training data would signal a bug.

Python A learned bigram model and its loss curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two models end up almost identical. That is the key insight of the project: training is search for the numbers that counting would have given you, in situations where counting is impossible. With a context of four characters there are about 390,000 possible contexts, far more than our text has characters, so a table can never be filled in. A neural network instead shares what it learns between similar contexts, which is the same promise embeddings made in Embeddings: Meaning as Numbers.

Step 3: a small neural network with context

Now we give the model four characters of context. The architecture is a miniature of what happens inside bigger models:

flowchart LR
    A["4 character ids"] --> B["Embedding table<br/>(8 numbers per character)"]
    B --> C["Concatenate: 32 numbers"]
    C --> D["Hidden layer: 48 units, tanh"]
    D --> E["Output layer: 25 logits"]
    E --> F["Softmax: probabilities"]

Characters become vectors, the hidden layer mixes them, and the output layer scores every possible next character.

The code writes the backward pass by hand, one layer at a time, applying the chain rule from the loss back to the embedding table. It is the same calculation a framework such as PyTorch does for you with loss.backward(). We also split off the last 10% of the text as a validation set that the model never trains on. Training loss tells you how well the model fits what it has seen; validation loss tells you how well it handles new text. You will see them diverge.

Python Train a four-character-context network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the two curves. The training loss keeps falling to a tiny value: the network has enough parameters (a few thousand) and the text is so short that it can simply memorize it. The validation loss falls for a while and then rises. That turning point is where a real training run would stop, an idea called early stopping. Past it, the model is getting better at the training text and worse at everything else.

A real LLM avoids this in two ways: it trains on trillions of tokens, so memorizing is impractical, and it is evaluated on held-out text throughout. Still, memorization does happen with repeated or rare text, which is part of why copyright and privacy matter.

Step 4: generate text and play with temperature

The final step is generation. Start with a prompt, feed the last four characters to the network, get probabilities for the next character, divide the logits by a temperature before the softmax, sample one character, append it and repeat. You know this loop from Logits, Softmax and Temperature; here it runs on a model you trained yourself.

This TryIt repeats the training (it takes a couple of seconds) so it is self-contained, then samples at three temperatures.

Python Sample text at three temperatures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At temperature 0.3 the output copies fragments of the training text almost word for word, because the overfit network is nearly certain about each next character. Around 0.8 it stitches phrases together, often fluently. At 1.5 the distribution flattens and you get invented, misspelled words. Real chatbots behave the same way; they just have far better distributions to flatten.

What you built, and what real models add

You trained a genuine language model. Compared with a production LLM it is missing a few big things, each of which has its own lesson:

  • A real tokenizer that splits text into subwords instead of characters (Byte-Pair Encoding from Scratch).
  • Attention instead of a fixed window of four characters, so any earlier token can matter (Attention from Scratch and The Transformer Block).
  • Scale: billions of parameters and trillions of tokens, which is the subject of How LLMs Are Trained.
  • Better optimizers and regularization, such as Adam, dropout and learning-rate schedules, rather than plain gradient descent.

Try some experiments: change HIDDEN to 8 and see how overfitting changes; set CONTEXT to 1 and compare the validation loss with the bigram floor; replace the text with your own paragraph (the vocabulary updates automatically); stop training at step 100 instead of 400 and compare the samples.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What loss does a model get if it gives every one of V characters the same probability?

    Each correct character gets probability 1/V, so the loss is -ln(1/V) = ln(V). Any model that has learned something does better.
  2. 2. Why does the validation loss start rising while the training loss keeps falling?

    That divergence is overfitting. The network has enough capacity to memorize a tiny text, so held-out loss gets worse.
  3. 3. For softmax with cross-entropy loss, what is the gradient with respect to the logits?

    This simple form (probabilities minus the one-hot target) is why softmax and cross-entropy are always used together.
  4. 4. What does lowering the temperature do during sampling?

    Dividing logits by a small temperature widens the gaps between them, so the most likely characters dominate.

Summary

  • A language model is trained by predicting the next token, measuring cross-entropy loss and following the gradient downhill.
  • A character-level model needs no tokenizer; its vocabulary is just the distinct characters.
  • A counting bigram is the floor for any model that sees one character; gradient descent finds the same table.
  • A neural network shares what it learns across contexts, so it can use longer context than a table can.
  • Validation loss on held-out text reveals overfitting; early stopping picks the best moment to stop.
  • Sampling with a temperature turns the model's probabilities into text.

Next: test what you have learned in Exercises with Solutions.