Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: A Tiny Language Model

Build a word-level bigram language model with NumPy: counting, smoothing, sampling with temperature and measuring perplexity, the same ideas behind full-size LLMs.

A large language model and a bigram model do the same job: given what came before, produce a probability distribution over the next token. The bigram model only looks one token back and stores its answers in a table, so you can build it in a few lines and see every number. By the end you will have a model that trains, generates, and is scored with perplexity, exactly like its bigger relatives.

The corpus and the vocabulary

Our corpus is a few short sentences. We lowercase, split on spaces and add a start marker <s> and an end marker </s>, the simplest form of the special tokens from vocabulary and chat templates.

Python Vocabulary and token ids

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Count, then normalize

For every pair of neighbours (previous, next) we add one to a count matrix. Dividing each row by its sum turns counts into probabilities: row i is the model's distribution over the next token after token i. That matrix is the entire model.

Python Train by counting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After "the" the model splits its belief among cat, dog, mat and rug according to how often each followed "the" in the corpus. That is next-token prediction with a lookup table instead of a transformer (next-token prediction).

Generate with temperature

To generate, start at <s>, sample the next token from the current row, and repeat until </s>. Temperature rescales probabilities before sampling: below 1 sharpens them, above 1 flattens them (decoding).

Python Sampling with temperature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the model sees only one word of context, it can loop ("the dog sat on the dog sat on ..."): it has no memory of what it already said. A longer context fixes that, which is the reason for attention and bigger context windows. Low temperature repeats the most common path; the 1.0 and 2.0 rows happen to match here because the random draws land in the same places with this tiny table.

Smoothing and perplexity

If a test sentence contains a pair never seen in training, the probability is zero and the log is infinite. Add-k smoothing gives every pair a small pseudo-count so nothing is impossible. We then score held-out sentences with perplexity, the exponential of the average negative log-probability per token (lower is better).

Python Perplexity with add-k smoothing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scrambled third sentence gets a far higher perplexity than the two natural ones: the model is much more "surprised". Notice the effect of k: with a tiny k the unseen pairs in the scrambled sentence are punished enormously (432), while a large k flattens everything, so the natural sentences get worse (1.93 becomes 4.93) but the scrambled one is punished less. Choose k on held-out data.

From this toy to a real LLM

Idea Bigram model Transformer LLM
Context one previous token thousands of tokens
Parameters a V x V table billions of learned weights
Training counting gradient descent on cross-entropy
Tokens whole words subwords (BPE)
Output a row of probabilities a softmax over the vocabulary

The table also explains the scaling problem: a bigram table grows with the square of the vocabulary and still sees only one word. A neural network shares statistical strength across similar words through embeddings, so it generalises to word pairs it never saw.

Challenges

  1. Extend the model to a trigram (two previous tokens). How does the table size change and what happens to unseen contexts?
  2. Switch to character-level tokens and generate names from a list you write.
  3. Implement top-k sampling in sample_sentence.
  4. Compare perplexity on training versus held-out text and explain the gap (overfitting).
  5. Use the evaluation harness to score generated sentences with a rule such as "contains a verb".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a bigram model, what is the full set of learned parameters?

    Training only counts neighbouring pairs and normalizes rows, so the table is the model.
  2. 2. What does lowering the temperature to 0.3 do when sampling?

    Dividing logits by a small temperature sharpens the distribution.
  3. 3. Why is smoothing needed before computing perplexity?

    A zero probability gives an infinite negative log, so a pseudo-count keeps scores finite.
  4. 4. Why does the scrambled sentence receive a higher perplexity?

    Bigram probabilities depend on order; unusual pairs have low probability, so surprise is high.

Summary

  • A language model is a function from context to a distribution over the next token; a bigram table is the simplest one.
  • Training by counting and normalizing gives the probabilities; temperature reshapes them at sampling time.
  • Smoothing avoids zero probabilities, and perplexity measures how surprised the model is by held-out text.
  • Transformers replace the one-token window and the table with attention, embeddings and learned weights.

Next: Practice: Exercises and Review