Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 73 ยท Sequences, Attention and Transformers

How Large Language Models Work

Understand large language models: pretraining, BPE tokenizers, scaling laws, fine-tuning, RLHF, the KV cache, context windows and limits, with runnable code.

A large language model (LLM) is the model you just built, scaled up. The tiny language model predicted the next character from a few thousand characters of text. An LLM predicts the next token after training on trillions of them, with billions of parameters and dozens of transformer blocks. No new mathematical idea is needed to go from one to the other; what changes is scale, plus a few engineering and training tricks that turn a raw text predictor into a helpful assistant.

This lesson is the map of that journey. You will build a real byte-pair-encoding tokenizer, see a scaling law, train a toy reward model the way RLHF does, and measure what the KV cache saves. The code is small on purpose; the ideas are the ones used in production systems.

From text to assistant

Pretraining Predict the next token across a huge pile of text.
Fine-tuning Learn from examples of good answers to instructions.
Preference tuning Move toward answers that people rank higher.
Serving Generate token by token, fast, with a KV cache.
1 / 2

Pretraining: next-token prediction at scale

Pretraining uses exactly the objective from the last lesson: for every position in a text, predict the token that follows, and minimize the cross-entropy. The training text is a mix of web pages, books, code, scientific papers and other documents, cleaned and de-duplicated, and it runs to trillions of tokens. Nobody labels anything. The text itself provides the answers, which is why this is called self-supervised learning.

To predict the next token well across that much varied text, the model has to pick up grammar, facts, styles, code syntax, and some reasoning patterns. It is not told to; those skills are simply useful for reducing the loss. The result is a base model: very good at continuing text, but not yet good at following instructions. Ask a base model a question and it may continue with more questions, because that is what a list of questions on the web often looks like.

Training such a model takes thousands of accelerators for weeks, using parallelism tricks that split the model, the data and the optimizer state across machines. The code of one training step is the same as in your NumPy version: forward, loss, backward, update.

Tokenizers: what the model actually reads

A model with one token per character would need very long sequences, and attention cost grows with the square of the length. A model with one token per word would need an enormous vocabulary and could not handle new words. The standard compromise is sub-word tokenization, and the most common method is byte-pair encoding (BPE):

  1. Start with a vocabulary of single characters (production systems start from the 256 possible bytes, so any text can be encoded).
  2. Count every adjacent pair of tokens in the training text.
  3. Merge the most frequent pair into a new token.
  4. Repeat until the vocabulary reaches the target size, commonly tens of thousands to a couple of hundred thousand tokens.

To encode new text, you replay the merges in the order they were learned. Common words become a single token; rare words fall apart into pieces. Here is the whole algorithm on a tiny word list, with _ marking the end of a word.

Python Byte-pair encoding from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first merges are the most frequent pairs (k+e, t+o) and then build upward into token, talk and finally tokenize. Look at how the results differ. The common word token costs one token. talking becomes talk plus three single letters, because ing was not frequent enough in this tiny corpus to earn a merge. The unseen word tokenizing reuses the tokeni piece, and xyz falls all the way back to single characters. Real tokenizers behave the same way at a much larger scale, and their quirks explain some LLM oddities: a model sees strawberry as a few tokens, not ten letters, so "count the letters" questions are harder than they look. Text in languages that were rare in the training data also splits into more tokens, which makes it slower and costlier to process.

A useful rule of thumb for English is about 3 to 4 characters per token, or roughly three quarters of a word.

Scaling laws

One of the most influential findings in the field is that the loss of a language model falls smoothly and predictably as you increase three things: the number of parameters N, the amount of training data D (in tokens), and the compute C. On log-log axes, the improvement is close to a straight line (a power law). That predictability lets labs forecast the result of an enormous training run from many small ones.

A common way to write it is:

loss = floor + A / N^alpha + B / D^beta

There is an irreducible floor (the true randomness of text), a term that shrinks as the model grows, and a term that shrinks as the data grows. Training compute is roughly 6 * N * D floating-point operations. For a fixed budget, making the model bigger means training it on fewer tokens, so there is a best split. Studies of compute-optimal training found that earlier large models were too big for the data they saw, and that parameters and tokens should grow together, on the order of 20 tokens per parameter.

The constants below are illustrative values of the same flavor as published fits, not measurements.

Python A scaling law and a compute budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table row by row. A 1-million-parameter model barely improves with more data (6.586 to 5.442) because its own size is the bottleneck. A 100-billion-parameter model starved of data (3.011 with 1 billion tokens) is held back the other way. The plot shows nearly straight lines on log axes. With a fixed budget of 1e21 FLOPs, this toy law picks a model of about 1.8 billion parameters trained on about 94 billion tokens, roughly 53 tokens per parameter. The real ratio depends on the fitted constants (published fits land nearer 20), and in practice many teams train smaller models on far more tokens than compute-optimal, because a smaller model is cheaper to serve afterward.

From base model to assistant

Pretraining gives the model knowledge and skill. Everything after it shapes behavior: how the model responds, what it refuses, and the tone it uses.

flowchart LR
    A["Raw text<br/>(trillions of tokens)"] --> B["Pretraining"]
    B --> C["Base model"]
    C --> D["Supervised fine-tuning<br/>(instruction + answer pairs)"]
    D --> E["Preference tuning<br/>(RLHF or similar)"]
    E --> F["Assistant"]

The usual path from raw text to a chat assistant.

Supervised fine-tuning (SFT) continues training on a much smaller set of curated examples, each an instruction paired with a good answer, using the same next-token loss but only on the answer tokens. After SFT the model follows instructions instead of merely continuing text. Updating all the weights is expensive, so people often use parameter-efficient fine-tuning such as LoRA, which freezes the original weights and trains small low-rank add-on matrices.

Reinforcement learning from human feedback (RLHF) tackles something SFT cannot: it is easier for people to say which of two answers is better than to write a perfect answer themselves. The recipe has three steps:

  1. Collect pairs of model answers and have people choose the better one.
  2. Train a reward model that assigns each answer a score, so that preferred answers score higher than rejected ones.
  3. Adjust the language model to produce answers with high reward, while a penalty keeps it from drifting too far from the SFT model (otherwise it learns to exploit quirks of the reward model).

Newer methods such as direct preference optimization (DPO) skip the explicit reward model and optimize the preference data directly, and some systems use rewards that can be checked automatically, for example whether code passes its tests or a math answer is correct. The reward model itself is an ordinary small neural network trained on a simple pairwise loss, and you can train a toy one right now.

Python A toy reward model from preference pairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each answer here is described by two made-up numbers, and the "human" preferences are noisy, so even the true taste is not followed 100% of the time. The pairwise loss starts at 0.693 (which is ln 2: a coin flip) and settles at 0.378. The learned weights, about [2.01, -0.44], are close to the hidden taste [2, -0.5]: the model has discovered that people like the first feature and dislike the second. It ranks 83% of the pairs the same way the people did, which is about the ceiling given the noise. The last lines use it the simplest possible way, picking the best of five candidates ("best-of-n"). Real systems use the reward to train the language model itself.

Inference: generating one token at a time

Generating text is an autoregressive loop: predict a token, append it, predict the next. A naive loop would recompute attention keys and values for the entire prefix at every step, so the work grows quadratically with the length of the output. The key observation is that, because of the causal mask, the keys and values of earlier tokens never change. So you compute them once and store them: this is the KV cache.

With a cache, each step only computes the query, key and value for the one new token and attends to the stored keys and values. The next example generates 10 tokens both ways and counts the work.

Python Why the KV cache works

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The outputs are identical (True), but the uncached version does 55 key/value projections for 10 tokens versus 10 with the cache. For a 10,000-token output the gap is about 50 million versus 10 thousand. The price is memory: the cache stores two vectors (key and value) per layer for every token. For a model with 32 layers, width 4096 and a 4,096-token context in 16-bit numbers, that is about 2.0 GiB for each conversation, which is one reason serving long contexts to many users is hard.

Context windows

The context window is the maximum number of tokens the model can look at in one pass: the prompt, the conversation so far and the answer all share it. Anything outside is invisible to the model; it does not "remember" earlier chats unless the text is put back in the prompt. Windows have grown from a few thousand tokens to hundreds of thousands, helped by better position encodings, attention variants and cache management, but attention over T tokens still costs about T^2 work when the prompt is first processed, and long inputs are not used equally well everywhere in the window.

Prompting basics

Because an LLM is a conditional text predictor, the prompt is how you steer it.

  • Be specific. State the task, the audience, the length and the format you want. Vague prompts get generic answers.
  • Show examples. A few input-output pairs in the prompt (few-shot prompting) often work better than a long description.
  • Give structure. Ask for a specific format such as a list, a table or JSON, and say what to do when the answer is unknown.
  • Separate instructions from data. Mark where the text to be processed begins and ends so that instructions are not confused with content.
  • Let it work through steps. For multi-step problems, asking the model to reason step by step before answering often improves accuracy.
  • Mind the sampling settings. Low temperature gives consistent output (good for extraction); higher temperature gives variety (good for brainstorming), as you saw in the tiny language model.

Limitations you should always keep in mind

  • Hallucination. The model optimizes for plausible continuations, not truth. It can state invented facts, citations or code APIs with complete confidence. Verify anything that matters.
  • Knowledge cutoff. It knows only what was in its training data. Tools such as search or document retrieval can supply current or private facts at inference time.
  • Weak exactness. Arithmetic, counting and strict logic can fail, partly because of tokenization. Connecting a calculator or code interpreter fixes many of these.
  • Bias. Models absorb patterns, including unfair ones, from their data (see Responsible Deep Learning).
  • Prompt injection. When a model reads untrusted text (a web page, an email), instructions hidden in that text may be followed. Treat model output as untrusted input to the rest of your system.
  • Cost and latency. Each generated token needs a full pass through every block, so long answers take time and money. Quantization and efficient serving reduce both.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What objective is used in pretraining a large language model?

    Pretraining is self-supervised next-token prediction. The text supplies its own targets, so no manual labels are needed.
  2. 2. In byte-pair encoding, what happens at each step of tokenizer training?

    BPE repeatedly merges the most frequent neighboring pair. At encoding time those merges are replayed in order, so common words end up as few tokens and rare ones as many pieces.
  3. 3. What does the KV cache store?

    With a causal mask, earlier keys and values never change, so they can be saved and reused. The cost is memory that grows with the context length.
  4. 4. In RLHF, what is the reward model trained on?

    People find it easier to compare two answers than to write a perfect one. The reward model learns scores that rank the preferred answer above the rejected one.
  5. 5. Why can a language model state a false fact fluently?

    The training objective rewards likely text. Truth is only learned to the extent that it helps predict the data, so confident mistakes (hallucinations) happen.

Summary

  • An LLM is a very large decoder-only transformer pretrained to predict the next token on trillions of tokens of text; the objective is the same as in the tiny model.
  • Sub-word tokenizers such as BPE merge frequent pairs repeatedly; common words become one token, rare ones several, and unseen text falls back to small pieces.
  • Scaling laws show that loss falls as a power law in parameters, data and compute, so large runs can be planned from small ones.
  • Fine-tuning on instruction examples and preference tuning (RLHF, DPO and relatives) turn a base model into an assistant.
  • The KV cache stores keys and values of earlier tokens, turning quadratic recomputation into one new projection per token at the cost of memory; the context window limits what the model can see.
  • Good prompts are specific, structured and example-rich, and models still hallucinate, have a knowledge cutoff, and can be fooled by injected instructions.

Next: Autoencoders.