Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 53 ยท Getting Started

How LLMs Are Trained

Understand how large language models are trained: next-token prediction, scale and data, instruction tuning, RLHF, base versus chat models, loss and perplexity.

A chat model looks like one finished product, but it is usually the result of several training stages stacked on top of each other. First the model reads a gigantic amount of text and learns to continue it. Then it is taught to follow instructions. Finally it is nudged toward answers that people prefer. Knowing the stages explains many behaviours you will see later: why a raw model rambles, why chat models are polite, and why they sometimes refuse harmless requests.

This lesson walks through the stages in order and ends with a small calculation you can run: how loss and perplexity measure how well a model predicts text.

Three stages of training

Pretraining Learn to continue huge amounts of text.
Instruction tuning Learn to follow requests.
Preference tuning Learn which answers people like.

Each stage starts from the result of the one before.

1 / 2

Stage 1: pretraining on next-token prediction

Pretraining has a single, simple task. Take a chunk of text, hide the next token, and ask the model to guess it. Do that for every position in trillions of tokens of web pages, books, code and papers.

For the text the cat sat on the, the model outputs a probability for every token in its vocabulary. If the true next token was mat, the training signal says: make mat more likely next time in this context. The number that measures how wrong the model was is the loss.

Because the answer is already in the text, no human has to label anything. This is called self-supervised learning, and it is why training on enormous datasets is practical.

To get good at predicting text across all topics, the network has to absorb grammar, facts, writing styles, programming patterns and a lot about how people reason on paper. Nobody programs those abilities in; they appear because they help reduce the loss.

Scale and data

Three ingredients decide how capable a pretrained model becomes:

  • Parameters: the adjustable numbers inside the network. Models range from millions to hundreds of billions.
  • Data: the number of tokens the model sees, and how clean, varied and deduplicated they are.
  • Compute: the processor time spent. Training a large model takes thousands of accelerators for weeks or months.

Research on "scaling laws" found that loss falls smoothly and predictably as you increase these together. Data quality matters as much as quantity: filtering low-quality text, removing duplicates and balancing code, prose and multiple languages all change the result.

Base models versus chat models

The output of pretraining is a base model. It is excellent at continuing text, and that is all it does. Give it the question "What is the capital of France?" and it may continue with more quiz questions, because that is a likely continuation on the web.

A chat (or instruct) model has been trained further so that a question gets an answer. The difference is easy to feel with a toy example.

Python Base model vs chat model behaviour

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The chat model is not smarter at the core; the same knowledge is there. It has learned a format in which a user turn is followed by a helpful assistant turn. The exact marker strings differ between model families, and real APIs hide them behind a list of messages (see Calling an LLM API).

Stage 2: instruction tuning

Instruction tuning (also called supervised fine-tuning, SFT) continues training the base model on tens of thousands to millions of examples of the form "instruction in, good response out". People, or other models checked by people, write the responses. The loss is the same next-token loss, but computed only on the assistant's part of each example.

This stage teaches style and behaviour: answer the question asked, follow format requests, use a polite tone, decline clearly unsafe requests. It adds little new factual knowledge compared with pretraining. A common saying is that pretraining gives knowledge and tuning gives manners.

Stage 3: learning from preferences

An answer can be correct and still unhelpful, too long or rude. It is hard to write a loss function for "a good answer", but it is easy for a person to say which of two answers is better. Reinforcement learning from human feedback (RLHF) uses that:

  1. The model writes several answers to the same prompt.
  2. People rank them.
  3. A reward model is trained to predict those rankings.
  4. The chat model is adjusted to produce answers the reward model scores highly, while staying close to its previous behaviour so it does not drift into nonsense.

Newer methods such as DPO skip the separate reward model and learn straight from pairs of preferred and rejected answers. Both are covered in detail in Alignment: RLHF, DPO and Preference Data.

flowchart LR
    D["Web text, books, code"] --> B["Base model"]
    B --> S["Instruction tuning"]
    S --> R["Preference tuning"]
    R --> C["Chat model"]

Each stage reuses the weights from the previous one.

Measuring prediction quality: loss and perplexity

The loss used in pretraining is cross-entropy. For one position it is -log(p), where p is the probability the model gave to the token that actually came next. Confident and right gives a loss near 0; giving the right token a tiny probability gives a large loss. Averaging over many positions gives the average loss.

Perplexity is exp(average loss). You can read it as "the model is as unsure as if it were choosing uniformly among this many tokens". A model that splits probability evenly among 4 options everywhere has perplexity 4.

Python Loss and perplexity by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unsure model prints a perplexity of exactly 4.00, matching the "choosing among 4 options" reading. Lower is better, and you can only compare perplexity between models that use the same tokenizer and test text.

Watching loss fall

Training is just repeated nudging of parameters to lower the loss. The next example trains a model with only three weights (the scores for three possible next tokens in a single context) using gradient descent, and prints how the loss and the probability of the right token change.

Python A model learns by lowering loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The probabilities drift toward 0.7, 0.2 and 0.1, which are the frequencies in the data, and the loss settles at the entropy of that data. Real training does the same thing with billions of weights and many contexts at once, which is why a model can learn patterns such as grammar without anyone writing a rule.

What training does not give you

Training stops at a knowledge cutoff, so a model does not know recent events unless you provide them. The weights are fixed afterwards: a chat does not change the model. Training also does not store the dataset as searchable documents; it compresses statistics into the weights, which is why recall of exact details can be fuzzy. These facts motivate retrieval (RAG: Giving Models Your Data) and fine-tuning (Fine-Tuning vs Prompting).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the training task in pretraining?

    Pretraining hides the next token of ordinary text and rewards the model for assigning it high probability. No human labels are needed.
  2. 2. Which statement about base and chat models is correct?

    Chat models start from a pretrained base model and receive instruction tuning and preference tuning on top.
  3. 3. A model gives every next token probability 0.1 over ten equally likely options at every position. What is its perplexity?

    Perplexity is exp of the average loss. With p = 0.1 the loss is log(10), so the perplexity is 10, as if choosing among 10 options.
  4. 4. What is the purpose of the reward model in RLHF?

    The reward model learns from human rankings and then supplies a score that guides further tuning of the chat model.

Summary

  • Pretraining teaches a model to predict the next token on trillions of tokens, with no human labels.
  • Capability grows with parameters, data quality and quantity, and compute.
  • A base model continues text; instruction tuning teaches it to answer requests, and preference tuning (RLHF or DPO) steers it toward preferred answers.
  • Cross-entropy loss is -log(p) of the true next token; perplexity is exp(average loss).
  • Weights are frozen after training and the knowledge has a cutoff date.

Next: see how today's models compare in The Model Landscape.