Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 53 ยท Customizing Models

Alignment: RLHF, DPO and Preference Data

Learn how preference pairs, reward models, RLHF and DPO make models helpful and safe, with a toy Bradley-Terry reward model you can train yourself.

A model trained only to predict the next token is a good imitator of the internet, but not a good assistant. Ask it a question and it may continue with another question, ramble, or produce something harmful, because all of those appear in its training text. Alignment is the set of techniques that steer a model toward what people actually want: helpful, honest and harmless answers.

Instruction tuning on example conversations gets part of the way (see How LLMs Are Trained). The next step uses preferences: humans or other models compare two answers and say which is better. This lesson explains reward models, RLHF, DPO and related ideas, and trains a toy reward model from preference pairs.

Why preferences instead of examples?

Writing a perfect answer is hard. Choosing the better of two answers is easy and fast, and people agree with each other more often. A preference dataset is a list of records like this:

JSON
{"prompt": "Explain recursion to a child.",
 "chosen": "Imagine opening a box that has a smaller box inside, and keep going until you find the last one...",
 "rejected": "Recursion is a function invoking itself with a reduced argument domain."}

The alignment toolbox

Instruction tuning Learn from written examples.
Preference data Compare two answers, pick one.
Reward model Turns picks into a score.
Policy update Move the model toward high scores.

RLHF and DPO differ in how the last step is done.

1 / 2

The RLHF pipeline

Reinforcement learning from human feedback (RLHF) has three stages:

  1. Supervised fine-tuning (SFT): tune the base model on good demonstrations so it follows instructions.
  2. Reward model: collect comparisons of model answers, then train a separate model that outputs a single score for a prompt-answer pair. Higher means "people preferred this".
  3. Reinforcement learning: sample answers from the SFT model (the policy), score them with the reward model, and update the policy to raise the score, usually with an algorithm such as PPO. A KL penalty keeps the policy close to the SFT model so it does not drift into strange text that merely fools the reward model.
flowchart LR
    A["Base model"] --> B["SFT model"]
    B --> C["Sample answers"]
    C --> D["Humans rank pairs"]
    D --> E["Reward model"]
    E --> F["RL update (PPO + KL penalty)"]
    F --> B

RLHF loop: the reward model turns human comparisons into a training signal for the policy.

A well-known failure is reward hacking: the policy finds answers that score high but are not actually good, for example long, flattering or over-confident replies. The reward model is only a proxy for what people want.

The Bradley-Terry reward model

Reward models are trained on pairs with the Bradley-Terry model. Each answer gets a score r. The probability that answer A beats answer B depends on the score difference through a sigmoid:

TEXT
P(A preferred over B) = sigmoid(r(A) - r(B))
loss = -log sigmoid(r(chosen) - r(rejected))

If the chosen answer already scores higher, the loss is small; if not, the gradient raises the chosen score and lowers the rejected one. Real reward models are language models with a scalar head. The toy version below scores an answer as a weighted sum of simple features (length, politeness words, hedges, a "rude" flag) and learns the weights from pairs.

Python Train a toy reward model on preference pairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model learned that politeness and a sensible length raise the score while rudeness lowers it. It also learned that length is rewarded, which hints at reward hacking: a policy optimized against this model would drift toward longer answers. Real teams check for this and add length controls.

DPO: skip the reward model

Direct Preference Optimization (DPO) reaches a similar goal without training a separate reward model or running reinforcement learning. It trains the model directly on the preference pairs with a loss that increases the probability of the chosen answer relative to the rejected one, measured against a frozen reference model (usually the SFT model):

TEXT
loss = -log sigmoid( beta * [ (log pi(chosen) - log ref(chosen)) - (log pi(rejected) - log ref(rejected)) ] )

beta controls how far the model may move from the reference, which plays the role of the KL penalty. DPO is simpler, cheaper and more stable than PPO, so it is common in open-source tuning, and variants such as IPO, KTO and ORPO adjust the details. RLHF with a reward model remains useful when you want to reuse the scorer, for example to rank many sampled answers.

Python The DPO loss for one pair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts at about 0.693 (that is log 2, a coin flip) and falls as the policy shifts probability toward the chosen answer relative to the reference.

Other ways to get preferences

  • Constitutional approaches give the model a written set of principles and have a model critique and revise its own answers, or judge pairs against the principles. The results become preference data with far fewer human labels (often called RLAIF, reinforcement learning from AI feedback).
  • Rule-based or verifiable rewards: for math and code, a checker (unit tests, exact answer match) provides the reward directly, which is harder to hack than a learned score. This is a key ingredient of many reasoning models; see Reasoning prompts.
  • Red teaming adds adversarial prompts so the model learns to refuse harmful requests; see Prompt Injection and LLM Security.

Distillation and synthetic data

Distillation trains a small "student" model to imitate the outputs of a larger "teacher". The student sees the teacher's answers (and sometimes its probability distributions) and often gets surprisingly close at a fraction of the cost. Synthetic data means training examples generated by a model rather than written by people: question-answer pairs, instruction variations, preference pairs ranked by a judge. It is cheap and scalable, but errors and biases of the generator are copied and can be amplified over generations, so filter it, verify it where you can, and keep real data in the mix. Check the generator's license terms before using its outputs to train another model.

Limits of alignment

Alignment makes good behavior more likely; it does not make bad behavior impossible. Tuned models can still be jailbroken, still hallucinate, and can become sycophantic, agreeing with the user to win approval. Whose preferences count, and how to handle disagreement across cultures, is an open question. See Bias, Privacy, Copyright and Responsible Use.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a reward model trained on?

    Reward models learn to score answers so that preferred answers score higher than rejected ones.
  2. 2. What is the purpose of the KL penalty in RLHF?

    Without it, the policy can drift toward odd text that scores well but is not actually good.
  3. 3. How does DPO differ from RLHF with PPO?

    DPO folds the reward model into a loss on the policy itself, measured against a frozen reference model.
  4. 4. What is reward hacking?

    The reward model is only a proxy, and optimization pressure finds its weaknesses, such as rewarding length.

Summary

  • Preference data (chosen versus rejected answers) is easier to collect than perfect demonstrations.
  • RLHF: SFT, then a reward model trained with the Bradley-Terry loss, then RL with a KL penalty. Reward hacking is the main risk.
  • DPO optimizes the policy directly on pairs against a reference model, which is simpler and stable.
  • Constitutional/AI feedback, verifiable rewards, distillation and synthetic data reduce human labeling but need filtering.
  • Alignment shifts behavior; it does not guarantee safety.

Next: How Image Generation Works.