Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Fine-tuning and Alignment

RLHF and DPO: Learning from Preferences

Learn how LLMs are aligned with human preferences: Bradley-Terry reward models, the RLHF pipeline with a KL penalty, and DPO's closed-form loss, with NumPy examples.

After supervised fine-tuning a model can follow instructions, but it may still be verbose, evasive, unsafe or simply mediocre. People find it much easier to say which of two answers is better than to write a perfect answer. Preference tuning uses those comparisons to push the model toward answers people like. Two families dominate: RLHF (reinforcement learning from human feedback) and DPO (direct preference optimization).

flowchart LR
    A["Pretrained model"] --> B["SFT model (reference)"]
    B --> C["Sample answers, humans rank pairs"]
    C --> D["Reward model"]
    D --> E["RL (PPO or similar) with KL penalty"]
    E --> F["Aligned model"]
    C -.->|"DPO skips the reward model"| F

DPO trains on the same pairs but without a separate reward model or RL loop.

Preference data

Each example has a prompt x, a chosen answer y_w and a rejected answer y_l. Annotators (humans, or another model in "AI feedback" setups) pick the winner. Typical datasets contain tens of thousands to hundreds of thousands of pairs.

Reward models and Bradley-Terry

A reward model r(x, y) assigns a scalar score to an answer. We want r(chosen) > r(rejected). The standard assumption is the Bradley-Terry model: the probability that A beats B is

Text
P(A > B) = sigmoid(r(A) - r(B))

Training minimizes -log sigmoid(r(chosen) - r(rejected)) over the pairs. In a real system r is a transformer with a scalar head. Here we use a linear model on hand-made features so you can see the mechanics.

Python Fit a reward model from pairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The learned weights point in the same direction as the true ones (the scale is not identifiable exactly with limited noisy data). Pairwise accuracy tops out below 100% because annotators are noisy: that noise is built into the Bradley-Terry sampling.

The RLHF objective

With a reward model in hand, RLHF optimizes the policy pi (the model being trained) to maximize reward while staying close to the reference model pi_ref (the SFT model):

Text
maximize  E[ r(x, y) ]  -  beta * KL( pi(y|x) || pi_ref(y|x) )

The KL penalty is crucial. Without it the policy finds answers that fool the reward model (reward hacking), such as long flattering text, and drifts into gibberish. beta sets how far the policy may move. The optimization is usually done with PPO or a simpler policy-gradient variant, sampling answers from the model during training, which makes it expensive and finicky: four models are involved (policy, reference, reward, value).

DPO: the shortcut

The RLHF objective has a known closed-form optimum: pi*(y|x) is proportional to pi_ref(y|x) * exp(r(x,y) / beta). Rearranged, the reward equals beta * log(pi/pi_ref) plus a prompt-only constant. Substituting this into the Bradley-Terry loss gives the DPO loss:

Text
L = -log sigmoid( beta * [ log pi(y_w|x) - log pi_ref(y_w|x) ]
                - beta * [ log pi(y_l|x) - log pi_ref(y_l|x) ] )

It needs only the log-probabilities of the two answers under the policy and the frozen reference. No sampling, no reward model, plain supervised-style training.

We can watch it work on a toy: a policy over four candidate answers, parameterized by logits. The reference prefers answer 0, but the data says answer 2 is usually chosen.

Python DPO on four candidate answers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Probability mass moves from answer 0 toward answer 2. A small beta lets the policy travel far (large KL); a large beta anchors it near the reference. That is the same knob as the KL coefficient in RLHF.

Choosing and using preference methods

Method Needs Strength Weakness
RLHF (PPO) reward model, sampling flexible, handles any reward complex and unstable
DPO preference pairs simple, stable offline, tied to the data distribution
IPO, KTO, ORPO and others pairs or good/bad labels variants for noise or unpaired data less battle-tested
RLAIF / Constitutional AI model-written feedback scales without humans inherits the judge's biases
RL with verifiable rewards checkers such as unit tests or math answers no human labels only where answers can be verified

Things that go wrong

  • Length bias: judges often prefer longer answers, so models get wordy. Length-normalize or control for it.
  • Sycophancy: agreeing with the user wins comparisons, even when the user is wrong.
  • Reward hacking: the policy exploits quirks of the reward model.
  • Alignment tax: over-optimization can hurt abilities measured by benchmarks.
  • Noisy labels: annotators disagree; measure agreement (human evaluation).

For the safety side of alignment, see Hallucinations, Bias and Safety.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the Bradley-Terry model, P(A beats B) equals what?

    The probability depends on the difference of the scores passed through a sigmoid.
  2. 2. What is the KL penalty in RLHF for?

    Without it, the policy can drift to degenerate text that scores high on the reward model.
  3. 3. What does DPO remove compared with RLHF?

    DPO optimizes a closed-form loss directly on preference pairs, using log-probabilities from the policy and a frozen reference.
  4. 4. In the toy experiment, increasing beta did what?

    Beta scales the implicit KL anchor, so a large beta restrains how far the probabilities move.

Summary

  • Preference tuning learns from comparisons of answers: chosen vs rejected.
  • RLHF trains a Bradley-Terry reward model and optimizes the policy with a KL leash; DPO folds both steps into one supervised-style loss.
  • Beta controls the distance from the reference model.
  • Watch for length bias, sycophancy and reward hacking, and evaluate before and after.

Next: Decoding and Sampling