A03 — DPO and the small RLHF bridge
← A02 — Adaptation comparison · A04 — RLHF bridge → · Glossary
Prerequisites: A01, C03. Allow one session.
The code is src/llm_course/alignment.py; read response_log_probs and
dpo_loss beside this page, and leave _fit_reward_model and _ppo_policy for
A04. Every function validates its tensors and returns a new
value; the reference policy is never modified.
Sum the response log-probabilities
response_log_probs sums only unmasked response tokens; prompt and padding labels
are -100. Unlike A01’s sft_loss, it expects labels that are already shifted:
labels[b, t] is the token that logits[b, t] should predict. Take one sequence
of three positions over a three-token vocabulary whose softmax rows are
[0.5, 0.3, 0.2], [0.25, 0.5, 0.25] and [0.1, 0.1, 0.8], with labels
[-100, 1, 2]. The first position is prompt and contributes nothing; the sum is
log 0.5 + log 0.8 = -0.6931 - 0.2231 = -0.9163. Adding prompt positions cannot
change this sum, which is what makes the DPO margin a statement about responses.
A sequence with no response token, or a label outside the vocabulary, raises
ValueError before any arithmetic runs.
Trace one DPO pair
Direct Preference Optimization (DPO) learns from a chosen/rejected pair while a
frozen reference policy supplies a relative baseline. For one pair, the loss is
-log sigmoid(beta * ((log p(chosen)-log p(rejected)) - reference_margin)).
Take the first pair of compare_dpo_reference.py: policy log-probabilities
−0.7 (chosen) and −1.1 (rejected), reference −0.8 and −1.0, beta = 0.2. The
policy margin is 0.4, the reference margin 0.2, so the argument is
0.2 × (0.4 − 0.2) = 0.04 and the loss -log sigmoid(0.04) = 0.6733. With
sigmoid(0.04) = 0.5100, the gradient on policy_chosen is
-beta × (1 − 0.5100) = -0.0980, or -0.0245 once the script averages over
its four pairs; the rejected log-probability receives +0.0245. When the two
margins are equal the argument is zero and the loss is log 2 = 0.6931: the
pair has taught nothing yet. dpo_loss calls .detach() on both reference
tensors, so the script can mark them requires_grad and prove that their
.grad stays None.
Compare with TRL, then read the bandit
The lesson’s finite-action bridge makes the other pieces visible: a Bradley–Terry reward loss, categorical KL to the reference and PPO’s clipped surrogate. The example is a contextual bandit, not token-by-token production RLHF.
.venv/bin/python examples/run_alignment.py --dpo-steps 80 --ppo-steps 60
.venv/bin/python examples/compare_dpo_reference.pyThe report compares DPO with a reference implementation from pinned TRL 0.26.1:
loss and policy gradients match to zero at the recorded precision. The script
calls the installed DPOTrainer.dpo_loss with loss_type="sigmoid" on the same
four pairs; both sides print losses 0.6733, 0.7032, 0.8429, 0.6163 and
max_loss_absolute_difference and max_policy_gradient_absolute_difference
are 0.0. The PPO demonstration plants a shortcut in the reward model: naive PPO
raises proxy reward but lowers true quality and raises hack rate; KL-controlled
PPO improves the true quality on this tiny action table. This is a teaching
result, not a general safety claim.
The bandit has four Nordlicht contexts (billing, account access, export, a
secret request) with four actions each, always in the order correct, partial,
useless, harmful shortcut. The reference policy is the logit row
[1.4, 0.5, -0.6, -1.0], probabilities 0.6125, 0.2490, 0.0829, 0.0556, a
true quality of 0.713387 and a hack rate of 0.0556. _dpo_policy starts
from that row and trains the pairs (0,1), (0,2) and (0,3) with Adam for
the requested updates:
| DPO run (80 updates) | True quality | Hack rate | KL from reference |
|---|---|---|---|
beta = 0.05 |
0.9998 | 4.0e-5 | 0.4878 |
beta = 0.2 |
0.9995 | 9.5e-5 | 0.4850 |
beta = 1.0 |
0.9903 | 1.9e-3 | 0.4262 |
beta = 0.2, only pair (0,2) |
0.9891 | 2.9e-3 | 0.4188 |
A larger beta keeps the policy closer to the reference and leaves more mass on
the shortcut; dropping the (0,3) pair, the one that names the shortcut as
rejected, does the same. Every run still picks the correct action greedily in all
four contexts. Compare the PPO rows in A04 before concluding
that preference data alone is safe.
Predict → Trace → Build → Break → Measure → Explain
- Predict: Write down the loss and the sign of the
policy_rejectedgradient for the pair above before running anything. - Trace: Run
compare_dpo_reference.pyand matchours_losses[0]andours_policy_chosen_gradients[0]to the hand trace. - Build: Reproduce the −0.9163 sum with
response_log_probs(torch.log(probabilities)[None], labels[None]). - Break:
compare_dpo_reference.pyimportsdpo_loss, so either remove both.detach()calls inalignment.pytemporarily and revert afterwards, or monkeypatchllm_course.alignment.dpo_losswith a detach-free copy before calling itsmain();reference_inputs_frozenturns false because the reference tensors now carry a gradient that any optimizer holding them would apply. - Measure: Run
run_alignment.py --dpo-steps 20 --ppo-steps 60and record how far thebeta = 0.2true quality is from 0.9995 after a quarter of the updates. - Explain: State why a DPO policy with
hack_ratebelow 1e-4 on this table is not evidence about a language model’s escalation behaviour.
Independent exercise
If the chosen and rejected policy margins equal the reference margin, what is the DPO loss? Why must reference tensors be detached?
Hint
Set the argument of sigmoid to zero and evaluate -log sigmoid(0). Then
ask which tensors would receive a gradient if the reference were part of the
graph, and what the optimizer would do with it.
Reference answer
The loss is log(2). Detaching keeps the reference policy fixed so gradients
update only the policy being trained.
Checkpoint: compute one response log-probability sum, identify the clipped PPO ratio and explain proxy reward versus task success.
← A02 — Adaptation comparison · A04 — RLHF bridge → · Glossary