Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

A03 — DPO and the small RLHF bridge

← A02 — Adaptation comparison · A04 — RLHF bridge → · Glossary

Prerequisites: A01, C03. Allow one session. The code is src/llm_course/alignment.py; read response_log_probs and dpo_loss beside this page, and leave _fit_reward_model and _ppo_policy for A04. Every function validates its tensors and returns a new value; the reference policy is never modified.

Sum the response log-probabilities

response_log_probs sums only unmasked response tokens; prompt and padding labels are -100. Unlike A01’s sft_loss, it expects labels that are already shifted: labels[b, t] is the token that logits[b, t] should predict. Take one sequence of three positions over a three-token vocabulary whose softmax rows are [0.5, 0.3, 0.2], [0.25, 0.5, 0.25] and [0.1, 0.1, 0.8], with labels [-100, 1, 2]. The first position is prompt and contributes nothing; the sum is log 0.5 + log 0.8 = -0.6931 - 0.2231 = -0.9163. Adding prompt positions cannot change this sum, which is what makes the DPO margin a statement about responses. A sequence with no response token, or a label outside the vocabulary, raises ValueError before any arithmetic runs.

Trace one DPO pair

Direct Preference Optimization (DPO) learns from a chosen/rejected pair while a frozen reference policy supplies a relative baseline. For one pair, the loss is -log sigmoid(beta * ((log p(chosen)-log p(rejected)) - reference_margin)). Take the first pair of compare_dpo_reference.py: policy log-probabilities −0.7 (chosen) and −1.1 (rejected), reference −0.8 and −1.0, beta = 0.2. The policy margin is 0.4, the reference margin 0.2, so the argument is 0.2 × (0.4 − 0.2) = 0.04 and the loss -log sigmoid(0.04) = 0.6733. With sigmoid(0.04) = 0.5100, the gradient on policy_chosen is -beta × (1 − 0.5100) = -0.0980, or -0.0245 once the script averages over its four pairs; the rejected log-probability receives +0.0245. When the two margins are equal the argument is zero and the loss is log 2 = 0.6931: the pair has taught nothing yet. dpo_loss calls .detach() on both reference tensors, so the script can mark them requires_grad and prove that their .grad stays None.

Compare with TRL, then read the bandit

The lesson’s finite-action bridge makes the other pieces visible: a Bradley–Terry reward loss, categorical KL to the reference and PPO’s clipped surrogate. The example is a contextual bandit, not token-by-token production RLHF.

.venv/bin/python examples/run_alignment.py --dpo-steps 80 --ppo-steps 60
.venv/bin/python examples/compare_dpo_reference.py

The report compares DPO with a reference implementation from pinned TRL 0.26.1: loss and policy gradients match to zero at the recorded precision. The script calls the installed DPOTrainer.dpo_loss with loss_type="sigmoid" on the same four pairs; both sides print losses 0.6733, 0.7032, 0.8429, 0.6163 and max_loss_absolute_difference and max_policy_gradient_absolute_difference are 0.0. The PPO demonstration plants a shortcut in the reward model: naive PPO raises proxy reward but lowers true quality and raises hack rate; KL-controlled PPO improves the true quality on this tiny action table. This is a teaching result, not a general safety claim.

The bandit has four Nordlicht contexts (billing, account access, export, a secret request) with four actions each, always in the order correct, partial, useless, harmful shortcut. The reference policy is the logit row [1.4, 0.5, -0.6, -1.0], probabilities 0.6125, 0.2490, 0.0829, 0.0556, a true quality of 0.713387 and a hack rate of 0.0556. _dpo_policy starts from that row and trains the pairs (0,1), (0,2) and (0,3) with Adam for the requested updates:

DPO run (80 updates) True quality Hack rate KL from reference
beta = 0.05 0.9998 4.0e-5 0.4878
beta = 0.2 0.9995 9.5e-5 0.4850
beta = 1.0 0.9903 1.9e-3 0.4262
beta = 0.2, only pair (0,2) 0.9891 2.9e-3 0.4188

A larger beta keeps the policy closer to the reference and leaves more mass on the shortcut; dropping the (0,3) pair, the one that names the shortcut as rejected, does the same. Every run still picks the correct action greedily in all four contexts. Compare the PPO rows in A04 before concluding that preference data alone is safe.

Predict → Trace → Build → Break → Measure → Explain

  1. Predict: Write down the loss and the sign of the policy_rejected gradient for the pair above before running anything.
  2. Trace: Run compare_dpo_reference.py and match ours_losses[0] and ours_policy_chosen_gradients[0] to the hand trace.
  3. Build: Reproduce the −0.9163 sum with response_log_probs(torch.log(probabilities)[None], labels[None]).
  4. Break: compare_dpo_reference.py imports dpo_loss, so either remove both .detach() calls in alignment.py temporarily and revert afterwards, or monkeypatch llm_course.alignment.dpo_loss with a detach-free copy before calling its main(); reference_inputs_frozen turns false because the reference tensors now carry a gradient that any optimizer holding them would apply.
  5. Measure: Run run_alignment.py --dpo-steps 20 --ppo-steps 60 and record how far the beta = 0.2 true quality is from 0.9995 after a quarter of the updates.
  6. Explain: State why a DPO policy with hack_rate below 1e-4 on this table is not evidence about a language model’s escalation behaviour.

Independent exercise

If the chosen and rejected policy margins equal the reference margin, what is the DPO loss? Why must reference tensors be detached?

Hint

Set the argument of sigmoid to zero and evaluate -log sigmoid(0). Then ask which tensors would receive a gradient if the reference were part of the graph, and what the optimizer would do with it.

Reference answer

The loss is log(2). Detaching keeps the reference policy fixed so gradients update only the policy being trained.

Checkpoint: compute one response log-probability sum, identify the clipped PPO ratio and explain proxy reward versus task success.

← A02 — Adaptation comparison · A04 — RLHF bridge → · Glossary

Was this page helpful?