---
title: "A03 — DPO and the small RLHF bridge"
sidebar:
  label: "A03 — DPO"
---

<span id="a03-dpo-and-the-small-rlhf-bridge" />


[← A02 — Adaptation comparison](/adaptation-comparison) · [A04 — RLHF bridge →](/rlhf-bridge) · [Glossary](/glossary)

Prerequisites: [A01](/sft-lora), [C03](/company-evaluation). Allow one session.
The code is `src/llm_course/alignment.py`; read `response_log_probs` and
`dpo_loss` beside this page, and leave `_fit_reward_model` and `_ppo_policy` for
[A04](/rlhf-bridge). Every function validates its tensors and returns a new
value; the reference policy is never modified.

## Sum the response log-probabilities [#sum-the-response-log-probabilities]

`response_log_probs` sums only unmasked response tokens; prompt and padding labels
are `-100`. Unlike A01's `sft_loss`, it expects labels that are already shifted:
`labels[b, t]` is the token that `logits[b, t]` should predict. Take one sequence
of three positions over a three-token vocabulary whose softmax rows are
`[0.5, 0.3, 0.2]`, `[0.25, 0.5, 0.25]` and `[0.1, 0.1, 0.8]`, with labels
`[-100, 1, 2]`. The first position is prompt and contributes nothing; the sum is
`log 0.5 + log 0.8 = -0.6931 - 0.2231 = -0.9163`. Adding prompt positions cannot
change this sum, which is what makes the DPO margin a statement about responses.
A sequence with no response token, or a label outside the vocabulary, raises
`ValueError` before any arithmetic runs.

## Trace one DPO pair [#trace-one-dpo-pair]

Direct Preference Optimization (DPO) learns from a chosen/rejected pair while a
frozen reference policy supplies a relative baseline. For one pair, the loss is
`-log sigmoid(beta * ((log p(chosen)-log p(rejected)) - reference_margin))`.
Take the first pair of `compare_dpo_reference.py`: policy log-probabilities
−0.7 (chosen) and −1.1 (rejected), reference −0.8 and −1.0, `beta = 0.2`. The
policy margin is `0.4`, the reference margin `0.2`, so the argument is
`0.2 × (0.4 − 0.2) = 0.04` and the loss `-log sigmoid(0.04) = 0.6733`. With
`sigmoid(0.04) = 0.5100`, the gradient on `policy_chosen` is
`-beta × (1 − 0.5100) = -0.0980`, or `-0.0245` once the script averages over
its four pairs; the rejected log-probability receives `+0.0245`. When the two
margins are equal the argument is zero and the loss is `log 2 = 0.6931`: the
pair has taught nothing yet. `dpo_loss` calls `.detach()` on both reference
tensors, so the script can mark them `requires_grad` and prove that their
`.grad` stays `None`.

## Compare with TRL, then read the bandit [#compare-with-trl-then-read-the-bandit]

The lesson's finite-action bridge makes the other pieces visible: a Bradley–Terry
reward loss, categorical KL to the reference and PPO's clipped surrogate. The
example is a contextual bandit, not token-by-token production RLHF.

```bash
.venv/bin/python examples/run_alignment.py --dpo-steps 80 --ppo-steps 60
.venv/bin/python examples/compare_dpo_reference.py
```

The report compares DPO with a reference implementation from pinned TRL 0.26.1:
loss and policy gradients match to zero at the recorded precision. The script
calls the installed `DPOTrainer.dpo_loss` with `loss_type="sigmoid"` on the same
four pairs; both sides print losses `0.6733, 0.7032, 0.8429, 0.6163` and
`max_loss_absolute_difference` and `max_policy_gradient_absolute_difference`
are `0.0`. The PPO demonstration plants a shortcut in the reward model: naive PPO
raises proxy reward but lowers true quality and raises hack rate; KL-controlled
PPO improves the true quality on this tiny action table. This is a teaching
result, not a general safety claim.

The bandit has four Nordlicht contexts (billing, account access, export, a
secret request) with four actions each, always in the order correct, partial,
useless, harmful shortcut. The reference policy is the logit row
`[1.4, 0.5, -0.6, -1.0]`, probabilities `0.6125, 0.2490, 0.0829, 0.0556`, a
true quality of `0.713387` and a hack rate of `0.0556`. `_dpo_policy` starts
from that row and trains the pairs `(0,1)`, `(0,2)` and `(0,3)` with Adam for
the requested updates:

| DPO run (80 updates) | True quality | Hack rate | KL from reference |
| --- | ---: | ---: | ---: |
| `beta = 0.05` | 0.9998 | 4.0e-5 | 0.4878 |
| `beta = 0.2` | 0.9995 | 9.5e-5 | 0.4850 |
| `beta = 1.0` | 0.9903 | 1.9e-3 | 0.4262 |
| `beta = 0.2`, only pair `(0,2)` | 0.9891 | 2.9e-3 | 0.4188 |

A larger `beta` keeps the policy closer to the reference and leaves more mass on
the shortcut; dropping the `(0,3)` pair, the one that names the shortcut as
rejected, does the same. Every run still picks the correct action greedily in all
four contexts. Compare the PPO rows in [A04](/rlhf-bridge) before concluding
that preference data alone is safe.

## Predict → Trace → Build → Break → Measure → Explain [#predict-trace-build-break-measure-explain]

1. **Predict:** Write down the loss and the sign of the `policy_rejected` gradient for the pair above before running anything.
2. **Trace:** Run `compare_dpo_reference.py` and match `ours_losses[0]` and `ours_policy_chosen_gradients[0]` to the hand trace.
3. **Build:** Reproduce the −0.9163 sum with `response_log_probs(torch.log(probabilities)[None], labels[None])`.
4. **Break:** `compare_dpo_reference.py` imports `dpo_loss`, so either remove both `.detach()` calls in `alignment.py` temporarily and revert afterwards, or monkeypatch `llm_course.alignment.dpo_loss` with a detach-free copy before calling its `main()`; `reference_inputs_frozen` turns false because the reference tensors now carry a gradient that any optimizer holding them would apply.
5. **Measure:** Run `run_alignment.py --dpo-steps 20 --ppo-steps 60` and record how far the `beta = 0.2` true quality is from 0.9995 after a quarter of the updates.
6. **Explain:** State why a DPO policy with `hack_rate` below 1e-4 on this table is not evidence about a language model's escalation behaviour.

### Independent exercise [#independent-exercise]

If the chosen and rejected policy margins equal the reference margin, what is the
DPO loss? Why must reference tensors be detached?

<details>
<summary>Hint</summary>

Set the argument of `sigmoid` to zero and evaluate `-log sigmoid(0)`. Then
ask which tensors would receive a gradient if the reference were part of the
graph, and what the optimizer would do with it.


</details>

<details>
<summary>Reference answer</summary>

The loss is `log(2)`. Detaching keeps the reference policy fixed so gradients
update only the policy being trained.


</details>

Checkpoint: compute one response log-probability sum, identify the clipped PPO
ratio and explain proxy reward versus task success.

[← A02 — Adaptation comparison](/adaptation-comparison) · [A04 — RLHF bridge →](/rlhf-bridge) · [Glossary](/glossary)
