A01 — SFT and LoRA: teach a response, preserve a base
← C04 — Continued pretraining · A02 — Adaptation comparison → · Glossary
Prerequisites: the decoder, training, and the fictional company data/evaluation contracts. Run the explicit CPU mechanism before the optional pretrained lab. Both entry paths need no private data or teacher API.
Trace the response mask before training
An instruction contains a prompt and a desired response. The prompt conditions the
response but is not itself the answer we want to score. For token IDs
[11, 12 | 21, 22], construct unshifted labels [-100, -100, 21, 22].
logits[1] predicts 21 and logits[2] predicts 22. The loss function shifts once;
passing already shifted labels would silently teach the wrong transition.
Padding is also -100, while its attention mask is zero.
If the two correct-token probabilities are 0.5 and 0.25, response mean NLL is
(-log(0.5) - log(0.25)) / 2 = 1.0397. Adding ten prompt tokens must not dilute
that mean. Changing masked target content leaves this loss unchanged for fixed
logits; changing a prompt input can still change the model’s response logits.
The code tokenizes prompt and response separately to make the boundary explicit. It includes the assistant chat header in the prompt and the end-of-turn token in the response. A record that exceeds the context is rejected instead of truncating away a safety instruction or its answer.
Build the low-rank update before PEFT
Full SFT updates all weights. LoRA computes
y = Wx + (alpha / r) BAx, where A is [r, input] and B is [output, r].
The original W and bias are frozen. For a 512×512 layer with rank 4, full weight
training uses 262,144 parameters; A and B use 4×512 + 512×4 = 4,096, or 1.5625%.
This ratio excludes other layers, biases, optimizer state and activations.
Initialize A randomly and B to zero: the initial adapted output equals the base. On the first backward pass B can receive a gradient while A’s gradient is zero because B is zero. After B changes, A can learn too. A test that demands a nonzero A gradient on the very first step misunderstands the mechanism.
LoRALinear.merged() copies W + (alpha/r)BA into a fresh linear layer. Adapter
files contain a version and a hash of the exact base weights; another base with
the same dimensions is rejected. Core checkpoints use torch.load(weights_only=True);
the productive library path uses safetensors. Merge checks reject nonfinite weights.
Preserve the immutable base to roll back.
Run and measure
.venv/bin/pytest tests/test_post_training.py -q
.venv/bin/python examples/run_sft_lora.py --steps 120
.venv/bin/pip install -e '.[post-training]'
.venv/bin/python examples/run_open_model_lora.py --download --device cpu
.venv/bin/python examples/run_open_model_lora.py --device mps --output artifacts/open-model-lora-mpsThe first experiment pairs random MiniGPT full SFT and manual LoRA with the same
training/development examples, seed, optimizer, learning rate, steps and batch
size. The pretrained experiment uses the real Apache-2.0
SmolLM2-135M-Instruct model,
fixed at revision 12fd25f77366fa6b3b4b768ec3050bf629380bac. It downloads only
allowlisted configuration/tokenizer files and model.safetensors, loads with
trust_remote_code=False, and trains rank-4 q/v adapters in float32.
All methods consume training instructions and separately authored development
cases; none opens gold. The same evaluator checks actual generated predictions
before/after. Response NLL and exact/format/safety metrics answer different
questions. A lower teacher-forced NLL does not prove usable generated answers.
Reload and merged outputs are checked against the trained adapter. See the
adaptation comparison and docs/baselines/sft-lora-v1.md
for measured results, runtime, memory scope and limitations.
Predict → Trace → Build → Break → Measure → Explain
- Predict: Which token receives the first response loss? What is A’s first gradient?
- Trace: Print IDs, labels and attention masks for unequal-length examples.
- Build: Implement
response_batch,sft_lossandLoRALinearfrom the formulas. - Break: Unfreeze W, remove the label shift, or load onto a different base. Observe a test fail.
- Measure: Record train/dev NLL, format failures, general-control NLL, time and trainable count.
- Explain: Decide whether to keep the adapter. A regression is evidence for rollback.
Independent exercise
Use a 12-input, 8-output layer, rank 2 and alpha 6. Compute adapter count and scale. Then replace the target of a prompt token and a response token separately while holding logits fixed. Add a test proving which edit affects loss. Save, reload and merge after two optimizer steps; verify base gradients remain absent.
Hint
Count both rectangular factors. The response label at position t is predicted by logits at t−1. Do not move the mask independently of its target.
Reference answer
A has 24 and B has 16 parameters: 40 total versus 96 base weights; scale is 3. Masked prompt target edits leave loss unchanged. Response target edits usually change loss; choose unequal log-probabilities to make the assertion meaningful. A learns after the zero-initialized B has begun changing.
Checkpoint: explain loss alignment without a Trainer; demonstrate zero initial LoRA delta, frozen base, safe reload and merge; report a failed quality gate honestly. The mechanism follows LoRA; production wrappers are documented by PEFT.
← C04 — Continued pretraining · A02 — Adaptation comparison → · Glossary