C04 — Continued pretraining without losing the baseline
← C03 — Protected evaluation · A01 — SFT & LoRA → · Glossary
Prerequisites: C02, E01. Allow one session.
The code is src/llm_course/continued_pretraining.py; read document_batches,
adaptation_decision and run_continued_pretraining beside this page.
What DAPT changes and what it cannot
Continued pretraining (also called domain-adaptive pretraining, DAPT) continues
the language-model objective on approved domain text. It teaches vocabulary and
style distributions; it does not directly teach a JSON response contract.
document_batches encodes each document with a zero-merge byte tokenizer
(vocabulary 260, so nothing is fitted to held-out text) and cuts it into windows
of 48 next-byte targets that never cross a document boundary. The model is T05’s
MiniGPT at one layer, width 32 and four heads: 30,816 parameters, the 123,264
bytes of float32 storage the report lists as parameter_bytes.
The example splits whole documents before any window is cut: every fourth
tiny_lm sentence is held out as a general control (12 train, 4 held out) and
every fourth company_documents() record, at another offset, is a domain
held-out (9 train, 3 held out: access-secret-v1, escalation-v1 and
support-format-v1). Instruction
and gold cases are never read. run_continued_pretraining normalises both
training corpora against both held-outs and raises ValueError on any overlap.
Trace the decision gate
adaptation_decision reads four NLLs and two preregistered thresholds:
gain = domain_before − domain_after must reach minimum_gain = 0.1 nats, and
regression = general_after − general_before must stay within
maximum_regression = 0.15. Both gates must pass; a lower domain NLL cannot
hide forgetting. For seed 7 the domain held-out moves from 3.7068 to 3.2961, a
gain of 0.4107; the general control moves from 2.5574 to 2.6662, a regression
of 0.1087. Both gates pass and the decision is keep_for_further_review,
with the scope string raw-text validation gates only; no instruction or production-quality claim. C01’s choose_strategy
applies the same two thresholds when it is given this evidence.
| Seed | Domain NLL before → after | Gain | General NLL before → after | Regression |
|---|---|---|---|---|
| 7 | 3.7068 → 3.2961 | 0.4107 | 2.5574 → 2.6662 | 0.1087 |
| 19 | 3.6247 → 3.2551 | 0.3696 | 2.4640 → 2.5500 | 0.0860 |
| 43 | 3.5880 → 3.2296 | 0.3584 | 2.5191 → 2.5515 | 0.0323 |
Run, resume, roll back
run_continued_pretraining.py uses the approved company snapshot, byte-level
fallback tokenization, disjoint domain held-outs and two general controls. It
trains a tiny CPU model, saves an immutable base, resumes from an interrupted
checkpoint, and can roll back by reloading the base.
.venv/bin/python examples/run_continued_pretraining.py --seeds 7 19 43The measured seeds improved domain NLL by 0.359–0.411 nats while general-control NLL regressed by 0.032–0.109 nats. The report therefore keeps the candidate for further review rather than promoting it. Exact resume and rollback identity are strong engineering checks; they are not a quality claim.
Each seed writes artifacts/continued-pretraining/<seed>/report.json after 80
base steps and 40 adaptation steps at learning rate 0.002, 2,940 adaptation
tokens. The resume check trains a second copy for 20 steps, saves
interrupted.pt, loads it into a fresh trainer and finishes the remaining 20:
parameter_max_difference is 0.0 and optimizer_exact is true for every
seed. The rollback check reloads general-reference.pt and compares logits on
one held-out batch: logits_exact is true. Because the decision is keep,
selected_checkpoint is adapted-model.pt; a rollback decision would point
it at the reference instead. A 60-second budget per seed stops training at an
optimizer boundary and saves budget-stop.pt.
Predict → Trace → Build → Break → Measure → Explain
- Predict: Which of gain 0.05 with regression 0.02, or gain 0.5 with regression 0.2, does
adaptation_decisionroll back? Both? Write it down. - Trace: Run the command above and read
decision,resumeandrollbackfor seed 7 inartifacts/continued-pretraining/7/report.json. - Build: Call
adaptation_decision(3.7068, 3.2961, 2.5574, 2.6662)and reproduce the seed-7 row; then passmaximum_regression=0.1and readreasons. - Break: Append one
domain_evaldocument todomain_trainin a copy of the example and observe theValueErrorbefore any step runs. - Measure: Run
--seeds 7 --adapt-steps 80and record gain and regression next to the 40-step row; state which gate moves. - Explain: Why does a passing gate still not promote the checkpoint? Name the evaluation that has to run next.
Independent exercise
Why are a domain gain and a general regression reported separately? Which change would make this experiment invalid: adding a general held-out sentence to the training text, or changing only the learning-rate seed?
Hint
Look at what run_continued_pretraining checks before training and at what
the report records for each seed. One change is caught by code; the other is
caught only by the person keeping the record.
Reference answer
Separate scores expose forgetting. Adding held-out text contaminates the test; a preregistered seed change is a new run that must be recorded, not hidden.
Checkpoint: state a promotion threshold, name the rollback artifact and explain why DAPT must be followed by instruction evaluation.
← C03 — Protected evaluation · A01 — SFT & LoRA → · Glossary