T06 — Core gate: code, diagnose, explain
← T05 — Mini-GPT · I01 — Decoding, sampling & KV cache → · Glossary
Take this after the decoder lesson, the data pipeline
and attention. Allow 60–90 minutes. The goal is to demonstrate an
independent implementation, not just run the supplied solution. Keep your work in
artifacts/mini-gpt/; no generated corpus or weights belong in Git.
1. Code a pre-norm residual block (4 points)
Save this incomplete scaffold as artifacts/mini-gpt/exercise.py. Replace the
NotImplementedError with your implementation. Use the provided block’s modules
but do not call block.forward inside your function. Keep the shape [B,T,D],
apply both pre-normalizations, both residual paths and padding isolation.
Before coding, predict the result if both update branches always return zero.
import torch
from llm_course.attention import causal_mask
from llm_course.mini_gpt import DecoderBlock, GPTConfig
def learner_forward(block, inputs, allowed, valid):
# TODO: normalize -> attention -> dropout -> residual -> zero padding.
# TODO: normalize -> FFN -> dropout -> residual -> zero padding.
raise NotImplementedError("implement both residual branches")
torch.set_num_threads(1)
config = GPTConfig(vocab_size=9, block_size=3, embedding_dim=4, heads=2, layers=1)
block = DecoderBlock(config, index=0).eval()
valid = torch.tensor([[[1], [1], [0]]], dtype=torch.bool)
inputs = torch.tensor([[[1., 2., 3., 4.], [4., 3., 2., 1.], [0., 0., 0., 0.]]])
allowed = causal_mask(3)[None, None] & valid.squeeze(-1)[:, None, None, :]
allowed |= ~valid[:, None] & torch.eye(3, dtype=torch.bool)[None, None]
actual = learner_forward(block, inputs, allowed, valid)
expected = block(inputs, allowed, valid)
assert actual.shape == (1, 3, 4)
assert torch.count_nonzero(actual[:, 2]) == 0
assert torch.isfinite(actual).all()
torch.testing.assert_close(actual, expected)
print("residual block matches; now add the suffix-invariance test")python artifacts/mini-gpt/exercise.py
python -m pytest tests/test_mini_gpt.py -qAdd your own test: change the final valid token of a three-token input and
verify earlier outputs stay unchanged. Use all-valid masks for that test and
eval() to remove dropout noise. Explain why merely checking the output shape
would fail to detect attention over the wrong axis.
Award one point each for pre-norm attention, the first residual, pre-norm FFN with
the second residual, and an independent causality/padding assertion. The initial
NotImplementedError is expected; removing it without both computations earns no
credit. Record your code and failing-then-passing output before revealing answers.
2. Diagnose deliberate defects (3 points)
For each defect, write a counterexample, predict an observable symptom and propose a correction. One point each requires all three, not just naming the defect.
| Defect in a local copy | Evidence to predict |
|---|---|
Dataset returns labels = input_ids.clone() |
Very low loss can measure copying the visible token |
Replace allowed with all-True and set block.attention.causal=False |
Earlier logits depend on a changed future suffix |
| Average valid losses together with zeros for padded positions | Appending padding makes the reported mean artificially smaller |
Numerical check for the third defect: two valid losses 2 and 4 with two padded
slots give the wrong value (2+4+0+0)/4=1.5, while the correct value is 3.
Why does an entirely padded row require special treatment before softmax as well
as after the loss? Restore every defect after collecting its evidence.
3. Explain and compare (3 points)
Give a short written or spoken explanation with your own numbers:
- Follow IDs
[2,3]through embeddings[2,3,4], two heads[2,2,3,2], score matrices[2,2,3,3], FFN[2,3,16], and logits[2,3,5]. Distinguish feature mixing from position mixing, and explain why labels are shifted only once. - Explain why reproducible initialization must produce different blocks, why
sampling temporarily uses
eval(), and why saving only weights cannot resume AdamW training exactly. Name the information E01 must also save. - Run the three-seed command from the lesson. Report each paired no-norm minus norm validation NLL difference, its mean and sample standard deviation. Explain why two validation documents and three seeds cannot establish general superiority.
Each complete explanation earns one point. Pass: at least 8/10, including all 4 coding points and all 3 defect points, plus a successful supplied test suite, tiny-batch NLL <0.1 and exact reload evidence. The supplied tests include these last two checks; preserve their output. If the gate fails, revisit the relevant prerequisite and repeat with a fresh explanation, not a copied answer.
Hints — open only after an attempt
Hint 1: separate the two residual paths
Keep inputs unchanged while computing the normalized attention update.
Call attention with mask=allowed; it returns (output, weights). Only the first
is added to the residual. The second normalized branch uses the updated hidden
state, not the original input. valid broadcasts [B,T,1] across features.
Hint 2: distinguish the two masks
The attention mask determines which keys may contribute before softmax. The loss
mask decides which target predictions count in the objective. One cannot replace
the other. An attention row of only -inf produces an undefined distribution even
when its target is later ignored.
Reference answers — compare after recording your evidence
Reference block implementation
def learner_forward(block, inputs, allowed, valid):
attended, _ = block.attention(block.norm_attention(inputs), mask=allowed)
hidden = (inputs + block.dropout(attended)) * valid
update = block.ffn(block.norm_ffn(hidden))
return (hidden + block.dropout(update)) * validWith zero branches, valid output equals input; padded output is zero. GELU and the FFN act on features independently at each time position; masked attention permits information from earlier positions. The model’s distinct block seeds prevent identical initial projection weights while preserving whole-run reproducibility.
Reference diagnosis and explanation
- Visible input
cwith targetcrewards copying; use next-tokenainstead. Fix the dataset to providetokens[:-1]andtokens[1:]once. - For
[1,2,3]versus[1,2,8], a correct decoder has identical first two logits. Restore the causal triangle combined with valid keys. Both safeguards must be disabled to demonstrate the deliberate defect in this implementation. - Sum NLL only for
labels != -100, divide by their count. An all-ignored loss is differentiable zero. Give padded queries a safe dummy diagonal before softmax, then zero their outputs so dummy values cannot influence real tokens.
Sampling uses evaluation mode so dropout does not interfere with the local sample seed, then restores each module’s mode. Exact training resume also requires optimizer moments, scheduler/scaler state if present, step/batch position, RNG state, configuration and a data identity check. The norm-ablation differences are 2.389516, 1.105527 and 1.325092; mean 1.606712, sample standard deviation 0.686760. This result describes the fixed toy validation probe. It is not a safety, factuality or broad language benchmark.
← T05 — Mini-GPT · I01 — Decoding, sampling & KV cache → · Glossary