Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

T06 — Core gate: code, diagnose, explain

← T05 — Mini-GPT · I01 — Decoding, sampling & KV cache → · Glossary

Take this after the decoder lesson, the data pipeline and attention. Allow 60–90 minutes. The goal is to demonstrate an independent implementation, not just run the supplied solution. Keep your work in artifacts/mini-gpt/; no generated corpus or weights belong in Git.

1. Code a pre-norm residual block (4 points)

Save this incomplete scaffold as artifacts/mini-gpt/exercise.py. Replace the NotImplementedError with your implementation. Use the provided block’s modules but do not call block.forward inside your function. Keep the shape [B,T,D], apply both pre-normalizations, both residual paths and padding isolation. Before coding, predict the result if both update branches always return zero.

import torch
from llm_course.attention import causal_mask
from llm_course.mini_gpt import DecoderBlock, GPTConfig


def learner_forward(block, inputs, allowed, valid):
    # TODO: normalize -> attention -> dropout -> residual -> zero padding.
    # TODO: normalize -> FFN -> dropout -> residual -> zero padding.
    raise NotImplementedError("implement both residual branches")


torch.set_num_threads(1)
config = GPTConfig(vocab_size=9, block_size=3, embedding_dim=4, heads=2, layers=1)
block = DecoderBlock(config, index=0).eval()
valid = torch.tensor([[[1], [1], [0]]], dtype=torch.bool)
inputs = torch.tensor([[[1., 2., 3., 4.], [4., 3., 2., 1.], [0., 0., 0., 0.]]])
allowed = causal_mask(3)[None, None] & valid.squeeze(-1)[:, None, None, :]
allowed |= ~valid[:, None] & torch.eye(3, dtype=torch.bool)[None, None]
actual = learner_forward(block, inputs, allowed, valid)
expected = block(inputs, allowed, valid)
assert actual.shape == (1, 3, 4)
assert torch.count_nonzero(actual[:, 2]) == 0
assert torch.isfinite(actual).all()
torch.testing.assert_close(actual, expected)
print("residual block matches; now add the suffix-invariance test")
python artifacts/mini-gpt/exercise.py
python -m pytest tests/test_mini_gpt.py -q

Add your own test: change the final valid token of a three-token input and verify earlier outputs stay unchanged. Use all-valid masks for that test and eval() to remove dropout noise. Explain why merely checking the output shape would fail to detect attention over the wrong axis.

Award one point each for pre-norm attention, the first residual, pre-norm FFN with the second residual, and an independent causality/padding assertion. The initial NotImplementedError is expected; removing it without both computations earns no credit. Record your code and failing-then-passing output before revealing answers.

2. Diagnose deliberate defects (3 points)

For each defect, write a counterexample, predict an observable symptom and propose a correction. One point each requires all three, not just naming the defect.

Defect in a local copy Evidence to predict
Dataset returns labels = input_ids.clone() Very low loss can measure copying the visible token
Replace allowed with all-True and set block.attention.causal=False Earlier logits depend on a changed future suffix
Average valid losses together with zeros for padded positions Appending padding makes the reported mean artificially smaller

Numerical check for the third defect: two valid losses 2 and 4 with two padded slots give the wrong value (2+4+0+0)/4=1.5, while the correct value is 3. Why does an entirely padded row require special treatment before softmax as well as after the loss? Restore every defect after collecting its evidence.

3. Explain and compare (3 points)

Give a short written or spoken explanation with your own numbers:

  • Follow IDs [2,3] through embeddings [2,3,4], two heads [2,2,3,2], score matrices [2,2,3,3], FFN [2,3,16], and logits [2,3,5]. Distinguish feature mixing from position mixing, and explain why labels are shifted only once.
  • Explain why reproducible initialization must produce different blocks, why sampling temporarily uses eval(), and why saving only weights cannot resume AdamW training exactly. Name the information E01 must also save.
  • Run the three-seed command from the lesson. Report each paired no-norm minus norm validation NLL difference, its mean and sample standard deviation. Explain why two validation documents and three seeds cannot establish general superiority.

Each complete explanation earns one point. Pass: at least 8/10, including all 4 coding points and all 3 defect points, plus a successful supplied test suite, tiny-batch NLL <0.1 and exact reload evidence. The supplied tests include these last two checks; preserve their output. If the gate fails, revisit the relevant prerequisite and repeat with a fresh explanation, not a copied answer.

Hints — open only after an attempt

Hint 1: separate the two residual paths

Keep inputs unchanged while computing the normalized attention update. Call attention with mask=allowed; it returns (output, weights). Only the first is added to the residual. The second normalized branch uses the updated hidden state, not the original input. valid broadcasts [B,T,1] across features.

Hint 2: distinguish the two masks

The attention mask determines which keys may contribute before softmax. The loss mask decides which target predictions count in the objective. One cannot replace the other. An attention row of only -inf produces an undefined distribution even when its target is later ignored.

Reference answers — compare after recording your evidence

Reference block implementation
def learner_forward(block, inputs, allowed, valid):
    attended, _ = block.attention(block.norm_attention(inputs), mask=allowed)
    hidden = (inputs + block.dropout(attended)) * valid
    update = block.ffn(block.norm_ffn(hidden))
    return (hidden + block.dropout(update)) * valid

With zero branches, valid output equals input; padded output is zero. GELU and the FFN act on features independently at each time position; masked attention permits information from earlier positions. The model’s distinct block seeds prevent identical initial projection weights while preserving whole-run reproducibility.

Reference diagnosis and explanation
  1. Visible input c with target c rewards copying; use next-token a instead. Fix the dataset to provide tokens[:-1] and tokens[1:] once.
  2. For [1,2,3] versus [1,2,8], a correct decoder has identical first two logits. Restore the causal triangle combined with valid keys. Both safeguards must be disabled to demonstrate the deliberate defect in this implementation.
  3. Sum NLL only for labels != -100, divide by their count. An all-ignored loss is differentiable zero. Give padded queries a safe dummy diagonal before softmax, then zero their outputs so dummy values cannot influence real tokens.

Sampling uses evaluation mode so dropout does not interfere with the local sample seed, then restores each module’s mode. Exact training resume also requires optimizer moments, scheduler/scaler state if present, step/batch position, RNG state, configuration and a data identity check. The norm-ablation differences are 2.389516, 1.105527 and 1.325092; mean 1.606712, sample standard deviation 0.686760. This result describes the fixed toy validation probe. It is not a safety, factuality or broad language benchmark.

← T05 — Mini-GPT · I01 — Decoding, sampling & KV cache → · Glossary

Was this page helpful?