---
title: "T06 — Core gate: code, diagnose, explain"
sidebar:
  label: "T06 — Core gate"
---

<span id="t06-core-gate-code-diagnose-explain" />


[← T05 — Mini-GPT](/mini-gpt) · [I01 — Decoding, sampling & KV cache →](/inference-decoding) · [Glossary](/glossary)

Take this after [the decoder lesson](/mini-gpt), [the data pipeline](/data-pipeline)
and [attention](/attention). Allow 60–90 minutes. The goal is to demonstrate an
independent implementation, not just run the supplied solution. Keep your work in
`artifacts/mini-gpt/`; no generated corpus or weights belong in Git.

## 1. Code a pre-norm residual block (4 points) [#1-code-a-pre-norm-residual-block-4-points]

Save this incomplete scaffold as `artifacts/mini-gpt/exercise.py`. Replace the
`NotImplementedError` with your implementation. Use the provided block's modules
but do not call `block.forward` inside your function. Keep the shape `[B,T,D]`,
apply both pre-normalizations, both residual paths and padding isolation.
Before coding, predict the result if both update branches always return zero.

```python
import torch
from llm_course.attention import causal_mask
from llm_course.mini_gpt import DecoderBlock, GPTConfig


def learner_forward(block, inputs, allowed, valid):
    # TODO: normalize -> attention -> dropout -> residual -> zero padding.
    # TODO: normalize -> FFN -> dropout -> residual -> zero padding.
    raise NotImplementedError("implement both residual branches")


torch.set_num_threads(1)
config = GPTConfig(vocab_size=9, block_size=3, embedding_dim=4, heads=2, layers=1)
block = DecoderBlock(config, index=0).eval()
valid = torch.tensor([[[1], [1], [0]]], dtype=torch.bool)
inputs = torch.tensor([[[1., 2., 3., 4.], [4., 3., 2., 1.], [0., 0., 0., 0.]]])
allowed = causal_mask(3)[None, None] & valid.squeeze(-1)[:, None, None, :]
allowed |= ~valid[:, None] & torch.eye(3, dtype=torch.bool)[None, None]
actual = learner_forward(block, inputs, allowed, valid)
expected = block(inputs, allowed, valid)
assert actual.shape == (1, 3, 4)
assert torch.count_nonzero(actual[:, 2]) == 0
assert torch.isfinite(actual).all()
torch.testing.assert_close(actual, expected)
print("residual block matches; now add the suffix-invariance test")
```

```bash
python artifacts/mini-gpt/exercise.py
python -m pytest tests/test_mini_gpt.py -q
```

Add your own test: change the final **valid** token of a three-token input and
verify earlier outputs stay unchanged. Use all-valid masks for that test and
`eval()` to remove dropout noise. Explain why merely checking the output shape
would fail to detect attention over the wrong axis.

Award one point each for pre-norm attention, the first residual, pre-norm FFN with
the second residual, and an independent causality/padding assertion. The initial
`NotImplementedError` is expected; removing it without both computations earns no
credit. Record your code and failing-then-passing output before revealing answers.

## 2. Diagnose deliberate defects (3 points) [#2-diagnose-deliberate-defects-3-points]

For each defect, write a counterexample, predict an observable symptom and propose
a correction. One point each requires all three, not just naming the defect.

| Defect in a local copy | Evidence to predict |
| --- | --- |
| Dataset returns `labels = input_ids.clone()` | Very low loss can measure copying the visible token |
| Replace `allowed` with all-True **and** set `block.attention.causal=False` | Earlier logits depend on a changed future suffix |
| Average valid losses together with zeros for padded positions | Appending padding makes the reported mean artificially smaller |

Numerical check for the third defect: two valid losses 2 and 4 with two padded
slots give the wrong value `(2+4+0+0)/4=1.5`, while the correct value is 3.
Why does an entirely padded row require special treatment before softmax as well
as after the loss? Restore every defect after collecting its evidence.

## 3. Explain and compare (3 points) [#3-explain-and-compare-3-points]

Give a short written or spoken explanation with your own numbers:

- Follow IDs `[2,3]` through embeddings `[2,3,4]`, two heads `[2,2,3,2]`, score
  matrices `[2,2,3,3]`, FFN `[2,3,16]`, and logits `[2,3,5]`. Distinguish feature
  mixing from position mixing, and explain why labels are shifted only once.
- Explain why reproducible initialization must produce different blocks, why
  sampling temporarily uses `eval()`, and why saving only weights cannot resume
  AdamW training exactly. Name the information E01 must also save.
- Run the three-seed command from the lesson. Report each paired no-norm minus norm
  validation NLL difference, its mean and sample standard deviation. Explain why
  two validation documents and three seeds cannot establish general superiority.

Each complete explanation earns one point. **Pass:** at least 8/10, including all
4 coding points and all 3 defect points, plus a successful supplied test suite,
tiny-batch NLL &lt;0.1 and exact reload evidence. The supplied tests include these last
two checks; preserve their output. If the gate fails, revisit the relevant
prerequisite and repeat with a fresh explanation, not a copied answer.

## Hints — open only after an attempt [#hints-open-only-after-an-attempt]

<details>
<summary>Hint 1: separate the two residual paths</summary>

Keep `inputs` unchanged while computing the normalized attention update.
Call attention with `mask=allowed`; it returns `(output, weights)`. Only the first
is added to the residual. The second normalized branch uses the updated hidden
state, not the original input. `valid` broadcasts `[B,T,1]` across features.

</details>

<details>
<summary>Hint 2: distinguish the two masks</summary>

The attention mask determines which keys may contribute before softmax. The loss
mask decides which target predictions count in the objective. One cannot replace
the other. An attention row of only `-inf` produces an undefined distribution even
when its target is later ignored.

</details>

## Reference answers — compare after recording your evidence [#reference-answers-compare-after-recording-your-evidence]

<details>
<summary>Reference block implementation</summary>

```python
def learner_forward(block, inputs, allowed, valid):
    attended, _ = block.attention(block.norm_attention(inputs), mask=allowed)
    hidden = (inputs + block.dropout(attended)) * valid
    update = block.ffn(block.norm_ffn(hidden))
    return (hidden + block.dropout(update)) * valid
```

With zero branches, valid output equals input; padded output is zero. GELU and the
FFN act on features independently at each time position; masked attention permits
information from earlier positions. The model's distinct block seeds prevent
identical initial projection weights while preserving whole-run reproducibility.

</details>

<details>
<summary>Reference diagnosis and explanation</summary>

1. Visible input `c` with target `c` rewards copying; use next-token `a` instead.
   Fix the dataset to provide `tokens[:-1]` and `tokens[1:]` once.
2. For `[1,2,3]` versus `[1,2,8]`, a correct decoder has identical first two logits.
   Restore the causal triangle combined with valid keys. Both safeguards must be
   disabled to demonstrate the deliberate defect in this implementation.
3. Sum NLL only for `labels != -100`, divide by their count. An all-ignored loss is
   differentiable zero. Give padded queries a safe dummy diagonal before softmax,
   then zero their outputs so dummy values cannot influence real tokens.

Sampling uses evaluation mode so dropout does not interfere with the local sample
seed, then restores each module's mode. Exact training resume also requires
optimizer moments, scheduler/scaler state if present, step/batch position, RNG
state, configuration and a data identity check. The norm-ablation differences are
2.389516, 1.105527 and 1.325092; mean 1.606712, sample standard deviation 0.686760.
This result describes the fixed toy validation probe. It is not a safety,
factuality or broad language benchmark.

</details>

[← T05 — Mini-GPT](/mini-gpt) · [I01 — Decoding, sampling & KV cache →](/inference-decoding) · [Glossary](/glossary)
