F06 — PyTorch translation and foundations checkpoint
← F05 — MLP, backprop & autograd · T01 — Bigram baseline → · Glossary
Learning outcome
You will translate the NumPy pipeline to PyTorch and pass the common gate by implementing, checking, breaking, and explaining it independently.
If you already write Python, plan about an hour up to the checkpoint and 1.5–2 hours for the checkpoint. If programming is new, plan about 1.5 hours up to the checkpoint (including the class primer) and 3–5 hours for the checkpoint, spread across two or three sessions. These times are guides, not deadlines.
One complete framework update
PyTorch stores tensors and remembers operations for the backward pass.
nn.Linear(3,2) holds a weight matrix and bias: three input values become two
scores. model.parameters() returns the adjustable tensors. SGD (stochastic
gradient descent) performs the familiar subtraction of learning rate times
gradient. The library manages bookkeeping; the mathematics stays the same.
Save this in artifacts/my-work/one_update.py and run
python artifacts/my-work/one_update.py:
import torch
from torch import nn
torch.manual_seed(7)
x = torch.tensor([[1., 0., 0.], [0., 1., 0.], [0., 0., 1.], [1., 1., 0.]])
y = torch.tensor([0, 1, 1, 0])
model = nn.Linear(3, 2)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
logits = model(x)
assert logits.shape == (4, 2)
loss = nn.functional.cross_entropy(logits, y)
optimizer.zero_grad()
loss.backward()
assert all(torch.isfinite(p.grad).all() for p in model.parameters())
optimizer.step()
after = nn.functional.cross_entropy(model(x), y)
print(round(loss.item(), 6), round(after.item(), 6))
assert after.item() < loss.item()It prints 0.798075 0.770493: the loss before and after the update, the second
smaller. torch.manual_seed(7) fixes the random starting weights of
nn.Linear, so every run prints the same two numbers — the seed from
Unit 0. .item() reads a scalar tensor as a Python number.
Pass logits to cross_entropy, not probabilities already transformed by
softmax: the function includes stable log-softmax. Integer targets select the correct column.
backward() computes gradients; only step() changes parameters. zero_grad()
prevents unintended addition of the previous step’s gradients. The one rule for
the order: clear the gradients any time before backward() (here right before it;
T01 clears them at the start of each loop step), and check they are finite after
backward() and before step().
Why custom models use a class
A class groups stored data and related functions (methods). PyTorch models are classes, so the next block uses one.
New to classes? A class in ten lines (about 10–15 minutes)
Save this as artifacts/my-work/counter.py and predict the three printed
lines before you run python artifacts/my-work/counter.py:
class Counter:
def __init__(self, start):
self.count = start
def add(self, amount):
self.count = self.count + amount
return self.count
clicks = Counter(10)
print(clicks.add(1))
print(clicks.add(5))
other = Counter(0)
print(other.count, clicks.count)It prints 11, 16 and 0 16. class Counter: describes a new kind of
object. Counter(10) makes one such object and runs __init__ with
start=10. Inside the class, self is the object being used:
self.count = start stores a value on it. add is a method, a function
that belongs to the class; clicks.add(1) passes clicks as self for you.
Each object keeps its own count, so other is unaffected by clicks.
class SmallMLP(nn.Module) below goes one step further: SmallMLP starts
with everything PyTorch’s nn.Module already has, and adds its own layers.
class SmallMLP(nn.Module) extends PyTorch’s module class. __init__ sets up
layers once, self means this particular model object, and super().__init__()
initializes PyTorch’s parameter bookkeeping. Storing layers as self.hidden
registers their parameters with the optimizer. forward describes the calculation;
mlp(x) invokes it through PyTorch’s bookkeeping. Append this below the earlier example:
class SmallMLP(nn.Module):
def __init__(self):
super().__init__()
self.hidden = nn.Linear(3, 4)
self.output = nn.Linear(4, 2)
def forward(self, inputs):
hidden = torch.tanh(self.hidden(inputs))
return self.output(hidden)
mlp = SmallMLP()
assert mlp(x).shape == (4, 2)This is the familiar MLP: 3 → 4 → 2 with tanh between layers. The Transformer
uses the same class structure with a longer forward calculation.
Translate concepts, not just syntax
| From scratch | PyTorch |
|---|---|
| parameter arrays | torch.nn.Parameter inside a module |
| manual forward equations | module(batch) |
| manual reverse pass | loss.backward() |
| subtract learning-rate times gradient | optimizer.step() |
| reset stored gradients | optimizer.zero_grad() |
For a batch [4,3] and Linear(3,2), logits must be [4,2]; four integer
targets must be [4]. Predict these shapes before running anything.
Independent checkpoint
You write the gate yourself, but not from an empty file. Copy the skeleton into your work folder (with your file manager, or in the terminal from the course root) and run it:
cp examples/checkpoint_skeleton.py artifacts/my-work/checkpoint.py
python artifacts/my-work/checkpoint.pyOn Windows PowerShell, cp works too. The first run stops at the first
unfinished step:
Not finished yet. Step 1: write softmax(logits) (F03).
Everything before this step passed its checks.Fill in one function at a time, top to bottom, and rerun after each; the message
moves on to the next step. Each stub’s docstring says what to return and which
page taught it. Everything below the line # ---- Provided checks is given: the
data, the fixed starting weights, the checks every step must pass, and the
report. Do not import llm_course; plain NumPy and PyTorch only.
| Step | You write | Taught in |
|---|---|---|
| 1 | softmax(logits): stable, one row at a time |
F03 |
| 2 | cross_entropy(logits, targets): mean over the rows |
F03 |
| 3 | mlp_forward_backward(X, targets, W1, b1, W2, b2): forward pass, then every gradient |
F05 |
| 4 | gradient_check(...): dW1[0, 0] against a central difference |
F05 |
| 5 | train_neuron(x, y, update_sign, learning_rate, steps): -1 is healthy, +1 the broken copy |
F04 |
| 6 | torch_mlp_step(x, y): one SGD step of SmallMLP |
this page |
| last | DIAGNOSIS: two sentences |
— |
Step 3 needs one-hot targets and step 4 must nudge a copy of W1, never W1
itself: both are in F05’s gate toolkit.
Step 5’s data and settings come in as arguments. Step 6 is the one_update.py step with SmallMLP() in
place of nn.Linear(3, 2), the same x and y, learning rate 0.1, and
torch.manual_seed(7) right before SmallMLP(); its loss goes from 0.773160
to 0.766216. For the diagnosis, write first which part of the update is wrong
in the broken copy, then which numbers in your report show it and what repairs it.
The script’s last stop prints those numbers for you to cite.
When every step passes, the script runs itself a second time in a fresh Python process, compares every number, and prints one line per rubric row below. The reference prints the same numbers; only the diagnosis wording will differ.
Reference oracle
After your own version passes, run the reference and compare its report with yours key by key:
python examples/foundations_checkpoint.pyIt is the skeleton with the six steps filled in, so every number should match,
except abs_error: it is rounding noise, only has to be below 1e-6, and its
exact value varies. If another number differs, the key tells you which step to
reopen. Compare contracts and
evidence, not variable names. The reference does not replace your diagnosis.
Reference report
{
"implementation": {"imports_llm_course": false},
"shapes_numerics": {"softmax_row_sums": [1.0, 1.0], "cross_entropy": 0.12692801, "mlp_logits_shape": [4, 2], "mlp_loss": 0.63454324},
"gradient_check": {"parameter": "W1[0, 0]", "analytic": -0.12041858, "numeric": -0.12041858, "abs_error": 1.6e-11},
"failure_diagnosis": {"healthy_loss": [9.0, 0.00465361], "broken_loss": [9.0, 6941.78228773], "final_weight": {"healthy": 1.93977626, "broken": -51.09469617}, "diagnosis": ["The broken copy adds learning rate times gradient (update_sign=+1) instead of subtracting it, so every step moves w and b uphill.", "Its loss rises from 9 to about 6942 while the healthy loss falls to 0.005, and its w runs away from the target 2 to about -51; setting update_sign back to -1 repairs it."]},
"pytorch": {"logits_shape": [4, 2], "loss_before": 0.77316, "loss_after": 0.766216},
"reproducibility": {"max_difference": 0.0, "limit": 1e-07}
}Pass rubric
Each row is one key in the report.
| Evidence | Report key | Pass condition |
|---|---|---|
| implementation | implementation |
imports_llm_course is false: no course function in your file |
| shapes/numerics | shapes_numerics |
both softmax rows sum to 1 (softmax_row_sums), cross_entropy is 0.126928 as in F03, the MLP logits are [4, 2] and every gradient has its parameter’s shape |
| gradient check | gradient_check |
abs_error between analytic (backprop) and numeric (central difference) is below 1e-6 |
| failure diagnosis | failure_diagnosis |
healthy_loss [first, last] falls, broken_loss rises, final_weight shows the broken w moving away from 2; the two diagnosis sentences name the wrong sign, cite that evidence, and give the repair |
| PyTorch | pytorch |
logits_shape is [4, 2], loss_after is below loss_before; self-check, the report cannot show it: your code clears gradients before backward() and checks them finite before step() |
| reproducibility | reproducibility |
max_difference between the two runs is at most limit (1e-7) |
If one row fails, return only to its F01–F05 re-entry link in the diagnostic, then retry that row.
What’s next
The neural-foundations milestone is complete. Record the common gate in the diagnostic, then continue with the bigram language-model baseline.
← F05 — MLP, backprop & autograd · T01 — Bigram baseline → · Glossary