F05 — MLP, backpropagation, and tiny autograd
← F04 — Neuron & gradient descent · F06 — PyTorch & checkpoint → · Glossary
Learning outcome
You will push one example through a small two-layer network, follow its gradients back through every layer, and verify them numerically. Then you run a tiny autograd engine that does this bookkeeping for you.
How this page treats the formulas. You will not derive the backward
formulas; that would take calculus. Each is stated as a rule, you compute it on
real numbers, and then you check it by nudging a weight and watching the loss,
as in F04. Plan about an hour if you already write
Python, 2–3 hours across two sessions if programming and ML are new; these are
guides, not deadlines. Every file below goes into artifacts/my-work/; run it
from the course root with python artifacts/my-work/<name>.py.
From one neuron to a hidden layer
An MLP (multilayer perceptron) combines several neurons. A hidden layer is an
intermediate vector rather than the final answer. tanh bends each value into
(-1,1): tanh(0)=0, tanh(1)≈0.762. An activation is computed for the current
input; a parameter is a stored weight used across inputs. Without the bend, two
linear layers collapse into one. artifacts/my-work/collapse.py:
import numpy as np
X = np.array([[1.0, 2.0]])
W1 = np.array([[0.5, -0.5, 0.0], [0.25, 0.5, -0.25]])
W2 = np.array([[1.0, -1.0], [-0.5, 0.5], [0.5, 0.0]])
print(X @ W1 @ W2) # two layers, no tanh
print(X @ (W1 @ W2)) # one merged [2, 2] matrix: the same scores
print(np.tanh(X @ W1) @ W2) # tanh between the layers: different scoresOutput:
[[ 0.5 -0.75]]
[[ 0.5 -0.75]]
[[ 0.299477 -0.53053558]]W1 @ W2 is one [2, 2] matrix with the same scores: without tanh, two
layers can do nothing that one could not.
MLP shapes before values
For X:[2,2], W1:[2,3], b1:[3], W2:[3,2], and b2:[2]:
H = tanh(X @ W1 + b1) shape [2, 3]
Z = H @ W2 + b2 shape [2, 2]Rows are examples; the two columns of Z are class logits. Every gradient has
its parameter’s shape, and below that rule places every transpose. The
walk-through uses one row, X:[1,2].
Walk one example through the network
Steps 1–11 go into artifacts/my-work/one_example.py, each block below the
previous one. Run the file after each block; predict one printed number first.
Forward: steps 1–5
import numpy as np
X = np.array([[1.0, 2.0]]) # one example with 2 features
W1 = np.array([[0.5, -0.5, 0.0],
[0.25, 0.5, -0.25]]) # [2, 3]
b1 = np.array([0.0, 0.0, 0.25]) # [3]
W2 = np.array([[1.0, -1.0],
[-0.5, 0.5],
[0.5, 0.0]]) # [3, 2]
b2 = np.array([0.0, 0.0]) # [2]
target = 1 # the correct class
A = X @ W1 + b1 # step 1
print("A", A.round(4))
H = np.tanh(A) # step 2
print("H", H.round(4))
Z = H @ W2 + b2 # step 3
print("Z", Z.round(4))
e = np.exp(Z - Z.max()) # step 4: softmax, as in F03
P = e / e.sum()
print("P", P.round(4))
loss = -np.log(P[0, target]) # step 5
print("loss", loss.round(4))Output:
A [[ 1. 0.5 -0.25]]
H [[ 0.7616 0.4621 -0.2449]]
Z [[ 0.4081 -0.5305]]
P [[0.7188 0.2812]]
loss 1.2688A[0] = 1×0.5 + 2×0.25 + 0 = 1, andA[2] = 1×0 + 2×(−0.25) + 0.25 = −0.25.H = tanh(A);tanh(1) = 0.7616.Z[0] = 0.7616×1 + 0.4621×(−0.5) + (−0.2449)×0.5 = 0.4081.- Class 0 gets 0.7188, but the target is class 1.
loss = −ln(0.2812) = 1.2688.
Step 6: where the backward pass starts
Y = np.array([[0.0, 1.0]]) # one-hot: 1 in the target column
dZ = P - Y # step 6
print("dZ", dZ.round(4))Output: dZ [[ 0.7188 -0.7188]]
Y is the one-hot target: 1 in the correct class column, 0 elsewhere.
dZ says how fast the loss changes when each score moves; for softmax followed
by cross-entropy the rule is probability minus target. Take it on trust; step 11
checks it. The signs fit: lowering the wrong score (positive) or raising the
correct one (negative) lowers the loss.
Steps 7–8: the second layer and the shape rule
dW2 = H.T @ dZ # step 7
db2 = dZ.sum(axis=0)
print("dW2", dW2.round(4))
print("db2", db2.round(4))
dH = dZ @ W2.T # step 8
print("dH", dH.round(4))Output:
dW2 [[ 0.5474 -0.5474]
[ 0.3322 -0.3322]
[-0.1761 0.1761]]
db2 [ 0.7188 -0.7188]
dH [[ 1.4376 -0.7188 0.3594]]Start with one entry. W2[0, 0] multiplies H[0] = 0.7616 on its way into
Z[0]. From F04, w×x changes at rate x when you nudge w, so Z[0] moves
at rate 0.7616, and Z[0] moves the loss at rate dZ[0] = 0.7188. Multiply
the rates (the chain rule): 0.7616 × 0.7188 = 0.5474, the top-left of dW2.
Every entry is such a product, H[j] × dZ[k].
The shape rule places the transpose (.T swaps rows and columns). dW2
must have W2’s shape [3, 2]; from H: [1, 3] and dZ: [1, 2] only
H.T @ dZ, [3, 1] @ [1, 2], fits. dH must have H’s shape [1, 3]: only
dZ @ W2.T, [1, 2] @ [2, 3], fits. When a backward line puzzles you, write
down the shapes.
dH[0] = 1.4376 adds two contributions because H[0] feeds both scores:
0.7188 × 1 + (−0.7188) × (−1). With one row, db2 equals dZ.
Steps 9–10: through tanh into the first layer
dA = dH * (1 - H * H) # step 9: back through tanh
print("dA", dA.round(4))
dW1 = X.T @ dA # step 10
db1 = dA.sum(axis=0)
print("dW1", dW1.round(4))
print("db1", db1.round(4))Output:
dA [[ 0.6038 -0.5653 0.3379]]
dW1 [[ 0.6038 -0.5653 0.3379]
[ 1.2075 -1.1306 0.6757]]
db1 [ 0.6038 -0.5653 0.3379]The rule: the slope of tanh at A is 1 − tanh(A)², which is 1 - H*H:
1 at 0, near 0 where the curve flattens. * is elementwise, one slope per
hidden value. At A[0] = 1: 1 − 0.7616² = 0.42, and
dA[0] = 1.4376 × 0.42 = 0.6038. dW1 = X.T @ dA follows the shape rule,
[2, 1] @ [1, 3]. Its second row is twice the first because the second input
is 2.
Gradient check
Central difference estimates a derivative from the loss alone, without any backward formula:
f'(x) ≈ (f(x + 0.000001) - f(x - 0.000001)) / 0.000002Nudge up, nudge down, divide the change in loss by the total nudge. Step 11
does this for all six entries of W1; dW1 sits at the end of steps 6–10, so
it is only right if every rule on the way is. Add it below step 10:
def loss_with(W): # steps 1-5 again, with W in place of W1
Z = np.tanh(X @ W + b1) @ W2 + b2
e = np.exp(Z - Z.max())
return -np.log(e[0, target] / e.sum())
h = 0.000001 # step 11: the nudge
numeric = np.zeros_like(W1) # same shape as W1, all zeros
for i in range(2):
for j in range(3):
up = W1.copy()
up[i, j] += h
down = W1.copy()
down[i, j] -= h
numeric[i, j] = (loss_with(up) - loss_with(down)) / (2 * h)
print("numeric", numeric.round(4))
print("max error", np.abs(numeric - dW1).max())Output:
numeric [[ 0.6038 -0.5653 0.3379]
[ 1.2075 -1.1306 0.6757]]
max error 1.422688633567759e-10loss_with repeats steps 1–5 with W in place of W1 (names inside a
function are its own). np.zeros_like(W1) makes zeros in W1’s shape;
range(2) gives 0 and 1, so the loops visit all six [i, j]. W1.copy() makes
a separate array to nudge (Gate toolkit). The largest gap is
about 1e-10, well below the required 1e-6.
Now break it: change step 9 to dA = dH, as if the tanh slope were forgotten,
and run again. Predict first: which output changes, dW1 or numeric?
Answer
Only dW1 changes; numeric never uses a backward formula. The check now
fails by a wide margin:
numeric [[ 0.6038 -0.5653 0.3379]
[ 1.2075 -1.1306 0.6757]]
max error 1.6677342424389465A broken local derivative is caught before any training starts. Change step 9 back.
From one row to a batch
The gate and the course’s helper work on several rows at once. Softmax then
runs per row, as in the batch step of F03.
The one-hot rows come from np.eye(C)[targets] for C classes: np.eye(2) is
[[1, 0], [0, 1]], and picking its rows [1, 0] gives [[0, 1], [1, 0]].
dZ = (P - Y) / B, because the loss is the mean over B rows. H.T @ dZ,
[3, B] @ [B, 2], and the biases’ .sum(axis=0) add up the rows’
contributions.
In artifacts/my-work/batch.py, write my_gradients(X, targets, W1, b1, W2, b2)
returning dW1, db1, dW2, db2: steps 1–10 with these changes. It goes where the
comment is; the rest compares it with the course helper:
import numpy as np
from llm_course import mlp_loss_and_gradients
# ... your my_gradients function here ...
X = np.array([[1.0, 2.0],
[0.5, -1.0]])
targets = np.array([1, 0])
W1 = np.array([[0.5, -0.5, 0.0], [0.25, 0.5, -0.25]])
b1 = np.array([0.0, 0.0, 0.25])
W2 = np.array([[1.0, -1.0], [-0.5, 0.5], [0.5, 0.0]])
b2 = np.array([0.0, 0.0])
mine = my_gradients(X, targets, W1, b1, W2, b2)
ref = mlp_loss_and_gradients(X, targets, W1, b1, W2, b2)
names = ["dW1", "db1", "dW2", "db2"]
for name, a, b in zip(names, mine, [ref.dw1, ref.db1, ref.dw2, ref.db2], strict=True):
print(name, a.shape, np.abs(a - b).max())ref.dw1 is the helper’s dW1, and so on. Expected: each gradient has its
parameter’s shape, and each gap is 0.0 or a tiny number such as 1e-17:
dW1 (2, 3) 0.0
db1 (3,) 0.0
dW2 (3, 2) 0.0
db2 (2,) 0.0Hint
Start with B = X.shape[0], the number of rows. Steps 1–3 are unchanged.
Step 4 needs axis=-1, keepdims=True twice, Y comes from np.eye, and
C is Z.shape[1]. Return the four gradients separated by commas.
Reference
def my_gradients(X, targets, W1, b1, W2, b2):
B = X.shape[0] # rows in the batch
H = np.tanh(X @ W1 + b1)
Z = H @ W2 + b2
e = np.exp(Z - Z.max(axis=-1, keepdims=True))
P = e / e.sum(axis=-1, keepdims=True)
Y = np.eye(Z.shape[1])[targets]
dZ = (P - Y) / B
dW2 = H.T @ dZ
db2 = dZ.sum(axis=0)
dA = (dZ @ W2.T) * (1 - H * H)
dW1 = X.T @ dA
db1 = dA.sum(axis=0)
return dW1, db1, dW2, db2Backpropagation is repeated local multiplication
Steps 6–10 multiplied local rates by hand. An autograd engine records each
operation in the forward pass and does the multiplying for you, as PyTorch’s
loss.backward() will in F06. You read and run the course’s tiny one, Value
in src/llm_course/autograd.py, rather than build it: the gate asks for the
NumPy backward pass you have written. For q=(2x+1)² at x=3, 2x+1=7 and
q=49; the backward path multiplies local derivatives:
dq/dx = 2(2x + 1) × 2 = 2 × 7 × 2 = 28artifacts/my-work/value_demo.py:
from llm_course import Value
x = Value(3.0)
q = (2.0 * x + 1.0) ** 2
q.backward()
print(q.data, x.grad)
a = Value(3.0)
y = a * a
y.backward()
print(y.data, a.grad)Output:
49.0 28.0
9.0 6.0Value is a class (F06 explains classes): each Value holds a number in
.data and, after backward(), its derivative in .grad.
How 2.0 * x becomes a Value
Python turns x * 2.0 into a call of the method x.__mul__(2.0). For
2.0 * x, the plain number 2.0 cannot multiply a Value, so Python asks
the right-hand side instead: x.__rmul__(2.0), r for right. + and **
work the same way through __add__, __radd__ and __pow__. That is why
(2.0 * x + 1.0) ** 2 builds new Values instead of plain numbers.
Recording. Each operation returns a new Value that remembers its inputs
and a small function sending the gradient back. For *, self and
other_value are the factors and output the product:
def backward():
self.grad += other_value.data * output.grad
other_value.grad += self.data * output.gradThat is F04’s rule (a product moves at the rate of the other factor) times the gradient arriving from above: the chain rule, one operation at a time.
Order. q.backward() sets q.grad = 1 and visits each Value only after
everything that uses it, so its gradient is complete before it is passed on.
In y = a * a, a reaches y along two routes, a shared path. Each
contributes 3 × 1 = 3, and += adds them: a.grad is 6, the slope 2a of
a². Predict: with = instead of += in both lines, what would a.grad be?
Answer
3.0. self and other_value are the same a, so the second line
overwrites the first instead of adding to it. Adding with += is why a
value used twice gets the gradient of both uses, and the same reason db1
sums over rows.
Gate toolkit
F06’s gate asks you to write the pipeline yourself. Beyond the steps above it
needs three rules; for keepdims, axis=-1 and P[np.arange(B), targets],
see the batch step in F03.
- One-hot targets:
Y = np.eye(C)[targets], from the batch step. - Copy before you nudge:
up = W1.copy().up = W1copies nothing, so nudgingupwould change the realW1and every later number. - Stay in float64, NumPy’s default. float32, from
dtype=np.float32or from PyTorch tensors, makes a correct gradient check fail;print(W1.dtype)shows which one you have.
artifacts/my-work/copy_demo.py:
import numpy as np
W = np.array([0.5, -0.5])
same = W # a second name for the same array
same[0] += 0.000001
print(W) # W changed too
W = np.array([0.5, -0.5])
up = W.copy() # a separate array
up[0] += 0.000001
print(W) # W is untouchedOutput:
[ 0.500001 -0.5 ]
[ 0.5 -0.5]Demo: float32 fails a correct gradient check
artifacts/my-work/float_demo.py:
import numpy as np
print(np.float32(0.5) + np.float32(0.000001) - np.float32(0.5))
print(np.float64(0.5) + np.float64(0.000001) - np.float64(0.5))Output:
1.013279e-06
1.0000000000287557e-06float32 keeps about 7 significant digits, float64 about 16, so in float32 a
nudge of 0.000001 on 0.5 is stored 1.3 % too large. Try it: add
W1 = W1.astype(np.float32) below target = 1 in one_example.py. The
formula still divides by 0.000002, so every numeric estimate for an entry
of ±0.5 comes out 1.3 % too large; the max error becomes about 0.015, and
the check fails although every formula is right. Remove the line again.
Completion evidence
- steps 1–10 print the numbers shown, each gradient in its parameter’s shape;
- the gradient check passes below
1e-6and fails without the tanh slope; - your batch gradients match
mlp_loss_and_gradients; - you can explain why
a.gradis 6 fory = a * a.
What’s next
Translate the mechanisms into framework code in F06 — PyTorch and checkpoint.
← F04 — Neuron & gradient descent · F06 — PyTorch & checkpoint → · Glossary