Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

F05 — MLP, backpropagation, and tiny autograd

← F04 — Neuron & gradient descent · F06 — PyTorch & checkpoint → · Glossary

Learning outcome

You will push one example through a small two-layer network, follow its gradients back through every layer, and verify them numerically. Then you run a tiny autograd engine that does this bookkeeping for you.

How this page treats the formulas. You will not derive the backward formulas; that would take calculus. Each is stated as a rule, you compute it on real numbers, and then you check it by nudging a weight and watching the loss, as in F04. Plan about an hour if you already write Python, 2–3 hours across two sessions if programming and ML are new; these are guides, not deadlines. Every file below goes into artifacts/my-work/; run it from the course root with python artifacts/my-work/<name>.py.

From one neuron to a hidden layer

An MLP (multilayer perceptron) combines several neurons. A hidden layer is an intermediate vector rather than the final answer. tanh bends each value into (-1,1): tanh(0)=0, tanh(1)≈0.762. An activation is computed for the current input; a parameter is a stored weight used across inputs. Without the bend, two linear layers collapse into one. artifacts/my-work/collapse.py:

import numpy as np

X = np.array([[1.0, 2.0]])
W1 = np.array([[0.5, -0.5, 0.0], [0.25, 0.5, -0.25]])
W2 = np.array([[1.0, -1.0], [-0.5, 0.5], [0.5, 0.0]])
print(X @ W1 @ W2)            # two layers, no tanh
print(X @ (W1 @ W2))          # one merged [2, 2] matrix: the same scores
print(np.tanh(X @ W1) @ W2)   # tanh between the layers: different scores

Output:

[[ 0.5  -0.75]]
[[ 0.5  -0.75]]
[[ 0.299477   -0.53053558]]

W1 @ W2 is one [2, 2] matrix with the same scores: without tanh, two layers can do nothing that one could not.

MLP shapes before values

For X:[2,2], W1:[2,3], b1:[3], W2:[3,2], and b2:[2]:

H = tanh(X @ W1 + b1)    shape [2, 3]
Z = H @ W2 + b2          shape [2, 2]

Rows are examples; the two columns of Z are class logits. Every gradient has its parameter’s shape, and below that rule places every transpose. The walk-through uses one row, X:[1,2].

Walk one example through the network

Steps 1–11 go into artifacts/my-work/one_example.py, each block below the previous one. Run the file after each block; predict one printed number first.

Forward: steps 1–5

import numpy as np

X = np.array([[1.0, 2.0]])                # one example with 2 features
W1 = np.array([[0.5, -0.5, 0.0],
               [0.25, 0.5, -0.25]])       # [2, 3]
b1 = np.array([0.0, 0.0, 0.25])           # [3]
W2 = np.array([[1.0, -1.0],
               [-0.5, 0.5],
               [0.5, 0.0]])               # [3, 2]
b2 = np.array([0.0, 0.0])                 # [2]
target = 1                                # the correct class

A = X @ W1 + b1                           # step 1
print("A", A.round(4))
H = np.tanh(A)                            # step 2
print("H", H.round(4))
Z = H @ W2 + b2                           # step 3
print("Z", Z.round(4))
e = np.exp(Z - Z.max())                   # step 4: softmax, as in F03
P = e / e.sum()
print("P", P.round(4))
loss = -np.log(P[0, target])              # step 5
print("loss", loss.round(4))

Output:

A [[ 1.    0.5  -0.25]]
H [[ 0.7616  0.4621 -0.2449]]
Z [[ 0.4081 -0.5305]]
P [[0.7188 0.2812]]
loss 1.2688
  1. A[0] = 1×0.5 + 2×0.25 + 0 = 1, and A[2] = 1×0 + 2×(−0.25) + 0.25 = −0.25.
  2. H = tanh(A); tanh(1) = 0.7616.
  3. Z[0] = 0.7616×1 + 0.4621×(−0.5) + (−0.2449)×0.5 = 0.4081.
  4. Class 0 gets 0.7188, but the target is class 1.
  5. loss = −ln(0.2812) = 1.2688.

Step 6: where the backward pass starts

Y = np.array([[0.0, 1.0]])                # one-hot: 1 in the target column
dZ = P - Y                                # step 6
print("dZ", dZ.round(4))

Output: dZ [[ 0.7188 -0.7188]]

Y is the one-hot target: 1 in the correct class column, 0 elsewhere. dZ says how fast the loss changes when each score moves; for softmax followed by cross-entropy the rule is probability minus target. Take it on trust; step 11 checks it. The signs fit: lowering the wrong score (positive) or raising the correct one (negative) lowers the loss.

Steps 7–8: the second layer and the shape rule

dW2 = H.T @ dZ                            # step 7
db2 = dZ.sum(axis=0)
print("dW2", dW2.round(4))
print("db2", db2.round(4))
dH = dZ @ W2.T                            # step 8
print("dH", dH.round(4))

Output:

dW2 [[ 0.5474 -0.5474]
 [ 0.3322 -0.3322]
 [-0.1761  0.1761]]
db2 [ 0.7188 -0.7188]
dH [[ 1.4376 -0.7188  0.3594]]

Start with one entry. W2[0, 0] multiplies H[0] = 0.7616 on its way into Z[0]. From F04, w×x changes at rate x when you nudge w, so Z[0] moves at rate 0.7616, and Z[0] moves the loss at rate dZ[0] = 0.7188. Multiply the rates (the chain rule): 0.7616 × 0.7188 = 0.5474, the top-left of dW2. Every entry is such a product, H[j] × dZ[k].

The shape rule places the transpose (.T swaps rows and columns). dW2 must have W2’s shape [3, 2]; from H: [1, 3] and dZ: [1, 2] only H.T @ dZ, [3, 1] @ [1, 2], fits. dH must have H’s shape [1, 3]: only dZ @ W2.T, [1, 2] @ [2, 3], fits. When a backward line puzzles you, write down the shapes.

dH[0] = 1.4376 adds two contributions because H[0] feeds both scores: 0.7188 × 1 + (−0.7188) × (−1). With one row, db2 equals dZ.

Steps 9–10: through tanh into the first layer

dA = dH * (1 - H * H)                     # step 9: back through tanh
print("dA", dA.round(4))
dW1 = X.T @ dA                            # step 10
db1 = dA.sum(axis=0)
print("dW1", dW1.round(4))
print("db1", db1.round(4))

Output:

dA [[ 0.6038 -0.5653  0.3379]]
dW1 [[ 0.6038 -0.5653  0.3379]
 [ 1.2075 -1.1306  0.6757]]
db1 [ 0.6038 -0.5653  0.3379]

The rule: the slope of tanh at A is 1 − tanh(A)², which is 1 - H*H: 1 at 0, near 0 where the curve flattens. * is elementwise, one slope per hidden value. At A[0] = 1: 1 − 0.7616² = 0.42, and dA[0] = 1.4376 × 0.42 = 0.6038. dW1 = X.T @ dA follows the shape rule, [2, 1] @ [1, 3]. Its second row is twice the first because the second input is 2.

Gradient check

Central difference estimates a derivative from the loss alone, without any backward formula:

f'(x) ≈ (f(x + 0.000001) - f(x - 0.000001)) / 0.000002

Nudge up, nudge down, divide the change in loss by the total nudge. Step 11 does this for all six entries of W1; dW1 sits at the end of steps 6–10, so it is only right if every rule on the way is. Add it below step 10:

def loss_with(W):                         # steps 1-5 again, with W in place of W1
    Z = np.tanh(X @ W + b1) @ W2 + b2
    e = np.exp(Z - Z.max())
    return -np.log(e[0, target] / e.sum())


h = 0.000001                              # step 11: the nudge
numeric = np.zeros_like(W1)               # same shape as W1, all zeros
for i in range(2):
    for j in range(3):
        up = W1.copy()
        up[i, j] += h
        down = W1.copy()
        down[i, j] -= h
        numeric[i, j] = (loss_with(up) - loss_with(down)) / (2 * h)
print("numeric", numeric.round(4))
print("max error", np.abs(numeric - dW1).max())

Output:

numeric [[ 0.6038 -0.5653  0.3379]
 [ 1.2075 -1.1306  0.6757]]
max error 1.422688633567759e-10

loss_with repeats steps 1–5 with W in place of W1 (names inside a function are its own). np.zeros_like(W1) makes zeros in W1’s shape; range(2) gives 0 and 1, so the loops visit all six [i, j]. W1.copy() makes a separate array to nudge (Gate toolkit). The largest gap is about 1e-10, well below the required 1e-6.

Now break it: change step 9 to dA = dH, as if the tanh slope were forgotten, and run again. Predict first: which output changes, dW1 or numeric?

Answer

Only dW1 changes; numeric never uses a backward formula. The check now fails by a wide margin:

numeric [[ 0.6038 -0.5653  0.3379]
 [ 1.2075 -1.1306  0.6757]]
max error 1.6677342424389465

A broken local derivative is caught before any training starts. Change step 9 back.

From one row to a batch

The gate and the course’s helper work on several rows at once. Softmax then runs per row, as in the batch step of F03. The one-hot rows come from np.eye(C)[targets] for C classes: np.eye(2) is [[1, 0], [0, 1]], and picking its rows [1, 0] gives [[0, 1], [1, 0]]. dZ = (P - Y) / B, because the loss is the mean over B rows. H.T @ dZ, [3, B] @ [B, 2], and the biases’ .sum(axis=0) add up the rows’ contributions.

In artifacts/my-work/batch.py, write my_gradients(X, targets, W1, b1, W2, b2) returning dW1, db1, dW2, db2: steps 1–10 with these changes. It goes where the comment is; the rest compares it with the course helper:

import numpy as np

from llm_course import mlp_loss_and_gradients


# ... your my_gradients function here ...


X = np.array([[1.0, 2.0],
              [0.5, -1.0]])
targets = np.array([1, 0])
W1 = np.array([[0.5, -0.5, 0.0], [0.25, 0.5, -0.25]])
b1 = np.array([0.0, 0.0, 0.25])
W2 = np.array([[1.0, -1.0], [-0.5, 0.5], [0.5, 0.0]])
b2 = np.array([0.0, 0.0])

mine = my_gradients(X, targets, W1, b1, W2, b2)
ref = mlp_loss_and_gradients(X, targets, W1, b1, W2, b2)
names = ["dW1", "db1", "dW2", "db2"]
for name, a, b in zip(names, mine, [ref.dw1, ref.db1, ref.dw2, ref.db2], strict=True):
    print(name, a.shape, np.abs(a - b).max())

ref.dw1 is the helper’s dW1, and so on. Expected: each gradient has its parameter’s shape, and each gap is 0.0 or a tiny number such as 1e-17:

dW1 (2, 3) 0.0
db1 (3,) 0.0
dW2 (3, 2) 0.0
db2 (2,) 0.0
Hint

Start with B = X.shape[0], the number of rows. Steps 1–3 are unchanged. Step 4 needs axis=-1, keepdims=True twice, Y comes from np.eye, and C is Z.shape[1]. Return the four gradients separated by commas.

Reference
def my_gradients(X, targets, W1, b1, W2, b2):
    B = X.shape[0]                               # rows in the batch
    H = np.tanh(X @ W1 + b1)
    Z = H @ W2 + b2
    e = np.exp(Z - Z.max(axis=-1, keepdims=True))
    P = e / e.sum(axis=-1, keepdims=True)
    Y = np.eye(Z.shape[1])[targets]
    dZ = (P - Y) / B
    dW2 = H.T @ dZ
    db2 = dZ.sum(axis=0)
    dA = (dZ @ W2.T) * (1 - H * H)
    dW1 = X.T @ dA
    db1 = dA.sum(axis=0)
    return dW1, db1, dW2, db2

Backpropagation is repeated local multiplication

Steps 6–10 multiplied local rates by hand. An autograd engine records each operation in the forward pass and does the multiplying for you, as PyTorch’s loss.backward() will in F06. You read and run the course’s tiny one, Value in src/llm_course/autograd.py, rather than build it: the gate asks for the NumPy backward pass you have written. For q=(2x+1)² at x=3, 2x+1=7 and q=49; the backward path multiplies local derivatives:

dq/dx = 2(2x + 1) × 2 = 2 × 7 × 2 = 28

artifacts/my-work/value_demo.py:

from llm_course import Value

x = Value(3.0)
q = (2.0 * x + 1.0) ** 2
q.backward()
print(q.data, x.grad)

a = Value(3.0)
y = a * a
y.backward()
print(y.data, a.grad)

Output:

49.0 28.0
9.0 6.0

Value is a class (F06 explains classes): each Value holds a number in .data and, after backward(), its derivative in .grad.

How 2.0 * x becomes a Value

Python turns x * 2.0 into a call of the method x.__mul__(2.0). For 2.0 * x, the plain number 2.0 cannot multiply a Value, so Python asks the right-hand side instead: x.__rmul__(2.0), r for right. + and ** work the same way through __add__, __radd__ and __pow__. That is why (2.0 * x + 1.0) ** 2 builds new Values instead of plain numbers.

Recording. Each operation returns a new Value that remembers its inputs and a small function sending the gradient back. For *, self and other_value are the factors and output the product:

def backward():
    self.grad += other_value.data * output.grad
    other_value.grad += self.data * output.grad

That is F04’s rule (a product moves at the rate of the other factor) times the gradient arriving from above: the chain rule, one operation at a time.

Order. q.backward() sets q.grad = 1 and visits each Value only after everything that uses it, so its gradient is complete before it is passed on.

In y = a * a, a reaches y along two routes, a shared path. Each contributes 3 × 1 = 3, and += adds them: a.grad is 6, the slope 2a of a². Predict: with = instead of += in both lines, what would a.grad be?

Answer

3.0. self and other_value are the same a, so the second line overwrites the first instead of adding to it. Adding with += is why a value used twice gets the gradient of both uses, and the same reason db1 sums over rows.

Gate toolkit

F06’s gate asks you to write the pipeline yourself. Beyond the steps above it needs three rules; for keepdims, axis=-1 and P[np.arange(B), targets], see the batch step in F03.

  1. One-hot targets: Y = np.eye(C)[targets], from the batch step.
  2. Copy before you nudge: up = W1.copy(). up = W1 copies nothing, so nudging up would change the real W1 and every later number.
  3. Stay in float64, NumPy’s default. float32, from dtype=np.float32 or from PyTorch tensors, makes a correct gradient check fail; print(W1.dtype) shows which one you have.

artifacts/my-work/copy_demo.py:

import numpy as np

W = np.array([0.5, -0.5])
same = W                  # a second name for the same array
same[0] += 0.000001
print(W)                  # W changed too

W = np.array([0.5, -0.5])
up = W.copy()             # a separate array
up[0] += 0.000001
print(W)                  # W is untouched

Output:

[ 0.500001 -0.5     ]
[ 0.5 -0.5]
Demo: float32 fails a correct gradient check

artifacts/my-work/float_demo.py:

import numpy as np

print(np.float32(0.5) + np.float32(0.000001) - np.float32(0.5))
print(np.float64(0.5) + np.float64(0.000001) - np.float64(0.5))

Output:

1.013279e-06
1.0000000000287557e-06

float32 keeps about 7 significant digits, float64 about 16, so in float32 a nudge of 0.000001 on 0.5 is stored 1.3 % too large. Try it: add W1 = W1.astype(np.float32) below target = 1 in one_example.py. The formula still divides by 0.000002, so every numeric estimate for an entry of ±0.5 comes out 1.3 % too large; the max error becomes about 0.015, and the check fails although every formula is right. Remove the line again.

Completion evidence

  • steps 1–10 print the numbers shown, each gradient in its parameter’s shape;
  • the gradient check passes below 1e-6 and fails without the tanh slope;
  • your batch gradients match mlp_loss_and_gradients;
  • you can explain why a.grad is 6 for y = a * a.

What’s next

Translate the mechanisms into framework code in F06 — PyTorch and checkpoint.

← F04 — Neuron & gradient descent · F06 — PyTorch & checkpoint → · Glossary

Was this page helpful?