---
title: "F05 — MLP, backpropagation, and tiny autograd"
sidebar:
  label: "F05 — MLP, backprop & autograd"
---

<span id="f05-mlp-backpropagation-and-tiny-autograd" />


[← F04 — Neuron & gradient descent](/foundations-04-neuron) · [F06 — PyTorch & checkpoint →](/foundations-06-pytorch-checkpoint) · [Glossary](/glossary)

## Learning outcome [#learning-outcome]

You will push one example through a small two-layer network, follow its
gradients back through every layer, and verify them numerically. Then you run a
tiny autograd engine that does this bookkeeping for you.

**How this page treats the formulas.** You will not derive the backward
formulas; that would take calculus. Each is stated as a rule, you compute it on
real numbers, and then you check it by nudging a weight and watching the loss,
as in [F04](/foundations-04-neuron). Plan about an hour if you already write
Python, 2–3 hours across two sessions if programming and ML are new; these are
guides, not deadlines. Every file below goes into `artifacts/my-work/`; run it
from the course root with `python artifacts/my-work/<name>.py`.

## From one neuron to a hidden layer [#from-one-neuron-to-a-hidden-layer]

An MLP (multilayer perceptron) combines several neurons. A hidden layer is an
intermediate vector rather than the final answer. `tanh` bends each value into
(-1,1): `tanh(0)=0`, `tanh(1)≈0.762`. An activation is computed for the current
input; a parameter is a stored weight used across inputs. Without the bend, two
linear layers collapse into one. `artifacts/my-work/collapse.py`:

```python
import numpy as np

X = np.array([[1.0, 2.0]])
W1 = np.array([[0.5, -0.5, 0.0], [0.25, 0.5, -0.25]])
W2 = np.array([[1.0, -1.0], [-0.5, 0.5], [0.5, 0.0]])
print(X @ W1 @ W2)            # two layers, no tanh
print(X @ (W1 @ W2))          # one merged [2, 2] matrix: the same scores
print(np.tanh(X @ W1) @ W2)   # tanh between the layers: different scores
```

Output:

```text
[[ 0.5  -0.75]]
[[ 0.5  -0.75]]
[[ 0.299477   -0.53053558]]
```

`W1 @ W2` is one `[2, 2]` matrix with the same scores: without `tanh`, two
layers can do nothing that one could not.

## MLP shapes before values [#mlp-shapes-before-values]

For `X:[2,2]`, `W1:[2,3]`, `b1:[3]`, `W2:[3,2]`, and `b2:[2]`:

```text
H = tanh(X @ W1 + b1)    shape [2, 3]
Z = H @ W2 + b2          shape [2, 2]
```

Rows are examples; the two columns of `Z` are class logits. Every gradient has
its parameter's shape, and below that rule places every transpose. The
walk-through uses one row, `X:[1,2]`.

## Walk one example through the network [#walk-one-example-through-the-network]

Steps 1–11 go into `artifacts/my-work/one_example.py`, each block below the
previous one. Run the file after each block; predict one printed number first.

### Forward: steps 1–5 [#forward-steps-15]

```python
import numpy as np

X = np.array([[1.0, 2.0]])                # one example with 2 features
W1 = np.array([[0.5, -0.5, 0.0],
               [0.25, 0.5, -0.25]])       # [2, 3]
b1 = np.array([0.0, 0.0, 0.25])           # [3]
W2 = np.array([[1.0, -1.0],
               [-0.5, 0.5],
               [0.5, 0.0]])               # [3, 2]
b2 = np.array([0.0, 0.0])                 # [2]
target = 1                                # the correct class

A = X @ W1 + b1                           # step 1
print("A", A.round(4))
H = np.tanh(A)                            # step 2
print("H", H.round(4))
Z = H @ W2 + b2                           # step 3
print("Z", Z.round(4))
e = np.exp(Z - Z.max())                   # step 4: softmax, as in F03
P = e / e.sum()
print("P", P.round(4))
loss = -np.log(P[0, target])              # step 5
print("loss", loss.round(4))
```

Output:

```text
A [[ 1.    0.5  -0.25]]
H [[ 0.7616  0.4621 -0.2449]]
Z [[ 0.4081 -0.5305]]
P [[0.7188 0.2812]]
loss 1.2688
```

1. `A[0] = 1×0.5 + 2×0.25 + 0 = 1`, and `A[2] = 1×0 + 2×(−0.25) + 0.25 = −0.25`.
2. `H = tanh(A)`; `tanh(1) = 0.7616`.
3. `Z[0] = 0.7616×1 + 0.4621×(−0.5) + (−0.2449)×0.5 = 0.4081`.
4. Class 0 gets 0.7188, but the target is class 1.
5. `loss = −ln(0.2812) = 1.2688`.

### Step 6: where the backward pass starts [#step-6-where-the-backward-pass-starts]

```python
Y = np.array([[0.0, 1.0]])                # one-hot: 1 in the target column
dZ = P - Y                                # step 6
print("dZ", dZ.round(4))
```

Output: `dZ [[ 0.7188 -0.7188]]`

`Y` is the **one-hot** target: 1 in the correct class column, 0 elsewhere.
`dZ` says how fast the loss changes when each score moves; for softmax followed
by cross-entropy the rule is probability minus target. Take it on trust; step 11
checks it. The signs fit: lowering the wrong score (positive) or raising the
correct one (negative) lowers the loss.

### Steps 7–8: the second layer and the shape rule [#steps-78-the-second-layer-and-the-shape-rule]

```python
dW2 = H.T @ dZ                            # step 7
db2 = dZ.sum(axis=0)
print("dW2", dW2.round(4))
print("db2", db2.round(4))
dH = dZ @ W2.T                            # step 8
print("dH", dH.round(4))
```

Output:

```text
dW2 [[ 0.5474 -0.5474]
 [ 0.3322 -0.3322]
 [-0.1761  0.1761]]
db2 [ 0.7188 -0.7188]
dH [[ 1.4376 -0.7188  0.3594]]
```

Start with one entry. `W2[0, 0]` multiplies `H[0] = 0.7616` on its way into
`Z[0]`. From F04, `w×x` changes at rate `x` when you nudge `w`, so `Z[0]` moves
at rate 0.7616, and `Z[0]` moves the loss at rate `dZ[0] = 0.7188`. Multiply
the rates (the chain rule): `0.7616 × 0.7188 = 0.5474`, the top-left of `dW2`.
Every entry is such a product, `H[j] × dZ[k]`.

**The shape rule** places the transpose (`.T` swaps rows and columns). `dW2`
must have `W2`'s shape `[3, 2]`; from `H: [1, 3]` and `dZ: [1, 2]` only
`H.T @ dZ`, `[3, 1] @ [1, 2]`, fits. `dH` must have `H`'s shape `[1, 3]`: only
`dZ @ W2.T`, `[1, 2] @ [2, 3]`, fits. When a backward line puzzles you, write
down the shapes.

`dH[0] = 1.4376` adds two contributions because `H[0]` feeds both scores:
`0.7188 × 1 + (−0.7188) × (−1)`. With one row, `db2` equals `dZ`.

### Steps 9–10: through tanh into the first layer [#steps-910-through-tanh-into-the-first-layer]

```python
dA = dH * (1 - H * H)                     # step 9: back through tanh
print("dA", dA.round(4))
dW1 = X.T @ dA                            # step 10
db1 = dA.sum(axis=0)
print("dW1", dW1.round(4))
print("db1", db1.round(4))
```

Output:

```text
dA [[ 0.6038 -0.5653  0.3379]]
dW1 [[ 0.6038 -0.5653  0.3379]
 [ 1.2075 -1.1306  0.6757]]
db1 [ 0.6038 -0.5653  0.3379]
```

The rule: the slope of `tanh` at `A` is `1 − tanh(A)²`, which is `1 - H*H`:
1 at 0, near 0 where the curve flattens. `*` is elementwise, one slope per
hidden value. At `A[0] = 1`: `1 − 0.7616² = 0.42`, and
`dA[0] = 1.4376 × 0.42 = 0.6038`. `dW1 = X.T @ dA` follows the shape rule,
`[2, 1] @ [1, 3]`. Its second row is twice the first because the second input
is 2.

## Gradient check [#gradient-check]

Central difference estimates a derivative from the loss alone, without any
backward formula:

```text
f'(x) ≈ (f(x + 0.000001) - f(x - 0.000001)) / 0.000002
```

Nudge up, nudge down, divide the change in loss by the total nudge. Step 11
does this for all six entries of `W1`; `dW1` sits at the end of steps 6–10, so
it is only right if every rule on the way is. Add it below step 10:

```python
def loss_with(W):                         # steps 1-5 again, with W in place of W1
    Z = np.tanh(X @ W + b1) @ W2 + b2
    e = np.exp(Z - Z.max())
    return -np.log(e[0, target] / e.sum())


h = 0.000001                              # step 11: the nudge
numeric = np.zeros_like(W1)               # same shape as W1, all zeros
for i in range(2):
    for j in range(3):
        up = W1.copy()
        up[i, j] += h
        down = W1.copy()
        down[i, j] -= h
        numeric[i, j] = (loss_with(up) - loss_with(down)) / (2 * h)
print("numeric", numeric.round(4))
print("max error", np.abs(numeric - dW1).max())
```

Output:

```text
numeric [[ 0.6038 -0.5653  0.3379]
 [ 1.2075 -1.1306  0.6757]]
max error 1.422688633567759e-10
```

`loss_with` repeats steps 1–5 with `W` in place of `W1` (names inside a
function are its own). `np.zeros_like(W1)` makes zeros in `W1`'s shape;
`range(2)` gives 0 and 1, so the loops visit all six `[i, j]`. `W1.copy()` makes
a separate array to nudge ([Gate toolkit](#gate-toolkit)). The largest gap is
about `1e-10`, well below the required `1e-6`.

Now break it: change step 9 to `dA = dH`, as if the tanh slope were forgotten,
and run again. Predict first: which output changes, `dW1` or `numeric`?

<details>
<summary>Answer</summary>

Only `dW1` changes; `numeric` never uses a backward formula. The check now
fails by a wide margin:

```text
numeric [[ 0.6038 -0.5653  0.3379]
 [ 1.2075 -1.1306  0.6757]]
max error 1.6677342424389465
```

A broken local derivative is caught before any training starts. Change
step 9 back.


</details>

## From one row to a batch [#from-one-row-to-a-batch]

The gate and the course's helper work on several rows at once. Softmax then
runs per row, as in the batch step of [F03](/foundations-03-probability).
The one-hot rows come from `np.eye(C)[targets]` for `C` classes: `np.eye(2)` is
`[[1, 0], [0, 1]]`, and picking its rows `[1, 0]` gives `[[0, 1], [1, 0]]`.
`dZ = (P - Y) / B`, because the loss is the mean over `B` rows. `H.T @ dZ`,
`[3, B] @ [B, 2]`, and the biases' `.sum(axis=0)` add up the rows'
contributions.

In `artifacts/my-work/batch.py`, write `my_gradients(X, targets, W1, b1, W2, b2)`
returning `dW1, db1, dW2, db2`: steps 1–10 with these changes. It goes where the
comment is; the rest compares it with the course helper:

```python
import numpy as np

from llm_course import mlp_loss_and_gradients


# ... your my_gradients function here ...


X = np.array([[1.0, 2.0],
              [0.5, -1.0]])
targets = np.array([1, 0])
W1 = np.array([[0.5, -0.5, 0.0], [0.25, 0.5, -0.25]])
b1 = np.array([0.0, 0.0, 0.25])
W2 = np.array([[1.0, -1.0], [-0.5, 0.5], [0.5, 0.0]])
b2 = np.array([0.0, 0.0])

mine = my_gradients(X, targets, W1, b1, W2, b2)
ref = mlp_loss_and_gradients(X, targets, W1, b1, W2, b2)
names = ["dW1", "db1", "dW2", "db2"]
for name, a, b in zip(names, mine, [ref.dw1, ref.db1, ref.dw2, ref.db2], strict=True):
    print(name, a.shape, np.abs(a - b).max())
```

`ref.dw1` is the helper's `dW1`, and so on. Expected: each gradient has its
parameter's shape, and each gap is 0.0 or a tiny number such as `1e-17`:

```text
dW1 (2, 3) 0.0
db1 (3,) 0.0
dW2 (3, 2) 0.0
db2 (2,) 0.0
```

<details>
<summary>Hint</summary>

Start with `B = X.shape[0]`, the number of rows. Steps 1–3 are unchanged.
Step 4 needs `axis=-1, keepdims=True` twice, `Y` comes from `np.eye`, and
`C` is `Z.shape[1]`. Return the four gradients separated by commas.


</details>

<details>
<summary>Reference</summary>

```python
def my_gradients(X, targets, W1, b1, W2, b2):
    B = X.shape[0]                               # rows in the batch
    H = np.tanh(X @ W1 + b1)
    Z = H @ W2 + b2
    e = np.exp(Z - Z.max(axis=-1, keepdims=True))
    P = e / e.sum(axis=-1, keepdims=True)
    Y = np.eye(Z.shape[1])[targets]
    dZ = (P - Y) / B
    dW2 = H.T @ dZ
    db2 = dZ.sum(axis=0)
    dA = (dZ @ W2.T) * (1 - H * H)
    dW1 = X.T @ dA
    db1 = dA.sum(axis=0)
    return dW1, db1, dW2, db2
```


</details>

## Backpropagation is repeated local multiplication [#backpropagation-is-repeated-local-multiplication]

Steps 6–10 multiplied local rates by hand. An autograd engine records each
operation in the forward pass and does the multiplying for you, as PyTorch's
`loss.backward()` will in F06. You read and run the course's tiny one, `Value`
in `src/llm_course/autograd.py`, rather than build it: the gate asks for the
NumPy backward pass you have written. For `q=(2x+1)²` at `x=3`, `2x+1=7` and
`q=49`; the backward path multiplies local derivatives:

```text
dq/dx = 2(2x + 1) × 2 = 2 × 7 × 2 = 28
```

`artifacts/my-work/value_demo.py`:

```python
from llm_course import Value

x = Value(3.0)
q = (2.0 * x + 1.0) ** 2
q.backward()
print(q.data, x.grad)

a = Value(3.0)
y = a * a
y.backward()
print(y.data, a.grad)
```

Output:

```text
49.0 28.0
9.0 6.0
```

`Value` is a class (F06 explains classes): each `Value` holds a number in
`.data` and, after `backward()`, its derivative in `.grad`.

<details>
<summary>How 2.0 * x becomes a Value</summary>

Python turns `x * 2.0` into a call of the method `x.__mul__(2.0)`. For
`2.0 * x`, the plain number 2.0 cannot multiply a `Value`, so Python asks
the right-hand side instead: `x.__rmul__(2.0)`, r for right. `+` and `**`
work the same way through `__add__`, `__radd__` and `__pow__`. That is why
`(2.0 * x + 1.0) ** 2` builds new `Value`s instead of plain numbers.


</details>

**Recording.** Each operation returns a new `Value` that remembers its inputs
and a small function sending the gradient back. For `*`, `self` and
`other_value` are the factors and `output` the product:

```python
def backward():
    self.grad += other_value.data * output.grad
    other_value.grad += self.data * output.grad
```

That is F04's rule (a product moves at the rate of the other factor) times the
gradient arriving from above: the chain rule, one operation at a time.

**Order.** `q.backward()` sets `q.grad = 1` and visits each `Value` only after
everything that uses it, so its gradient is complete before it is passed on.

In `y = a * a`, `a` reaches `y` along two routes, a **shared path**. Each
contributes `3 × 1 = 3`, and `+=` adds them: `a.grad` is 6, the slope `2a` of
`a²`. Predict: with `=` instead of `+=` in both lines, what would `a.grad` be?

<details>
<summary>Answer</summary>

3.0. `self` and `other_value` are the same `a`, so the second line
overwrites the first instead of adding to it. Adding with `+=` is why a
value used twice gets the gradient of both uses, and the same reason `db1`
sums over rows.


</details>

## Gate toolkit [#gate-toolkit]

F06's gate asks you to write the pipeline yourself. Beyond the steps above it
needs three rules; for `keepdims`, `axis=-1` and `P[np.arange(B), targets]`,
see the batch step in [F03](/foundations-03-probability).

1. **One-hot targets:** `Y = np.eye(C)[targets]`, from the
   [batch step](#from-one-row-to-a-batch).
2. **Copy before you nudge:** `up = W1.copy()`. `up = W1` copies nothing, so
   nudging `up` would change the real `W1` and every later number.
3. **Stay in float64**, NumPy's default. float32, from `dtype=np.float32` or
   from PyTorch tensors, makes a correct gradient check fail; `print(W1.dtype)`
   shows which one you have.

`artifacts/my-work/copy_demo.py`:

```python
import numpy as np

W = np.array([0.5, -0.5])
same = W                  # a second name for the same array
same[0] += 0.000001
print(W)                  # W changed too

W = np.array([0.5, -0.5])
up = W.copy()             # a separate array
up[0] += 0.000001
print(W)                  # W is untouched
```

Output:

```text
[ 0.500001 -0.5     ]
[ 0.5 -0.5]
```

<details>
<summary>Demo: float32 fails a correct gradient check</summary>

`artifacts/my-work/float_demo.py`:

```python
import numpy as np

print(np.float32(0.5) + np.float32(0.000001) - np.float32(0.5))
print(np.float64(0.5) + np.float64(0.000001) - np.float64(0.5))
```

Output:

```text
1.013279e-06
1.0000000000287557e-06
```

float32 keeps about 7 significant digits, float64 about 16, so in float32 a
nudge of 0.000001 on 0.5 is stored 1.3 % too large. Try it: add
`W1 = W1.astype(np.float32)` below `target = 1` in `one_example.py`. The
formula still divides by 0.000002, so every numeric estimate for an entry
of ±0.5 comes out 1.3 % too large; the max error becomes about 0.015, and
the check fails although every formula is right. Remove the line again.


</details>

## Completion evidence [#completion-evidence]

- steps 1–10 print the numbers shown, each gradient in its parameter's shape;
- the gradient check passes below `1e-6` and fails without the tanh slope;
- your batch gradients match `mlp_loss_and_gradients`;
- you can explain why `a.grad` is 6 for `y = a * a`.

## What's next [#whats-next]

Translate the mechanisms into framework code in
[F06 — PyTorch and checkpoint](/foundations-06-pytorch-checkpoint).

[← F04 — Neuron & gradient descent](/foundations-04-neuron) · [F06 — PyTorch & checkpoint →](/foundations-06-pytorch-checkpoint) · [Glossary](/glossary)
