---
title: "F04 — Neuron and gradient descent"
sidebar:
  label: "F04 — Neuron & gradient descent"
---

<span id="f04-neuron-and-gradient-descent" />


[← F03 — Probability & loss](/foundations-03-probability) · [F05 — MLP, backprop & autograd →](/foundations-05-mlp-autograd) · [Glossary](/glossary)

## Learning outcome [#learning-outcome]

You will calculate a neuron's forward pass and gradients, then prove that the
chosen update lowers its loss.

No prior calculus is needed. This page uses three derivative rules, and you
will not derive any of them: you check each one by nudging a number and
watching the loss.

If you already write Python, plan 60–75 minutes. If programming is new, plan
about 2.5–3 hours over two sittings: the nudging and the rules in the first,
the four-point update, your training loop and the diagnosis in the second.
These times are guides, not deadlines.

## What a neuron and a derivative mean here [#what-a-neuron-and-a-derivative-mean-here]

Our neuron is a small function with adjustable numbers `w` (weight) and `b`
(bias). A forward pass computes a prediction `ŷ = wx + b` from input `x`. The
target `y` is the desired answer. Squaring the difference gives a nonnegative
error, the loss `L = (ŷ - y)²`.

Save each block below as the file named above it in `artifacts/my-work/`,
predict what it prints, then run it with `python artifacts/my-work/<file>`. The
comment on a line is the answer, so cover it first.

### Nudge and watch [#nudge-and-watch]

Take one point, `x=2` with target `y=3`, and the starting numbers `w=0.5`,
`b=0.1`. Nudge `w` by 0.001 and recompute:

| `w` | prediction `2w + 0.1` | loss `(ŷ - 3)²` |
| --- | --- | --- |
| 0.5 | 1.1 | 3.61 |
| 0.501 | 1.102 | 3.602404 |
| change: +0.001 | +0.002 | −0.007596 |

The loss changed by −0.007596 for a nudge of 0.001, so about **−7.6 per unit
of `w`**. That rate is the **derivative** `dL/dw`. Its sign says which way to
go: negative means raising `w` lowers the loss. `artifacts/my-work/nudge.py`
repeats the table (`**` raises to a power, so `** 2` squares):

```python
x, y, b = 2.0, 3.0, 0.1


def loss(w):
    prediction = w * x + b
    return (prediction - y) ** 2


before = loss(0.5)
after = loss(0.501)
print(round(before, 6))                     # 3.61
print(round(after, 6))                      # 3.602404
print(round(after - before, 6))             # -0.007596: the loss fell
print(round((after - before) / 0.001, 3))   # -7.596: change per unit of w
```

Now change `0.501` to `0.5001` and the `0.001` in the last line to `0.0001`:
the loss changes ten times less (`-0.00076`), and the rate stays at −7.6
(−7.5996 before rounding). The smaller the nudge, the closer the rate gets to
the derivative.

### Three rules, each checked by nudging [#three-rules-each-checked-by-nudging]

Nudging works for any number, but it costs one extra run per parameter. The
rules below give the same rate directly:

1. **Square:** `u²` changes at rate `2u`. At `u=3`: `2 × 3 = 6`. In the loss,
   `u` is the error `ŷ - y`.
2. **Line:** `ŷ = wx + b` changes at rate `x` when `w` moves, and at rate 1
   when `b` moves. Here `x=2`.
3. **Chain:** when a nudge passes through two steps (`w` changes `ŷ`, `ŷ`
   changes `L`), multiply their rates. `ŷ - y` is `-1.9`, so the square step
   has rate `2 × -1.9 = -3.8`; times the line's `2` gives `-7.6`.

So `dL/dw = 2(ŷ-y)x` and `dL/db = 2(ŷ-y)`. Check the first two rules with a
nudge in `artifacts/my-work/rules.py`. Its third line does not nudge: it
computes the chain rule's answer, which you compare with nudge.py's −7.596.

```python
h = 0.001
print(round((3.001**2 - 3**2) / h, 3))         # 6.001: square rule says 2 × 3 = 6
print(round((1.102 - 1.1) / h, 3))             # 2.0: line rule says x = 2
print(round(2 * (1.1 - 3) * 2, 3))             # -7.6: chain rule, as nudge.py found
```

A **gradient** collects one derivative for each adjustable parameter, here
`(dL/dw, dL/db)`. To lower the loss, move each parameter against its
derivative. The **learning rate** controls the step size.

## One complete update with numbers [#one-complete-update-with-numbers]

A scalar neuron predicts `ŷ = wx + b`. Use `x=2`, `w=0.5`, `b=0.1`, target
`y=3`, and squared loss `L=(ŷ-y)²`:

1. `ŷ = 0.5 × 2 + 0.1 = 1.1`.
2. Error is `1.1 - 3 = -1.9`; loss is `(-1.9)² = 3.61`.
3. `dL/dw = 2(ŷ-y)x = 2 × -1.9 × 2 = -7.6`.
4. `dL/db = 2(ŷ-y) = -3.8`.
5. With learning rate `0.1`, subtract the gradients: `w=1.26`, `b=0.48`.
6. The new prediction is `1.26 × 2 + 0.48 = 3.0`; loss is `0`.

The exact zero is a convenient choice of numbers for one point. With several
points, one update only lowers the loss; training repeats it many times and
need not reach zero.

The sign has visible meaning: both gradients are negative, so subtracting them
raises `w` and `b`, the direction the nudge showed. Adding them instead would
lower `w` and `b` and make the loss grow; you will break the code that way below.

## Four points: the gradient is an average [#four-points-the-gradient-is-an-average]

The course function `train_scalar_neuron` fits four points, `x = [-1, 0, 1, 2]`
with targets `y = 2x + 1 = [-1, 1, 3, 5]`. It starts from `w=0`, `b=0`, uses
learning rate `0.05` and takes 20 steps. With four points the loss is the
**mean** of the four squared errors, and each gradient is the mean of the four
per-point gradients. Step 1 by hand (`ŷ = 0` for every point at the start):

| `x` | `y` | error `ŷ - y` | squared | `2(ŷ-y)x` | `2(ŷ-y)` |
| --- | --- | --- | --- | --- | --- |
| −1 | −1 | 1 | 1 | −2 | 2 |
| 0 | 1 | −1 | 1 | 0 | −2 |
| 1 | 3 | −3 | 9 | −6 | −6 |
| 2 | 5 | −5 | 25 | −20 | −10 |
| **mean** | | | **9** | **−7** | **−4** |

Update: `w = 0 - 0.05 × -7 = 0.35` and `b = 0 - 0.05 × -4 = 0.2`. The new
predictions `0.35x + 0.2` give a mean loss of `6.04375`, down from 9.

Now write the same step in `artifacts/my-work/neuron_step.py`:

```python
import numpy as np

x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0                      # [-1, 1, 3, 5]
w, b, lr = 0.0, 0.0, 0.05
print(x.shape, y.shape, (w * x + b).shape)   # (4,) (4,) (4,): one per point

errors = w * x + b - y
print(np.mean(errors**2))              # 9.0: loss before
dw = np.mean(2 * errors * x)
db = np.mean(2 * errors)
print(dw, db)                          # -7.0 -4.0
w = w - lr * dw
b = b - lr * db
print(round(w, 4), round(b, 4))        # 0.35 0.2
print(np.mean((w * x + b - y) ** 2))   # 6.04375: loss after
```

`w * x + b - y` works on all four points at once (F02's broadcasting), so
input, target, and prediction all have shape `(4,)`; `np.mean` averages them
like `.mean()` in F03.

## Train and inspect [#train-and-inspect]

`artifacts/my-work/train.py` runs the course function with the same defaults:

```python
import numpy as np

from llm_course import train_scalar_neuron

x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
trace = train_scalar_neuron(x, y)
for step in range(4):
    w = round(trace.weights[step], 4)
    b = round(trace.biases[step], 4)
    print(step, w, b, round(trace.losses[step], 4))
print(len(trace.losses), round(trace.weight, 3), round(trace.bias, 3))
print(round(trace.losses[-1], 4))
```

Output:

```text
0 0.0 0.0 9.0
1 0.35 0.2 6.0438
2 0.6375 0.3625 4.0596
3 0.8738 0.4944 2.7278
21 1.94 1.019
0.0047
```

`range(4)` counts 0, 1, 2, 3, so the loop prints the first four steps.

The trace records weight, bias, and loss on every step: `losses[i]` is the loss
of `weights[i]` and `biases[i]`. Index 0 is the start, so 20 steps give 21
entries; `weight` and `bias` are the final values. Rows 0 and 1 are your hand
calculation. After 20 steps `w=1.94` and `b=1.019` are close to the true 2 and 1
but not there yet. A final number alone cannot explain a failure; the path can.

## Build: your own training loop [#build-your-own-training-loop]

`train_scalar_neuron` is only `neuron_step.py` repeated. Write it yourself in
`artifacts/my-work/train_loop.py`: repeat the update 20 times, collect every
weight, bias, and loss in three lists as the trace does, then check that your
lists match `trace.weights`, `trace.biases`, and `trace.losses`.
`np.allclose(a, b)` is `True` when two lists agree up to tiny rounding
differences. Expected:

```text
21 1.94 1.019 0.0047
True
True
True
```

<details>
<summary>Loop hint</summary>

`for _ in range(20):` repeats the indented lines 20 times; `_` names a
counter you do not use (ruff reports an unused `step`). Start with
`weights = [w]`, `biases = [b]` and `losses = []`. `losses.append(value)`
adds a value to the end of the list. Inside the loop, append the loss
*before* the update, and append `w` and `b` *after* it. After the loop, append the final loss once more: 21 entries each.


</details>

<details>
<summary>Reference loop</summary>

```python
import numpy as np

from llm_course import train_scalar_neuron

x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
w, b, lr = 0.0, 0.0, 0.05
weights = [w]
biases = [b]
losses = []

for _ in range(20):
    errors = w * x + b - y
    losses.append(np.mean(errors**2))
    dw = np.mean(2 * errors * x)
    db = np.mean(2 * errors)
    w = w - lr * dw
    b = b - lr * db
    weights.append(w)
    biases.append(b)
losses.append(np.mean((w * x + b - y) ** 2))

print(len(losses), round(w, 3), round(b, 3), round(losses[-1], 4))
trace = train_scalar_neuron(x, y)
print(np.allclose(weights, trace.weights))
print(np.allclose(biases, trace.biases))
print(np.allclose(losses, trace.losses))
```


</details>

## Break and diagnose [#break-and-diagnose]

Two broken runs, one with a wrong update sign and one with a learning rate that
is too large. Do not inspect the reference implementation yet.
`artifacts/my-work/diagnose.py`:

```python
import numpy as np

from llm_course import train_scalar_neuron


def show(trace):
    for step in range(6):
        print(step, round(trace.losses[step], 1), round(trace.weights[step], 2))
    print("after 20 steps:", round(trace.losses[-1]), round(trace.weight, 1))


x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
show(train_scalar_neuron(x, y, fault="wrong_gradient_sign"))
print("---")
show(train_scalar_neuron(x, y, learning_rate=0.6))
```

Before running, predict for each run: does the loss rise, and does `w` move
steadily or jump back and forth? Then run it and compare.

<details>
<summary>Output (step, loss, weight)</summary>

```text
0 9.0 0.0
1 12.5 -0.35
2 17.5 -0.76
3 24.4 -1.25
4 34.0 -1.82
5 47.4 -2.5
after 20 steps: 6942 -51.1
---
0 9.0 0.0
1 12.3 4.2
2 16.9 -0.6
3 23.1 5.04
4 31.7 -1.56
5 43.4 6.17
after 20 steps: 4926 -42.4
```


</details>

Both losses explode from 9 to thousands; the loss alone cannot tell them apart.
The weight path can. Use only the loss curve and the parameter movement to name,
for each run, the mechanism, the evidence, and the repair.

<details>
<summary>Reference diagnosis</summary>

**Wrong sign.** `w` drifts steadily away from the target 2: −0.35, −0.76,
−1.25. Every step goes the wrong way, so the update adds the gradient
instead of subtracting it. Repair: `w = w - lr * dw` (and the same for `b`).

**Step too big.** `w` jumps across the target 2 and lands further away each
time: 4.2, −0.6, 5.04, −1.56. The direction is right, the step overshoots.
Repair: a smaller learning rate, such as the default 0.05.


</details>

## Completion evidence [#completion-evidence]

- your nudges agree with the three rules (−7.6, 6, 2);
- the hand-worked four-point step, `neuron_step.py`, and rows 0–1 of the trace
  agree (9 → 6.04375, `w=0.35`, `b=0.2`);
- your `train_loop.py` prints `True` three times;
- input, target, and prediction shapes match;
- healthy loss decreases and both broken losses increase;
- your diagnosis names mechanism, evidence (loss and weight path), and repair
  for each broken run;
- `ruff check artifacts/my-work` passes.

## What's next [#whats-next]

Continue with [F05 — MLP, backpropagation, and autograd](/foundations-05-mlp-autograd).

[← F03 — Probability & loss](/foundations-03-probability) · [F05 — MLP, backprop & autograd →](/foundations-05-mlp-autograd) · [Glossary](/glossary)
