F04 — Neuron and gradient descent
← F03 — Probability & loss · F05 — MLP, backprop & autograd → · Glossary
Learning outcome
You will calculate a neuron’s forward pass and gradients, then prove that the chosen update lowers its loss.
No prior calculus is needed. This page uses three derivative rules, and you will not derive any of them: you check each one by nudging a number and watching the loss.
If you already write Python, plan 60–75 minutes. If programming is new, plan about 2.5–3 hours over two sittings: the nudging and the rules in the first, the four-point update, your training loop and the diagnosis in the second. These times are guides, not deadlines.
What a neuron and a derivative mean here
Our neuron is a small function with adjustable numbers w (weight) and b
(bias). A forward pass computes a prediction ŷ = wx + b from input x. The
target y is the desired answer. Squaring the difference gives a nonnegative
error, the loss L = (ŷ - y)².
Save each block below as the file named above it in artifacts/my-work/,
predict what it prints, then run it with python artifacts/my-work/<file>. The
comment on a line is the answer, so cover it first.
Nudge and watch
Take one point, x=2 with target y=3, and the starting numbers w=0.5,
b=0.1. Nudge w by 0.001 and recompute:
w |
prediction 2w + 0.1 |
loss (ŷ - 3)² |
|---|---|---|
| 0.5 | 1.1 | 3.61 |
| 0.501 | 1.102 | 3.602404 |
| change: +0.001 | +0.002 | −0.007596 |
The loss changed by −0.007596 for a nudge of 0.001, so about −7.6 per unit
of w. That rate is the derivative dL/dw. Its sign says which way to
go: negative means raising w lowers the loss. artifacts/my-work/nudge.py
repeats the table (** raises to a power, so ** 2 squares):
x, y, b = 2.0, 3.0, 0.1
def loss(w):
prediction = w * x + b
return (prediction - y) ** 2
before = loss(0.5)
after = loss(0.501)
print(round(before, 6)) # 3.61
print(round(after, 6)) # 3.602404
print(round(after - before, 6)) # -0.007596: the loss fell
print(round((after - before) / 0.001, 3)) # -7.596: change per unit of wNow change 0.501 to 0.5001 and the 0.001 in the last line to 0.0001:
the loss changes ten times less (-0.00076), and the rate stays at −7.6
(−7.5996 before rounding). The smaller the nudge, the closer the rate gets to
the derivative.
Three rules, each checked by nudging
Nudging works for any number, but it costs one extra run per parameter. The rules below give the same rate directly:
- Square:
u²changes at rate2u. Atu=3:2 × 3 = 6. In the loss,uis the errorŷ - y. - Line:
ŷ = wx + bchanges at ratexwhenwmoves, and at rate 1 whenbmoves. Herex=2. - Chain: when a nudge passes through two steps (
wchangesŷ,ŷchangesL), multiply their rates.ŷ - yis-1.9, so the square step has rate2 × -1.9 = -3.8; times the line’s2gives-7.6.
So dL/dw = 2(ŷ-y)x and dL/db = 2(ŷ-y). Check the first two rules with a
nudge in artifacts/my-work/rules.py. Its third line does not nudge: it
computes the chain rule’s answer, which you compare with nudge.py’s −7.596.
h = 0.001
print(round((3.001**2 - 3**2) / h, 3)) # 6.001: square rule says 2 × 3 = 6
print(round((1.102 - 1.1) / h, 3)) # 2.0: line rule says x = 2
print(round(2 * (1.1 - 3) * 2, 3)) # -7.6: chain rule, as nudge.py foundA gradient collects one derivative for each adjustable parameter, here
(dL/dw, dL/db). To lower the loss, move each parameter against its
derivative. The learning rate controls the step size.
One complete update with numbers
A scalar neuron predicts ŷ = wx + b. Use x=2, w=0.5, b=0.1, target
y=3, and squared loss L=(ŷ-y)²:
ŷ = 0.5 × 2 + 0.1 = 1.1.- Error is
1.1 - 3 = -1.9; loss is(-1.9)² = 3.61. dL/dw = 2(ŷ-y)x = 2 × -1.9 × 2 = -7.6.dL/db = 2(ŷ-y) = -3.8.- With learning rate
0.1, subtract the gradients:w=1.26,b=0.48. - The new prediction is
1.26 × 2 + 0.48 = 3.0; loss is0.
The exact zero is a convenient choice of numbers for one point. With several points, one update only lowers the loss; training repeats it many times and need not reach zero.
The sign has visible meaning: both gradients are negative, so subtracting them
raises w and b, the direction the nudge showed. Adding them instead would
lower w and b and make the loss grow; you will break the code that way below.
Four points: the gradient is an average
The course function train_scalar_neuron fits four points, x = [-1, 0, 1, 2]
with targets y = 2x + 1 = [-1, 1, 3, 5]. It starts from w=0, b=0, uses
learning rate 0.05 and takes 20 steps. With four points the loss is the
mean of the four squared errors, and each gradient is the mean of the four
per-point gradients. Step 1 by hand (ŷ = 0 for every point at the start):
x |
y |
error ŷ - y |
squared | 2(ŷ-y)x |
2(ŷ-y) |
|---|---|---|---|---|---|
| −1 | −1 | 1 | 1 | −2 | 2 |
| 0 | 1 | −1 | 1 | 0 | −2 |
| 1 | 3 | −3 | 9 | −6 | −6 |
| 2 | 5 | −5 | 25 | −20 | −10 |
| mean | 9 | −7 | −4 |
Update: w = 0 - 0.05 × -7 = 0.35 and b = 0 - 0.05 × -4 = 0.2. The new
predictions 0.35x + 0.2 give a mean loss of 6.04375, down from 9.
Now write the same step in artifacts/my-work/neuron_step.py:
import numpy as np
x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0 # [-1, 1, 3, 5]
w, b, lr = 0.0, 0.0, 0.05
print(x.shape, y.shape, (w * x + b).shape) # (4,) (4,) (4,): one per point
errors = w * x + b - y
print(np.mean(errors**2)) # 9.0: loss before
dw = np.mean(2 * errors * x)
db = np.mean(2 * errors)
print(dw, db) # -7.0 -4.0
w = w - lr * dw
b = b - lr * db
print(round(w, 4), round(b, 4)) # 0.35 0.2
print(np.mean((w * x + b - y) ** 2)) # 6.04375: loss afterw * x + b - y works on all four points at once (F02’s broadcasting), so
input, target, and prediction all have shape (4,); np.mean averages them
like .mean() in F03.
Train and inspect
artifacts/my-work/train.py runs the course function with the same defaults:
import numpy as np
from llm_course import train_scalar_neuron
x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
trace = train_scalar_neuron(x, y)
for step in range(4):
w = round(trace.weights[step], 4)
b = round(trace.biases[step], 4)
print(step, w, b, round(trace.losses[step], 4))
print(len(trace.losses), round(trace.weight, 3), round(trace.bias, 3))
print(round(trace.losses[-1], 4))Output:
0 0.0 0.0 9.0
1 0.35 0.2 6.0438
2 0.6375 0.3625 4.0596
3 0.8738 0.4944 2.7278
21 1.94 1.019
0.0047range(4) counts 0, 1, 2, 3, so the loop prints the first four steps.
The trace records weight, bias, and loss on every step: losses[i] is the loss
of weights[i] and biases[i]. Index 0 is the start, so 20 steps give 21
entries; weight and bias are the final values. Rows 0 and 1 are your hand
calculation. After 20 steps w=1.94 and b=1.019 are close to the true 2 and 1
but not there yet. A final number alone cannot explain a failure; the path can.
Build: your own training loop
train_scalar_neuron is only neuron_step.py repeated. Write it yourself in
artifacts/my-work/train_loop.py: repeat the update 20 times, collect every
weight, bias, and loss in three lists as the trace does, then check that your
lists match trace.weights, trace.biases, and trace.losses.
np.allclose(a, b) is True when two lists agree up to tiny rounding
differences. Expected:
21 1.94 1.019 0.0047
True
True
TrueLoop hint
for _ in range(20): repeats the indented lines 20 times; _ names a
counter you do not use (ruff reports an unused step). Start with
weights = [w], biases = [b] and losses = []. losses.append(value)
adds a value to the end of the list. Inside the loop, append the loss
before the update, and append w and b after it. After the loop, append the final loss once more: 21 entries each.
Reference loop
import numpy as np
from llm_course import train_scalar_neuron
x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
w, b, lr = 0.0, 0.0, 0.05
weights = [w]
biases = [b]
losses = []
for _ in range(20):
errors = w * x + b - y
losses.append(np.mean(errors**2))
dw = np.mean(2 * errors * x)
db = np.mean(2 * errors)
w = w - lr * dw
b = b - lr * db
weights.append(w)
biases.append(b)
losses.append(np.mean((w * x + b - y) ** 2))
print(len(losses), round(w, 3), round(b, 3), round(losses[-1], 4))
trace = train_scalar_neuron(x, y)
print(np.allclose(weights, trace.weights))
print(np.allclose(biases, trace.biases))
print(np.allclose(losses, trace.losses))Break and diagnose
Two broken runs, one with a wrong update sign and one with a learning rate that
is too large. Do not inspect the reference implementation yet.
artifacts/my-work/diagnose.py:
import numpy as np
from llm_course import train_scalar_neuron
def show(trace):
for step in range(6):
print(step, round(trace.losses[step], 1), round(trace.weights[step], 2))
print("after 20 steps:", round(trace.losses[-1]), round(trace.weight, 1))
x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
show(train_scalar_neuron(x, y, fault="wrong_gradient_sign"))
print("---")
show(train_scalar_neuron(x, y, learning_rate=0.6))Before running, predict for each run: does the loss rise, and does w move
steadily or jump back and forth? Then run it and compare.
Output (step, loss, weight)
0 9.0 0.0
1 12.5 -0.35
2 17.5 -0.76
3 24.4 -1.25
4 34.0 -1.82
5 47.4 -2.5
after 20 steps: 6942 -51.1
---
0 9.0 0.0
1 12.3 4.2
2 16.9 -0.6
3 23.1 5.04
4 31.7 -1.56
5 43.4 6.17
after 20 steps: 4926 -42.4Both losses explode from 9 to thousands; the loss alone cannot tell them apart. The weight path can. Use only the loss curve and the parameter movement to name, for each run, the mechanism, the evidence, and the repair.
Reference diagnosis
Wrong sign. w drifts steadily away from the target 2: −0.35, −0.76,
−1.25. Every step goes the wrong way, so the update adds the gradient
instead of subtracting it. Repair: w = w - lr * dw (and the same for b).
Step too big. w jumps across the target 2 and lands further away each
time: 4.2, −0.6, 5.04, −1.56. The direction is right, the step overshoots.
Repair: a smaller learning rate, such as the default 0.05.
Completion evidence
- your nudges agree with the three rules (−7.6, 6, 2);
- the hand-worked four-point step,
neuron_step.py, and rows 0–1 of the trace agree (9 → 6.04375,w=0.35,b=0.2); - your
train_loop.pyprintsTruethree times; - input, target, and prediction shapes match;
- healthy loss decreases and both broken losses increase;
- your diagnosis names mechanism, evidence (loss and weight path), and repair for each broken run;
ruff check artifacts/my-workpasses.
What’s next
Continue with F05 — MLP, backpropagation, and autograd.
← F03 — Probability & loss · F05 — MLP, backprop & autograd → · Glossary