Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

F04 — Neuron and gradient descent

← F03 — Probability & loss · F05 — MLP, backprop & autograd → · Glossary

Learning outcome

You will calculate a neuron’s forward pass and gradients, then prove that the chosen update lowers its loss.

No prior calculus is needed. This page uses three derivative rules, and you will not derive any of them: you check each one by nudging a number and watching the loss.

If you already write Python, plan 60–75 minutes. If programming is new, plan about 2.5–3 hours over two sittings: the nudging and the rules in the first, the four-point update, your training loop and the diagnosis in the second. These times are guides, not deadlines.

What a neuron and a derivative mean here

Our neuron is a small function with adjustable numbers w (weight) and b (bias). A forward pass computes a prediction ŷ = wx + b from input x. The target y is the desired answer. Squaring the difference gives a nonnegative error, the loss L = (ŷ - y)².

Save each block below as the file named above it in artifacts/my-work/, predict what it prints, then run it with python artifacts/my-work/<file>. The comment on a line is the answer, so cover it first.

Nudge and watch

Take one point, x=2 with target y=3, and the starting numbers w=0.5, b=0.1. Nudge w by 0.001 and recompute:

w prediction 2w + 0.1 loss (ŷ - 3)²
0.5 1.1 3.61
0.501 1.102 3.602404
change: +0.001 +0.002 −0.007596

The loss changed by −0.007596 for a nudge of 0.001, so about −7.6 per unit of w. That rate is the derivative dL/dw. Its sign says which way to go: negative means raising w lowers the loss. artifacts/my-work/nudge.py repeats the table (** raises to a power, so ** 2 squares):

x, y, b = 2.0, 3.0, 0.1


def loss(w):
    prediction = w * x + b
    return (prediction - y) ** 2


before = loss(0.5)
after = loss(0.501)
print(round(before, 6))                     # 3.61
print(round(after, 6))                      # 3.602404
print(round(after - before, 6))             # -0.007596: the loss fell
print(round((after - before) / 0.001, 3))   # -7.596: change per unit of w

Now change 0.501 to 0.5001 and the 0.001 in the last line to 0.0001: the loss changes ten times less (-0.00076), and the rate stays at −7.6 (−7.5996 before rounding). The smaller the nudge, the closer the rate gets to the derivative.

Three rules, each checked by nudging

Nudging works for any number, but it costs one extra run per parameter. The rules below give the same rate directly:

  1. Square: u² changes at rate 2u. At u=3: 2 × 3 = 6. In the loss, u is the error ŷ - y.
  2. Line: ŷ = wx + b changes at rate x when w moves, and at rate 1 when b moves. Here x=2.
  3. Chain: when a nudge passes through two steps (w changes ŷ, ŷ changes L), multiply their rates. ŷ - y is -1.9, so the square step has rate 2 × -1.9 = -3.8; times the line’s 2 gives -7.6.

So dL/dw = 2(ŷ-y)x and dL/db = 2(ŷ-y). Check the first two rules with a nudge in artifacts/my-work/rules.py. Its third line does not nudge: it computes the chain rule’s answer, which you compare with nudge.py’s −7.596.

h = 0.001
print(round((3.001**2 - 3**2) / h, 3))         # 6.001: square rule says 2 × 3 = 6
print(round((1.102 - 1.1) / h, 3))             # 2.0: line rule says x = 2
print(round(2 * (1.1 - 3) * 2, 3))             # -7.6: chain rule, as nudge.py found

A gradient collects one derivative for each adjustable parameter, here (dL/dw, dL/db). To lower the loss, move each parameter against its derivative. The learning rate controls the step size.

One complete update with numbers

A scalar neuron predicts ŷ = wx + b. Use x=2, w=0.5, b=0.1, target y=3, and squared loss L=(ŷ-y)²:

  1. ŷ = 0.5 × 2 + 0.1 = 1.1.
  2. Error is 1.1 - 3 = -1.9; loss is (-1.9)² = 3.61.
  3. dL/dw = 2(ŷ-y)x = 2 × -1.9 × 2 = -7.6.
  4. dL/db = 2(ŷ-y) = -3.8.
  5. With learning rate 0.1, subtract the gradients: w=1.26, b=0.48.
  6. The new prediction is 1.26 × 2 + 0.48 = 3.0; loss is 0.

The exact zero is a convenient choice of numbers for one point. With several points, one update only lowers the loss; training repeats it many times and need not reach zero.

The sign has visible meaning: both gradients are negative, so subtracting them raises w and b, the direction the nudge showed. Adding them instead would lower w and b and make the loss grow; you will break the code that way below.

Four points: the gradient is an average

The course function train_scalar_neuron fits four points, x = [-1, 0, 1, 2] with targets y = 2x + 1 = [-1, 1, 3, 5]. It starts from w=0, b=0, uses learning rate 0.05 and takes 20 steps. With four points the loss is the mean of the four squared errors, and each gradient is the mean of the four per-point gradients. Step 1 by hand (ŷ = 0 for every point at the start):

x y error ŷ - y squared 2(ŷ-y)x 2(ŷ-y)
−1 −1 1 1 −2 2
0 1 −1 1 0 −2
1 3 −3 9 −6 −6
2 5 −5 25 −20 −10
mean 9 −7 −4

Update: w = 0 - 0.05 × -7 = 0.35 and b = 0 - 0.05 × -4 = 0.2. The new predictions 0.35x + 0.2 give a mean loss of 6.04375, down from 9.

Now write the same step in artifacts/my-work/neuron_step.py:

import numpy as np

x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0                      # [-1, 1, 3, 5]
w, b, lr = 0.0, 0.0, 0.05
print(x.shape, y.shape, (w * x + b).shape)   # (4,) (4,) (4,): one per point

errors = w * x + b - y
print(np.mean(errors**2))              # 9.0: loss before
dw = np.mean(2 * errors * x)
db = np.mean(2 * errors)
print(dw, db)                          # -7.0 -4.0
w = w - lr * dw
b = b - lr * db
print(round(w, 4), round(b, 4))        # 0.35 0.2
print(np.mean((w * x + b - y) ** 2))   # 6.04375: loss after

w * x + b - y works on all four points at once (F02’s broadcasting), so input, target, and prediction all have shape (4,); np.mean averages them like .mean() in F03.

Train and inspect

artifacts/my-work/train.py runs the course function with the same defaults:

import numpy as np

from llm_course import train_scalar_neuron

x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
trace = train_scalar_neuron(x, y)
for step in range(4):
    w = round(trace.weights[step], 4)
    b = round(trace.biases[step], 4)
    print(step, w, b, round(trace.losses[step], 4))
print(len(trace.losses), round(trace.weight, 3), round(trace.bias, 3))
print(round(trace.losses[-1], 4))

Output:

0 0.0 0.0 9.0
1 0.35 0.2 6.0438
2 0.6375 0.3625 4.0596
3 0.8738 0.4944 2.7278
21 1.94 1.019
0.0047

range(4) counts 0, 1, 2, 3, so the loop prints the first four steps.

The trace records weight, bias, and loss on every step: losses[i] is the loss of weights[i] and biases[i]. Index 0 is the start, so 20 steps give 21 entries; weight and bias are the final values. Rows 0 and 1 are your hand calculation. After 20 steps w=1.94 and b=1.019 are close to the true 2 and 1 but not there yet. A final number alone cannot explain a failure; the path can.

Build: your own training loop

train_scalar_neuron is only neuron_step.py repeated. Write it yourself in artifacts/my-work/train_loop.py: repeat the update 20 times, collect every weight, bias, and loss in three lists as the trace does, then check that your lists match trace.weights, trace.biases, and trace.losses. np.allclose(a, b) is True when two lists agree up to tiny rounding differences. Expected:

21 1.94 1.019 0.0047
True
True
True
Loop hint

for _ in range(20): repeats the indented lines 20 times; _ names a counter you do not use (ruff reports an unused step). Start with weights = [w], biases = [b] and losses = []. losses.append(value) adds a value to the end of the list. Inside the loop, append the loss before the update, and append w and b after it. After the loop, append the final loss once more: 21 entries each.

Reference loop
import numpy as np

from llm_course import train_scalar_neuron

x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
w, b, lr = 0.0, 0.0, 0.05
weights = [w]
biases = [b]
losses = []

for _ in range(20):
    errors = w * x + b - y
    losses.append(np.mean(errors**2))
    dw = np.mean(2 * errors * x)
    db = np.mean(2 * errors)
    w = w - lr * dw
    b = b - lr * db
    weights.append(w)
    biases.append(b)
losses.append(np.mean((w * x + b - y) ** 2))

print(len(losses), round(w, 3), round(b, 3), round(losses[-1], 4))
trace = train_scalar_neuron(x, y)
print(np.allclose(weights, trace.weights))
print(np.allclose(biases, trace.biases))
print(np.allclose(losses, trace.losses))

Break and diagnose

Two broken runs, one with a wrong update sign and one with a learning rate that is too large. Do not inspect the reference implementation yet. artifacts/my-work/diagnose.py:

import numpy as np

from llm_course import train_scalar_neuron


def show(trace):
    for step in range(6):
        print(step, round(trace.losses[step], 1), round(trace.weights[step], 2))
    print("after 20 steps:", round(trace.losses[-1]), round(trace.weight, 1))


x = np.array([-1.0, 0.0, 1.0, 2.0])
y = 2.0 * x + 1.0
show(train_scalar_neuron(x, y, fault="wrong_gradient_sign"))
print("---")
show(train_scalar_neuron(x, y, learning_rate=0.6))

Before running, predict for each run: does the loss rise, and does w move steadily or jump back and forth? Then run it and compare.

Output (step, loss, weight)
0 9.0 0.0
1 12.5 -0.35
2 17.5 -0.76
3 24.4 -1.25
4 34.0 -1.82
5 47.4 -2.5
after 20 steps: 6942 -51.1
---
0 9.0 0.0
1 12.3 4.2
2 16.9 -0.6
3 23.1 5.04
4 31.7 -1.56
5 43.4 6.17
after 20 steps: 4926 -42.4

Both losses explode from 9 to thousands; the loss alone cannot tell them apart. The weight path can. Use only the loss curve and the parameter movement to name, for each run, the mechanism, the evidence, and the repair.

Reference diagnosis

Wrong sign. w drifts steadily away from the target 2: −0.35, −0.76, −1.25. Every step goes the wrong way, so the update adds the gradient instead of subtracting it. Repair: w = w - lr * dw (and the same for b).

Step too big. w jumps across the target 2 and lands further away each time: 4.2, −0.6, 5.04, −1.56. The direction is right, the step overshoots. Repair: a smaller learning rate, such as the default 0.05.

Completion evidence

  • your nudges agree with the three rules (−7.6, 6, 2);
  • the hand-worked four-point step, neuron_step.py, and rows 0–1 of the trace agree (9 → 6.04375, w=0.35, b=0.2);
  • your train_loop.py prints True three times;
  • input, target, and prediction shapes match;
  • healthy loss decreases and both broken losses increase;
  • your diagnosis names mechanism, evidence (loss and weight path), and repair for each broken run;
  • ruff check artifacts/my-work passes.

What’s next

Continue with F05 — MLP, backpropagation, and autograd.

← F03 — Probability & loss · F05 — MLP, backprop & autograd → · Glossary

Was this page helpful?