F02 — Tensors and shapes
← F01 — Python & NumPy · F03 — Probability & loss → · Glossary
Learning outcome
You will predict the shape of an array before running the code, and write a small linear layer that rejects a wrong weight or bias with a clear message. On the way you meet two broadcasting mistakes: one that stops with an error and one that runs silently.
If you already use NumPy, skim the steps and go to the Build: about 30–45 minutes in all. If F01 was new to you, plan about 60–75 minutes for the steps and another 45–75 minutes for the Build, ideally in two sessions. These times are guides, not deadlines.
Shapes step by step
Each step asks you to predict before you run. The comment on each line is the
answer, so cover it first. Collect the blocks of all five steps in one file,
artifacts/my-work/shapes.py, each below the previous one, and run
python artifacts/my-work/shapes.py after adding each. When a step asks you to
try a line that fails, add it at the end, run, read the last line of the error,
then delete the line so the file runs again.
Shape and broadcasting
import numpy as np
a = np.array([[1, 2, 3], [4, 5, 6]])
row = np.array([10, 20, 30])
print(a.shape) # (2, 3): 2 rows, 3 columns
print(row.shape) # (3,): one axis with 3 entries
print((a + row).tolist()) # [[11, 22, 33], [14, 25, 36]]One number is a scalar, a row of numbers a vector, a table a matrix;
tensor is the general word for an array with any number of axes. shape
lists the length of each axis: (2, 3) is two rows of three. The comma in (3,)
marks a tuple with one item, so row has one axis. .tolist() turns an array
into a plain Python list that prints on one line.
a + row adds row to each row of a. This is broadcasting: NumPy lines
the two shapes up from the right, (2, 3) over (3,), and reuses the smaller
array along the axis it lacks. Compared axis by axis from the right, two sizes
must be equal, or one of them must be 1; a missing axis counts as 1.
Predict what print(a + np.array([10, 20])) does, then add it and run.
Check your prediction
It stops with
ValueError: operands could not be broadcast together with shapes (2,3) (2,).
From the right, 3 meets 2: they differ and neither is 1. Delete the line.
A concrete matrix multiplication
print((a * a).tolist()) # [[1, 4, 9], [16, 25, 36]]* multiplies entry by entry. @ is matrix multiplication, which combines
rows with columns:
x = np.array([[1, 2, 3], [0, 1, -1]])
w = np.array([[1, 0], [0, 1], [1, 1]])
print((x @ w).shape) # (2, 2)
print((x @ w).tolist()) # [[4, 5], [-1, 0]]The rule: entry (i, j) of x @ w is row i of x · column j of w. The dot
· is a dot product: multiply matching entries, then add. Top left, row 0 of
x with column 0 of w: 1×1 + 2×0 + 3×1 = 4. Top right, row 0 with column 1:
1×0 + 2×1 + 3×1 = 5. Work out the bottom row by hand before you run.
A row of x and a column of w must have the same length. x is (2, 3) and
w is (3, 2): the inner sizes match (3 and 3) and are summed away; the outer
sizes remain, so (2, 3) @ (3, 2) gives (2, 2). Predict whether
print(a @ a) works, then try it.
Check your prediction
It fails: (2, 3) @ (2, 3) puts 3 against 2 on the inside. NumPy’s message
ends in (size 2 is different from 3). Delete the line.
With square arrays, * and @ both work and give the same shape, but different
numbers. A shape check cannot tell them apart:
s = np.array([[1, 2], [3, 4]])
print((s * s).tolist()) # [[1, 4], [9, 16]]
print((s @ s).tolist()) # [[7, 10], [15, 22]]Reshape versus transpose
b = np.arange(6)
print(b.tolist()) # [0, 1, 2, 3, 4, 5]
print(b.reshape(2, 3).tolist()) # [[0, 1, 2], [3, 4, 5]]np.arange(6) counts from 0 to 5. reshape(2, 3) pours the same six values, in
the same reading order, into 2 rows of 3. Predict both lines below, then run:
print(b.reshape(3, 2).tolist()) # [[0, 1], [2, 3], [4, 5]]
print(b.reshape(2, 3).T.tolist()) # [[0, 3], [1, 4], [2, 5]]Both results have shape (3, 2), with different contents. reshape keeps the
reading order; .T (transpose) swaps rows and columns, so row 0 [0, 1, 2]
becomes column 0. Mixing them up gives the right shape with wrong values, and no
shape check notices.
Three axes: batch, time, channel
A language model works on several texts at once, and each text is a sequence of
token positions. Its arrays therefore have three axes, written [B, T, C]:
Batch (which example), Time (which token position) and Channel (the
numbers that describe one token). Later lessons also write D for the channel
width. Here are 2 examples, 3 positions and 4 channels:
t = np.arange(24).reshape(2, 3, 4)
print(t.shape) # (2, 3, 4)
print(t[0].shape) # (3, 4): example 0, all its positions
print(t[:, 0].shape) # (2, 4): position 0 of every example
print(t[0, 0].tolist()) # [0, 1, 2, 3]: the 4 channels of one tokenA number in the brackets picks one entry along that axis and removes the axis.
: means “all of this axis”, as in F01’s slices, so t[:, 0] keeps both
examples and picks position 0 in each. Predict the values, then add
print(t[:, 0].tolist()) and run.
Check your prediction
[[0, 1, 2, 3], [12, 13, 14, 15]]. Example 0 starts at 0; example 1 starts
after 3 × 4 = 12 values.
A linear layer: x @ W + b
A linear layer is the most common step in a neural network. It turns each
token’s channels into new channels with a weight matrix W, then adds a
bias b: one extra number per output channel, learned like the weights. As a
formula: linear(x) = x @ W + b. With x and w from the matrix multiplication
and the bias [10, 20]:
bias = np.array([10, 20])
print((x @ w + bias).tolist()) # [[14, 25], [9, 20]]x @ w is [[4, 5], [-1, 0]]; broadcasting adds 10 to the first column and 20
to the second. The same line works on [B, T, C] arrays: @ applies W at
every example and position, and only the last axis changes:
activations = np.zeros((4, 8, 32))
weight = np.zeros((32, 64))
bias = np.zeros(64)
print((activations @ weight).shape) # (4, 8, 64)
print((activations @ weight + bias).shape) # (4, 8, 64)np.zeros(shape) makes an array of zeros; only the shapes matter here. The bias
(64,) lines up with the last axis: one number per output channel, reused for
every example and position.
Now two wrong biases. Add
print((activations @ weight + np.zeros(8)).shape) and run: it stops with
operands could not be broadcast together with shapes (4,8,64) (8,). That
mistake is loud. Replace np.zeros(8) with np.zeros((8, 1)), predict, and run
again.
Check your prediction
It prints (4, 8, 64) with no error. From the right, 1 meets 64 (the 1
stretches) and 8 meets 8, so NumPy adds one number per position to all 64
channels. That is not a bias, and nothing complains. Only an explicit check
such as bias.shape == (weight.shape[1],) catches it. Delete the line.
Build and break
Work in one file, artifacts/my-work/test_f02.py: imports at the top, then your
function, then the tests. Write linear(x, weight, bias) that returns
x @ weight + bias, but first checks three things and raises ValueError with a
message that names the shapes:
weighthas two axes (weight.ndim == 2);- the last axis of
xhas the same size as the first axis ofweight; biashas shape(weight.shape[1],): one number per output channel.
Add four tests: the linear-layer example gives [[14, 25], [9, 20]]; a (4, 8, 32)
input with a (32, 64) weight gives shape (4, 8, 64); a transposed weight
np.zeros((32, 64)).T is rejected; the position bias np.zeros((8, 1)) is
rejected. Run python -m pytest artifacts/my-work/test_f02.py.
Inside pytest.raises the test passes quietly, so you never see your message.
To read it, delete the with pytest.raises(...) line of the transposed-weight
test for a moment and unindent the call below it. pytest then fails and shows
your message on the line that starts with E ValueError:. Say which two sizes
no longer agree, then put the line back.
A square weight, say (64, 64), keeps its shape when transposed, so no shape
check catches that mistake; only a test with known values, like the linear-layer
example, does. Stuck? Open the hint at the bottom.
Completion evidence
- you wrote each shape as a comment before running the line that prints it;
- you can say what reshape,
.Tand broadcasting each do, with an example; - the transposed weight fails inside
linearwith a message naming both shapes, not deep inside training; ruff check artifacts/my-workandpython -m pytest artifacts/my-workpass.
What’s next
Continue with F03 — Probability and loss.
Hint
x.shape[-1] is the last axis of x, weight.shape[0] the first axis of
weight. Compare with != (“not equal”) and raise, for example
raise ValueError(f"x {x.shape} does not fit weight {weight.shape}").
Each rejection test follows F01’s four-part skeleton: bad input,
pytest.raises, the call, the message check (see
Testing that bad input is rejected).
Reference answer
import numpy as np
import pytest
def linear(x, weight, bias):
"""Return x @ weight + bias after checking that the shapes fit."""
if weight.ndim != 2:
raise ValueError(f"weight must be 2D, got shape {weight.shape}")
if x.shape[-1] != weight.shape[0]:
raise ValueError(f"x {x.shape} does not fit weight {weight.shape}")
if bias.shape != (weight.shape[1],):
raise ValueError(f"bias {bias.shape} must be ({weight.shape[1]},)")
return x @ weight + bias
def test_worked_example():
x = np.array([[1, 2, 3], [0, 1, -1]])
weight = np.array([[1, 0], [0, 1], [1, 1]])
bias = np.array([10, 20])
assert linear(x, weight, bias).tolist() == [[14, 25], [9, 20]]
def test_batch_time_channel():
x = np.zeros((4, 8, 32))
result = linear(x, np.zeros((32, 64)), np.zeros(64))
assert result.shape == (4, 8, 64)
def test_rejects_transposed_weight():
x = np.zeros((4, 8, 32))
with pytest.raises(ValueError) as error:
linear(x, np.zeros((32, 64)).T, np.zeros(64))
assert "(64, 32)" in str(error.value)
def test_rejects_position_bias():
x = np.zeros((4, 8, 32))
with pytest.raises(ValueError) as error:
linear(x, np.zeros((32, 64)), np.zeros((8, 1)))
assert "(8, 1)" in str(error.value)The transposed weight gives x (4, 8, 32) does not fit weight (64, 32): the
32 channels of x meet the 64 rows of the transposed weight.
← F01 — Python & NumPy · F03 — Probability & loss → · Glossary