Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

F02 — Tensors and shapes

← F01 — Python & NumPy · F03 — Probability & loss → · Glossary

Learning outcome

You will predict the shape of an array before running the code, and write a small linear layer that rejects a wrong weight or bias with a clear message. On the way you meet two broadcasting mistakes: one that stops with an error and one that runs silently.

If you already use NumPy, skim the steps and go to the Build: about 30–45 minutes in all. If F01 was new to you, plan about 60–75 minutes for the steps and another 45–75 minutes for the Build, ideally in two sessions. These times are guides, not deadlines.

Shapes step by step

Each step asks you to predict before you run. The comment on each line is the answer, so cover it first. Collect the blocks of all five steps in one file, artifacts/my-work/shapes.py, each below the previous one, and run python artifacts/my-work/shapes.py after adding each. When a step asks you to try a line that fails, add it at the end, run, read the last line of the error, then delete the line so the file runs again.

Shape and broadcasting

import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6]])
row = np.array([10, 20, 30])
print(a.shape)              # (2, 3): 2 rows, 3 columns
print(row.shape)            # (3,): one axis with 3 entries
print((a + row).tolist())   # [[11, 22, 33], [14, 25, 36]]

One number is a scalar, a row of numbers a vector, a table a matrix; tensor is the general word for an array with any number of axes. shape lists the length of each axis: (2, 3) is two rows of three. The comma in (3,) marks a tuple with one item, so row has one axis. .tolist() turns an array into a plain Python list that prints on one line.

a + row adds row to each row of a. This is broadcasting: NumPy lines the two shapes up from the right, (2, 3) over (3,), and reuses the smaller array along the axis it lacks. Compared axis by axis from the right, two sizes must be equal, or one of them must be 1; a missing axis counts as 1.

Predict what print(a + np.array([10, 20])) does, then add it and run.

Check your prediction

It stops with ValueError: operands could not be broadcast together with shapes (2,3) (2,). From the right, 3 meets 2: they differ and neither is 1. Delete the line.

A concrete matrix multiplication

print((a * a).tolist())     # [[1, 4, 9], [16, 25, 36]]

* multiplies entry by entry. @ is matrix multiplication, which combines rows with columns:

x = np.array([[1, 2, 3], [0, 1, -1]])
w = np.array([[1, 0], [0, 1], [1, 1]])
print((x @ w).shape)        # (2, 2)
print((x @ w).tolist())     # [[4, 5], [-1, 0]]

The rule: entry (i, j) of x @ w is row i of x · column j of w. The dot · is a dot product: multiply matching entries, then add. Top left, row 0 of x with column 0 of w: 1×1 + 2×0 + 3×1 = 4. Top right, row 0 with column 1: 1×0 + 2×1 + 3×1 = 5. Work out the bottom row by hand before you run.

A row of x and a column of w must have the same length. x is (2, 3) and w is (3, 2): the inner sizes match (3 and 3) and are summed away; the outer sizes remain, so (2, 3) @ (3, 2) gives (2, 2). Predict whether print(a @ a) works, then try it.

Check your prediction

It fails: (2, 3) @ (2, 3) puts 3 against 2 on the inside. NumPy’s message ends in (size 2 is different from 3). Delete the line.

With square arrays, * and @ both work and give the same shape, but different numbers. A shape check cannot tell them apart:

s = np.array([[1, 2], [3, 4]])
print((s * s).tolist())     # [[1, 4], [9, 16]]
print((s @ s).tolist())     # [[7, 10], [15, 22]]

Reshape versus transpose

b = np.arange(6)
print(b.tolist())                   # [0, 1, 2, 3, 4, 5]
print(b.reshape(2, 3).tolist())     # [[0, 1, 2], [3, 4, 5]]

np.arange(6) counts from 0 to 5. reshape(2, 3) pours the same six values, in the same reading order, into 2 rows of 3. Predict both lines below, then run:

print(b.reshape(3, 2).tolist())     # [[0, 1], [2, 3], [4, 5]]
print(b.reshape(2, 3).T.tolist())   # [[0, 3], [1, 4], [2, 5]]

Both results have shape (3, 2), with different contents. reshape keeps the reading order; .T (transpose) swaps rows and columns, so row 0 [0, 1, 2] becomes column 0. Mixing them up gives the right shape with wrong values, and no shape check notices.

Three axes: batch, time, channel

A language model works on several texts at once, and each text is a sequence of token positions. Its arrays therefore have three axes, written [B, T, C]: Batch (which example), Time (which token position) and Channel (the numbers that describe one token). Later lessons also write D for the channel width. Here are 2 examples, 3 positions and 4 channels:

t = np.arange(24).reshape(2, 3, 4)
print(t.shape)              # (2, 3, 4)
print(t[0].shape)           # (3, 4): example 0, all its positions
print(t[:, 0].shape)        # (2, 4): position 0 of every example
print(t[0, 0].tolist())     # [0, 1, 2, 3]: the 4 channels of one token

A number in the brackets picks one entry along that axis and removes the axis. : means “all of this axis”, as in F01’s slices, so t[:, 0] keeps both examples and picks position 0 in each. Predict the values, then add print(t[:, 0].tolist()) and run.

Check your prediction

[[0, 1, 2, 3], [12, 13, 14, 15]]. Example 0 starts at 0; example 1 starts after 3 × 4 = 12 values.

A linear layer: x @ W + b

A linear layer is the most common step in a neural network. It turns each token’s channels into new channels with a weight matrix W, then adds a bias b: one extra number per output channel, learned like the weights. As a formula: linear(x) = x @ W + b. With x and w from the matrix multiplication and the bias [10, 20]:

bias = np.array([10, 20])
print((x @ w + bias).tolist())   # [[14, 25], [9, 20]]

x @ w is [[4, 5], [-1, 0]]; broadcasting adds 10 to the first column and 20 to the second. The same line works on [B, T, C] arrays: @ applies W at every example and position, and only the last axis changes:

activations = np.zeros((4, 8, 32))
weight = np.zeros((32, 64))
bias = np.zeros(64)
print((activations @ weight).shape)          # (4, 8, 64)
print((activations @ weight + bias).shape)   # (4, 8, 64)

np.zeros(shape) makes an array of zeros; only the shapes matter here. The bias (64,) lines up with the last axis: one number per output channel, reused for every example and position.

Now two wrong biases. Add print((activations @ weight + np.zeros(8)).shape) and run: it stops with operands could not be broadcast together with shapes (4,8,64) (8,). That mistake is loud. Replace np.zeros(8) with np.zeros((8, 1)), predict, and run again.

Check your prediction

It prints (4, 8, 64) with no error. From the right, 1 meets 64 (the 1 stretches) and 8 meets 8, so NumPy adds one number per position to all 64 channels. That is not a bias, and nothing complains. Only an explicit check such as bias.shape == (weight.shape[1],) catches it. Delete the line.

Build and break

Work in one file, artifacts/my-work/test_f02.py: imports at the top, then your function, then the tests. Write linear(x, weight, bias) that returns x @ weight + bias, but first checks three things and raises ValueError with a message that names the shapes:

  • weight has two axes (weight.ndim == 2);
  • the last axis of x has the same size as the first axis of weight;
  • bias has shape (weight.shape[1],): one number per output channel.

Add four tests: the linear-layer example gives [[14, 25], [9, 20]]; a (4, 8, 32) input with a (32, 64) weight gives shape (4, 8, 64); a transposed weight np.zeros((32, 64)).T is rejected; the position bias np.zeros((8, 1)) is rejected. Run python -m pytest artifacts/my-work/test_f02.py.

Inside pytest.raises the test passes quietly, so you never see your message. To read it, delete the with pytest.raises(...) line of the transposed-weight test for a moment and unindent the call below it. pytest then fails and shows your message on the line that starts with E ValueError:. Say which two sizes no longer agree, then put the line back.

A square weight, say (64, 64), keeps its shape when transposed, so no shape check catches that mistake; only a test with known values, like the linear-layer example, does. Stuck? Open the hint at the bottom.

Completion evidence

  • you wrote each shape as a comment before running the line that prints it;
  • you can say what reshape, .T and broadcasting each do, with an example;
  • the transposed weight fails inside linear with a message naming both shapes, not deep inside training;
  • ruff check artifacts/my-work and python -m pytest artifacts/my-work pass.

What’s next

Continue with F03 — Probability and loss.

Hint

x.shape[-1] is the last axis of x, weight.shape[0] the first axis of weight. Compare with != (“not equal”) and raise, for example raise ValueError(f"x {x.shape} does not fit weight {weight.shape}"). Each rejection test follows F01’s four-part skeleton: bad input, pytest.raises, the call, the message check (see Testing that bad input is rejected).

Reference answer
import numpy as np
import pytest


def linear(x, weight, bias):
    """Return x @ weight + bias after checking that the shapes fit."""
    if weight.ndim != 2:
        raise ValueError(f"weight must be 2D, got shape {weight.shape}")
    if x.shape[-1] != weight.shape[0]:
        raise ValueError(f"x {x.shape} does not fit weight {weight.shape}")
    if bias.shape != (weight.shape[1],):
        raise ValueError(f"bias {bias.shape} must be ({weight.shape[1]},)")
    return x @ weight + bias


def test_worked_example():
    x = np.array([[1, 2, 3], [0, 1, -1]])
    weight = np.array([[1, 0], [0, 1], [1, 1]])
    bias = np.array([10, 20])
    assert linear(x, weight, bias).tolist() == [[14, 25], [9, 20]]


def test_batch_time_channel():
    x = np.zeros((4, 8, 32))
    result = linear(x, np.zeros((32, 64)), np.zeros(64))
    assert result.shape == (4, 8, 64)


def test_rejects_transposed_weight():
    x = np.zeros((4, 8, 32))
    with pytest.raises(ValueError) as error:
        linear(x, np.zeros((32, 64)).T, np.zeros(64))
    assert "(64, 32)" in str(error.value)


def test_rejects_position_bias():
    x = np.zeros((4, 8, 32))
    with pytest.raises(ValueError) as error:
        linear(x, np.zeros((32, 64)), np.zeros((8, 1)))
    assert "(8, 1)" in str(error.value)

The transposed weight gives x (4, 8, 32) does not fit weight (64, 32): the 32 channels of x meet the 64 rows of the transposed weight.

← F01 — Python & NumPy · F03 — Probability & loss → · Glossary

Was this page helpful?