---
title: "F02 — Tensors and shapes"
sidebar:
  label: "F02 — Tensors & shapes"
---

<span id="f02-tensors-and-shapes" />


[← F01 — Python & NumPy](/foundations-01-python-numpy) · [F03 — Probability & loss →](/foundations-03-probability) · [Glossary](/glossary)

## Learning outcome [#learning-outcome]

You will predict the shape of an array before running the code, and write a small
linear layer that rejects a wrong weight or bias with a clear message. On the way
you meet two broadcasting mistakes: one that stops with an error and one that
runs silently.

If you already use NumPy, skim the steps and go to the Build: about 30–45 minutes
in all. If F01 was new to you, plan about 60–75 minutes for the steps and another
45–75 minutes for the Build, ideally in two sessions. These times are guides, not
deadlines.

## Shapes step by step [#shapes-step-by-step]

Each step asks you to predict before you run. The comment on each line is the
answer, so cover it first. Collect the blocks of all five steps in one file,
`artifacts/my-work/shapes.py`, each below the previous one, and run
`python artifacts/my-work/shapes.py` after adding each. When a step asks you to
try a line that fails, add it at the end, run, read the last line of the error,
then delete the line so the file runs again.

### Shape and broadcasting [#shape-and-broadcasting]

```python
import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6]])
row = np.array([10, 20, 30])
print(a.shape)              # (2, 3): 2 rows, 3 columns
print(row.shape)            # (3,): one axis with 3 entries
print((a + row).tolist())   # [[11, 22, 33], [14, 25, 36]]
```

One number is a **scalar**, a row of numbers a **vector**, a table a **matrix**;
**tensor** is the general word for an array with any number of axes. `shape`
lists the length of each axis: `(2, 3)` is two rows of three. The comma in `(3,)`
marks a tuple with one item, so `row` has one axis. `.tolist()` turns an array
into a plain Python list that prints on one line.

`a + row` adds `row` to *each* row of `a`. This is **broadcasting**: NumPy lines
the two shapes up from the right, `(2, 3)` over `(3,)`, and reuses the smaller
array along the axis it lacks. Compared axis by axis from the right, two sizes
must be equal, or one of them must be 1; a missing axis counts as 1.

Predict what `print(a + np.array([10, 20]))` does, then add it and run.

<details>
<summary>Check your prediction</summary>

It stops with
`ValueError: operands could not be broadcast together with shapes (2,3) (2,)`.
From the right, 3 meets 2: they differ and neither is 1. Delete the line.


</details>

### A concrete matrix multiplication [#a-concrete-matrix-multiplication]

```python
print((a * a).tolist())     # [[1, 4, 9], [16, 25, 36]]
```

`*` multiplies entry by entry. `@` is **matrix multiplication**, which combines
rows with columns:

```python
x = np.array([[1, 2, 3], [0, 1, -1]])
w = np.array([[1, 0], [0, 1], [1, 1]])
print((x @ w).shape)        # (2, 2)
print((x @ w).tolist())     # [[4, 5], [-1, 0]]
```

The rule: **entry (i, j) of `x @ w` is row i of `x` · column j of `w`**. The dot
`·` is a **dot product**: multiply matching entries, then add. Top left, row 0 of
`x` with column 0 of `w`: `1×1 + 2×0 + 3×1 = 4`. Top right, row 0 with column 1:
`1×0 + 2×1 + 3×1 = 5`. Work out the bottom row by hand before you run.

A row of `x` and a column of `w` must have the same length. `x` is `(2, 3)` and
`w` is `(3, 2)`: the inner sizes match (3 and 3) and are summed away; the outer
sizes remain, so `(2, 3) @ (3, 2)` gives `(2, 2)`. Predict whether
`print(a @ a)` works, then try it.

<details>
<summary>Check your prediction</summary>

It fails: `(2, 3) @ (2, 3)` puts 3 against 2 on the inside. NumPy's message
ends in `(size 2 is different from 3)`. Delete the line.


</details>

With square arrays, `*` and `@` both work and give the same shape, but different
numbers. A shape check cannot tell them apart:

```python
s = np.array([[1, 2], [3, 4]])
print((s * s).tolist())     # [[1, 4], [9, 16]]
print((s @ s).tolist())     # [[7, 10], [15, 22]]
```

### Reshape versus transpose [#reshape-versus-transpose]

```python
b = np.arange(6)
print(b.tolist())                   # [0, 1, 2, 3, 4, 5]
print(b.reshape(2, 3).tolist())     # [[0, 1, 2], [3, 4, 5]]
```

`np.arange(6)` counts from 0 to 5. `reshape(2, 3)` pours the same six values, in
the same reading order, into 2 rows of 3. Predict both lines below, then run:

```python
print(b.reshape(3, 2).tolist())     # [[0, 1], [2, 3], [4, 5]]
print(b.reshape(2, 3).T.tolist())   # [[0, 3], [1, 4], [2, 5]]
```

Both results have shape `(3, 2)`, with different contents. `reshape` keeps the
reading order; `.T` (**transpose**) swaps rows and columns, so row 0 `[0, 1, 2]`
becomes column 0. Mixing them up gives the right shape with wrong values, and no
shape check notices.

### Three axes: batch, time, channel [#three-axes-batch-time-channel]

A language model works on several texts at once, and each text is a sequence of
token positions. Its arrays therefore have three axes, written `[B, T, C]`:
**B**atch (which example), **T**ime (which token position) and **C**hannel (the
numbers that describe one token). Later lessons also write `D` for the channel
width. Here are 2 examples, 3 positions and 4 channels:

```python
t = np.arange(24).reshape(2, 3, 4)
print(t.shape)              # (2, 3, 4)
print(t[0].shape)           # (3, 4): example 0, all its positions
print(t[:, 0].shape)        # (2, 4): position 0 of every example
print(t[0, 0].tolist())     # [0, 1, 2, 3]: the 4 channels of one token
```

A number in the brackets picks one entry along that axis and removes the axis.
`:` means "all of this axis", as in F01's slices, so `t[:, 0]` keeps both
examples and picks position 0 in each. Predict the values, then add
`print(t[:, 0].tolist())` and run.

<details>
<summary>Check your prediction</summary>

`[[0, 1, 2, 3], [12, 13, 14, 15]]`. Example 0 starts at 0; example 1 starts
after 3 × 4 = 12 values.


</details>

### A linear layer: `x @ W + b` [#a-linear-layer-x-w-b]

A **linear layer** is the most common step in a neural network. It turns each
token's channels into new channels with a weight matrix `W`, then adds a
**bias** `b`: one extra number per output channel, learned like the weights. As a
formula: `linear(x) = x @ W + b`. With `x` and `w` from the matrix multiplication
and the bias `[10, 20]`:

```python
bias = np.array([10, 20])
print((x @ w + bias).tolist())   # [[14, 25], [9, 20]]
```

`x @ w` is `[[4, 5], [-1, 0]]`; broadcasting adds 10 to the first column and 20
to the second. The same line works on `[B, T, C]` arrays: `@` applies `W` at
every example and position, and only the last axis changes:

```python
activations = np.zeros((4, 8, 32))
weight = np.zeros((32, 64))
bias = np.zeros(64)
print((activations @ weight).shape)          # (4, 8, 64)
print((activations @ weight + bias).shape)   # (4, 8, 64)
```

`np.zeros(shape)` makes an array of zeros; only the shapes matter here. The bias
`(64,)` lines up with the last axis: one number per output channel, reused for
every example and position.

Now two wrong biases. Add
`print((activations @ weight + np.zeros(8)).shape)` and run: it stops with
`operands could not be broadcast together with shapes (4,8,64) (8,)`. That
mistake is loud. Replace `np.zeros(8)` with `np.zeros((8, 1))`, predict, and run
again.

<details>
<summary>Check your prediction</summary>

It prints `(4, 8, 64)` with no error. From the right, 1 meets 64 (the 1
stretches) and 8 meets 8, so NumPy adds one number per *position* to all 64
channels. That is not a bias, and nothing complains. Only an explicit check
such as `bias.shape == (weight.shape[1],)` catches it. Delete the line.


</details>

## Build and break [#build-and-break]

Work in one file, `artifacts/my-work/test_f02.py`: imports at the top, then your
function, then the tests. Write `linear(x, weight, bias)` that returns
`x @ weight + bias`, but first checks three things and raises `ValueError` with a
message that names the shapes:

- `weight` has two axes (`weight.ndim == 2`);
- the last axis of `x` has the same size as the first axis of `weight`;
- `bias` has shape `(weight.shape[1],)`: one number per output channel.

Add four tests: the linear-layer example gives `[[14, 25], [9, 20]]`; a `(4, 8, 32)`
input with a `(32, 64)` weight gives shape `(4, 8, 64)`; a transposed weight
`np.zeros((32, 64)).T` is rejected; the position bias `np.zeros((8, 1))` is
rejected. Run `python -m pytest artifacts/my-work/test_f02.py`.

Inside `pytest.raises` the test passes quietly, so you never see your message.
To read it, delete the `with pytest.raises(...)` line of the transposed-weight
test for a moment and unindent the call below it. pytest then fails and shows
your message on the line that starts with `E   ValueError:`. Say which two sizes
no longer agree, then put the line back.

A square weight, say `(64, 64)`, keeps its shape when transposed, so no shape
check catches that mistake; only a test with known values, like the linear-layer
example, does. Stuck? Open the hint at the bottom.

## Completion evidence [#completion-evidence]

- you wrote each shape as a comment before running the line that prints it;
- you can say what reshape, `.T` and broadcasting each do, with an example;
- the transposed weight fails inside `linear` with a message naming both shapes,
  not deep inside training;
- `ruff check artifacts/my-work` and `python -m pytest artifacts/my-work` pass.

## What's next [#whats-next]

Continue with [F03 — Probability and loss](/foundations-03-probability).

<details>
<summary>Hint</summary>

`x.shape[-1]` is the last axis of `x`, `weight.shape[0]` the first axis of
`weight`. Compare with `!=` ("not equal") and raise, for example
`raise ValueError(f"x {x.shape} does not fit weight {weight.shape}")`.
Each rejection test follows F01's four-part skeleton: bad input,
`pytest.raises`, the call, the message check (see
[Testing that bad input is rejected](/foundations-01-python-numpy#testing-that-bad-input-is-rejected)).


</details>

<details>
<summary>Reference answer</summary>

```python
import numpy as np
import pytest


def linear(x, weight, bias):
    """Return x @ weight + bias after checking that the shapes fit."""
    if weight.ndim != 2:
        raise ValueError(f"weight must be 2D, got shape {weight.shape}")
    if x.shape[-1] != weight.shape[0]:
        raise ValueError(f"x {x.shape} does not fit weight {weight.shape}")
    if bias.shape != (weight.shape[1],):
        raise ValueError(f"bias {bias.shape} must be ({weight.shape[1]},)")
    return x @ weight + bias


def test_worked_example():
    x = np.array([[1, 2, 3], [0, 1, -1]])
    weight = np.array([[1, 0], [0, 1], [1, 1]])
    bias = np.array([10, 20])
    assert linear(x, weight, bias).tolist() == [[14, 25], [9, 20]]


def test_batch_time_channel():
    x = np.zeros((4, 8, 32))
    result = linear(x, np.zeros((32, 64)), np.zeros(64))
    assert result.shape == (4, 8, 64)


def test_rejects_transposed_weight():
    x = np.zeros((4, 8, 32))
    with pytest.raises(ValueError) as error:
        linear(x, np.zeros((32, 64)).T, np.zeros(64))
    assert "(64, 32)" in str(error.value)


def test_rejects_position_bias():
    x = np.zeros((4, 8, 32))
    with pytest.raises(ValueError) as error:
        linear(x, np.zeros((32, 64)), np.zeros((8, 1)))
    assert "(8, 1)" in str(error.value)
```

The transposed weight gives `x (4, 8, 32) does not fit weight (64, 32)`: the
32 channels of `x` meet the 64 rows of the transposed weight.


</details>

[← F01 — Python & NumPy](/foundations-01-python-numpy) · [F03 — Probability & loss →](/foundations-03-probability) · [Glossary](/glossary)
