---
title: "F03 — Probability, logits, and loss"
sidebar:
  label: "F03 — Probability & loss"
---

<span id="f03-probability-logits-and-loss" />


[← F02 — Tensors & shapes](/foundations-02-shapes) · [F04 — Neuron & gradient descent →](/foundations-04-neuron) · [Glossary](/glossary)

## Learning outcome [#learning-outcome]

You will convert logits to probabilities stably, calculate one cross-entropy
by hand, and show how confidence changes the loss.

If you already write Python and remember `exp` and `ln`, skim the six steps in
about 20–30 minutes; the Build takes 30–45 minutes. If programming is new, plan about
60–75 minutes for the steps and another 60–90 minutes for the Build, over two
sessions. These times are guides, not deadlines.

## Step by step [#step-by-step]

One idea per step. Save each block as the file named above it, predict what it
prints, then run it from the course folder with `python artifacts/my-work/<file>`.
The comment on a line is the answer, so cover it first.

A **probability** lies between 0 and 1, and the probabilities of all
alternatives for the next token add up to 1. A **logit** is a score of any size,
bigger meaning more likely; it is not a probability. This page turns one into
the other and then measures how wrong a prediction was.

### exp and ln [#exp-and-ln]

`exp(x)` is `e` (about 2.71828) raised to the power `x`. `ln(x)`, the natural
logarithm, undoes it: `ln(exp(x)) = x`. On a calculator the `log` key usually
means base 10; use `ln`. In Python it is `math.log`. `artifacts/my-work/exp_ln.py`:

```python
import math

for x in [-2, 0, 1, 2]:
    print(x, round(math.exp(x), 4))

print(math.log(math.e))    # 1.0: ln undoes exp
print(math.log(1))         # 0.0
print(math.log10(100))     # 2.0: base-10 log, not the one this page uses
```

`round(x, 4)` keeps 4 decimals. Predict: is `exp(-2)` negative? The loop
prints

```text
-2 0.1353
0 1.0
1 2.7183
2 7.3891
```

`exp` of any number is positive: negative inputs land between 0 and 1. That is
why softmax can use it to turn any score into something that can be a share.

### Softmax of one row [#softmax-of-one-row]

**Softmax** turns a row of logits into probabilities: subtract the row's
largest value, take `exp` of each, divide each by their sum. By hand for logits
`[2, 0]`:

```text
subtract the max 2:   [0, -2]
exp of each:          [1, 0.1353]
sum:                  1.1353
divide by the sum:    [1/1.1353, 0.1353/1.1353] = [0.8808, 0.1192]
```

The same in NumPy, `artifacts/my-work/softmax_row.py`:

```python
import numpy as np

logits = np.array([2.0, 0.0])
shifted = logits - logits.max()
e = np.exp(shifted)
probs = e / e.sum()
print(shifted)            # [ 0. -2.]
print(e.round(4))         # [1.     0.1353]
print(e.sum().round(4))   # 1.1353
print(probs.round(6))     # [0.880797 0.119203]
print(probs.sum())        # 0.9999999999999999
```

`np.exp` works on every element at once; `.max()` and `.sum()` give the largest
value and the total; `.round(4)` only shortens what is printed. The last line
is not exactly 1: computers store most decimals slightly inaccurately. That is
why the checks below compare within a tolerance, `abs(a - b) < 1e-6`, and never
with `==`; `1e-6` is 0.000001.

### Shift all scores, or only one [#shift-all-scores-or-only-one]

Softmax only sees the *differences* between scores. `artifacts/my-work/shift.py`:

```python
import numpy as np


def softmax_row(logits):
    e = np.exp(logits - logits.max())
    return e / e.sum()


logits = np.array([2.0, 0.0])
print(softmax_row(logits).round(6))                          # [0.880797 0.119203]
print(softmax_row(logits + 10000).round(6))                  # the same
print(softmax_row(logits + np.array([0.0, 4.0])).round(6))   # [0.119203 0.880797]
```

Adding 10,000 to both scores keeps their difference at 2, so nothing changes.
Adding 4 to the second score only turns the difference from +2 into −2, and
the probabilities swap.

That second case is the edit you made in [Unit 0](/unit-00): +4.0 on the score
for "m" after a space moved its probability 0.089 → 0.843, and the other 32
characters lost that share. Adding 4.0 to all 33 scores in the row would have
changed nothing.

<details>
<summary>Where 0.843 comes from</summary>

Adding 4.0 to one logit multiplies that character's `exp` by
`exp(4) ≈ 54.6` and leaves the other 32 alone. Before the edit "m" had
0.0894 (shown rounded as 0.089) and the others shared 0.9106. After
dividing by the new total:
`0.0894 × 54.6 / (0.9106 + 0.0894 × 54.6) ≈ 0.843`.


</details>

### Why subtract the maximum [#why-subtract-the-maximum]

Leave the subtraction out and use big scores. `artifacts/my-work/overflow.py`:

```python
import numpy as np

big = np.array([1000.0, 1001.0])
e = np.exp(big)
print(e)
print(e / e.sum())

e = np.exp(big - big.max())
print((e / e.sum()).round(4))
```

Output (paths shortened):

```text
…/overflow.py:4: RuntimeWarning: overflow encountered in exp
  e = np.exp(big)
[inf inf]
…/overflow.py:6: RuntimeWarning: invalid value encountered in divide
  print(e / e.sum())
[nan nan]
[0.2689 0.7311]
```

A NumPy float holds numbers up to about 1.8 × 10³⁰⁸; `exp(1000)` is far larger,
so it becomes `inf` (infinity), and `inf / inf` is `nan` ("not a number").
NumPy only warns and keeps going, so a program can carry `nan` on silently.
Subtracting the maximum turns the scores into `[-1, 0]`: the largest `exp` is
then `exp(0) = 1`, and, as the previous step showed, the probabilities are
unchanged.

### Cross-entropy is a visible penalty [#cross-entropy-is-a-visible-penalty]

A **class index** is the position of the correct alternative in the row,
starting at 0. The **cross-entropy** of one prediction is `−ln(p)`, where `p` is
the probability given to the correct class — the "surprise" from Unit 0.
`artifacts/my-work/penalty.py`:

```python
import math

for p in [0.9, 0.5, 0.25, 0.1, 0.01]:
    print(p, round(-math.log(p), 3))
```

Predict which line prints the largest number, then run:

```text
0.9 0.105
0.5 0.693
0.25 1.386
0.1 2.303
0.01 4.605
```

A certain, correct prediction (`p = 1`) costs `−ln(1) = 0`. Halving `p` always
adds the same 0.693 (0.5 → 0.693, 0.25 → 1.386). For the logits `[2, 0]` from
the softmax step: if the target is class 0, the loss is
`−ln(0.880797) = 0.126928`; if it is class 1, `−ln(0.119203) = 2.126928`. The
same scores cost little when right and a lot when confidently wrong.

### A batch of two rows [#a-batch-of-two-rows]

A **batch** holds several rows at once: logits of shape `[batch, classes]` and
one target class index per row. `artifacts/my-work/batch_loss.py`:

```python
import numpy as np

logits = np.array([[2.0, 0.0],
                   [0.0, 0.0]])
targets = np.array([0, 1])

row_max = logits.max(axis=-1, keepdims=True)
print(row_max)
e = np.exp(logits - row_max)
probs = e / e.sum(axis=-1, keepdims=True)
print(probs.round(6))

chosen = probs[np.arange(2), targets]
print(np.arange(2), chosen.round(6))
losses = -np.log(chosen)
print(losses.round(6))
print(losses.mean().round(6))
```

Output:

```text
[[2.]
 [0.]]
[[0.880797 0.119203]
 [0.5      0.5     ]]
[0 1] [0.880797 0.5     ]
[0.126928 0.693147]
0.410038
```

- `axis=-1` works along the last axis, across each row's classes, so you get
  one max and one sum per row instead of one for the whole array.
- `keepdims=True` keeps the result as a column of shape `(2, 1)`, so F02's
  broadcasting subtracts each row's own max from that row.
- `np.arange(2)` is `[0, 1]`, the row numbers. `probs[np.arange(2), targets]`
  pairs them with the target columns: `probs[0, 0]` and `probs[1, 1]`.
- `np.log` is `ln` for every element; `.mean()` is the average. The batch loss
  is the average penalty: `(0.126928 + 0.693147) / 2 = 0.410038`.

<details>
<summary>What goes wrong without keepdims</summary>

`logits.max(axis=-1)` has shape `(2,)`: `[2., 0.]`. Broadcasting lines a
shape-`(2,)` array up with the *columns*, so both rows get `[2, 0]`
subtracted and the probabilities come out as `[[0.5, 0.5], [0.119203,
0.880797]]` — wrong, and with no error.


</details>

<details>
<summary>Check against the course&#x27;s own functions</summary>

Save as `artifacts/my-work/check_reference.py` and run it:

```python
from llm_course import cross_entropy_from_logits, stable_softmax

print(stable_softmax([[2.0, 0.0], [0.0, 0.0]]).round(6))
print(round(cross_entropy_from_logits([[2.0, 0.0], [0.0, 0.0]], [0, 1]), 6))
```

It prints the same probabilities and `0.410038`.


</details>

## Build and break [#build-and-break]

Work in one file, `artifacts/my-work/test_f03.py`, laid out like F01's
`test_f01.py`: imports (`numpy`, `pytest`), then two functions, then the tests.
Do not import `llm_course` here.

- `softmax(logits)`: the batch step's lines for a `[batch, classes]` array.
- `cross_entropy(logits, targets)`: raise `ValueError` when logits are not 2D,
  when `targets` does not have shape `(batch,)`, or when a target is below 0 or
  not below the number of classes, naming the failing value. Otherwise return
  the mean `−ln` of each row's target probability as a `float`.

One test function each, comparing within `1e-6`. Pass arrays, such as
`np.array([[2.0, 0.0]])`: a plain list has no `.max(axis=...)` and stops with an
`AttributeError`.

1. `softmax` of `[[2.0, 0.0]]` gives 0.880797 and 0.119203;
2. every row sums to one, including a row `[1000.0, 1001.0]`;
3. adding 10000 to every logit changes no probability;
4. `cross_entropy` of the two-row batch gives 0.410038;
5. a class index 2 in a two-class row is rejected;
6. targets of the wrong length are rejected.

Run `python -m pytest artifacts/my-work/test_f03.py`; the last line shows
`6 passed`. Stuck? Open the hint at the bottom.

Then break it: delete the max subtraction from `softmax`, predict which tests
fail, and run again. Expected: `2 failed, 4 passed, 4 warnings`, with the
overflow warning
you saw earlier listed above the summary. Restore the subtraction.

Compare with the reference solution at the bottom only after your tests pass.
The course's version is `stable_softmax` and `cross_entropy_from_logits` in
`src/llm_course/foundations.py`; it also rejects non-finite logits and
non-integer targets.

<details>
<summary>Stretch: when even the stable loss is inf</summary>

`cross_entropy_from_logits([[1000.0, -1000.0]], [1])` returns `inf` with
`RuntimeWarning: divide by zero encountered in log`: the target's
probability `exp(-2000)` is too small to store, so it becomes 0, and
`ln(0)` is infinite. The fix, **log-softmax**, never forms that
probability. Because `ln(a / b) = ln(a) − ln(b)`, the log-probabilities are
the shifted scores minus `ln` of their `exp`-sum (the "log-sum-exp"):

```python
import numpy as np

logits = np.array([[1000.0, -1000.0]])
shifted = logits - logits.max(axis=-1, keepdims=True)
log_probs = shifted - np.log(np.exp(shifted).sum(axis=-1, keepdims=True))
print(log_probs)         # [[    0. -2000.]]
print(-log_probs[0, 1])  # 2000.0
```

PyTorch's `cross_entropy`, which you meet in F06, works this way.


</details>

## Completion evidence [#completion-evidence]

- `python -m pytest artifacts/my-work/test_f03.py` shows `6 passed` and
  `ruff check artifacts/my-work` passes;
- your hand results (one row, the penalty, the batch) and your code agree
  within `1e-6`;
- you can say why adding one constant to every logit changes nothing, and why
  adding it to one logit moves probability;
- you have recorded what the break printed.

## What's next [#whats-next]

Continue with [F04 — Neuron and gradient descent](/foundations-04-neuron).

<details>
<summary>Build hint</summary>

`softmax` is the batch step's three lines. In `cross_entropy`, check
`logits.ndim != 2`, then `targets.shape != (logits.shape[0],)` — the comma
makes a one-item tuple — then `np.any(targets < 0)` and
`np.any(targets >= logits.shape[1])`. The rest is the batch step too:
`probs[np.arange(logits.shape[0]), targets]`, `-np.log(...)`, `.mean()`,
and `float(...)` around it. To compare whole arrays, use
`np.all(np.abs(a - b) < 1e-6)`; `np.abs` is `abs` for every element.


</details>

<details>
<summary>Reference solution</summary>

```python
import numpy as np
import pytest


def softmax(logits):
    """Turn each row of scores into probabilities that sum to 1."""
    shifted = logits - logits.max(axis=-1, keepdims=True)
    e = np.exp(shifted)
    return e / e.sum(axis=-1, keepdims=True)


def cross_entropy(logits, targets):
    """Mean of -ln(probability of the target) over the batch."""
    if logits.ndim != 2:
        raise ValueError(f"logits must be [batch, classes], got shape {logits.shape}")
    if targets.shape != (logits.shape[0],):
        raise ValueError(f"targets must have shape ({logits.shape[0]},), got {targets.shape}")
    if np.any(targets < 0) or np.any(targets >= logits.shape[1]):
        raise ValueError(f"target index outside 0..{logits.shape[1] - 1}: {targets.tolist()}")
    probs = softmax(logits)
    chosen = probs[np.arange(logits.shape[0]), targets]
    return float(-np.log(chosen).mean())


def test_softmax_matches_hand_calculation():
    probs = softmax(np.array([[2.0, 0.0]]))
    assert abs(probs[0, 0] - 0.880797) < 1e-6
    assert abs(probs[0, 1] - 0.119203) < 1e-6


def test_rows_sum_to_one():
    probs = softmax(np.array([[2.0, 0.0], [0.0, 0.0], [1000.0, 1001.0]]))
    assert np.all(np.abs(probs.sum(axis=-1) - 1) < 1e-6)


def test_common_constant_changes_nothing():
    logits = np.array([[2.0, 0.0]])
    assert np.all(np.abs(softmax(logits + 10000) - softmax(logits)) < 1e-6)


def test_cross_entropy_matches_hand_calculation():
    logits = np.array([[2.0, 0.0], [0.0, 0.0]])
    assert abs(cross_entropy(logits, np.array([0, 1])) - 0.410038) < 1e-6


def test_rejects_class_index_outside_the_row():
    with pytest.raises(ValueError) as error:
        cross_entropy(np.array([[2.0, 0.0]]), np.array([2]))
    assert "2" in str(error.value)


def test_rejects_targets_of_the_wrong_length():
    with pytest.raises(ValueError) as error:
        cross_entropy(np.array([[2.0, 0.0]]), np.array([0, 1]))
    assert "(2,)" in str(error.value)
```


</details>

[← F02 — Tensors & shapes](/foundations-02-shapes) · [F04 — Neuron & gradient descent →](/foundations-04-neuron) · [Glossary](/glossary)
