Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

F03 — Probability, logits, and loss

← F02 — Tensors & shapes · F04 — Neuron & gradient descent → · Glossary

Learning outcome

You will convert logits to probabilities stably, calculate one cross-entropy by hand, and show how confidence changes the loss.

If you already write Python and remember exp and ln, skim the six steps in about 20–30 minutes; the Build takes 30–45 minutes. If programming is new, plan about 60–75 minutes for the steps and another 60–90 minutes for the Build, over two sessions. These times are guides, not deadlines.

Step by step

One idea per step. Save each block as the file named above it, predict what it prints, then run it from the course folder with python artifacts/my-work/<file>. The comment on a line is the answer, so cover it first.

A probability lies between 0 and 1, and the probabilities of all alternatives for the next token add up to 1. A logit is a score of any size, bigger meaning more likely; it is not a probability. This page turns one into the other and then measures how wrong a prediction was.

exp and ln

exp(x) is e (about 2.71828) raised to the power x. ln(x), the natural logarithm, undoes it: ln(exp(x)) = x. On a calculator the log key usually means base 10; use ln. In Python it is math.log. artifacts/my-work/exp_ln.py:

import math

for x in [-2, 0, 1, 2]:
    print(x, round(math.exp(x), 4))

print(math.log(math.e))    # 1.0: ln undoes exp
print(math.log(1))         # 0.0
print(math.log10(100))     # 2.0: base-10 log, not the one this page uses

round(x, 4) keeps 4 decimals. Predict: is exp(-2) negative? The loop prints

-2 0.1353
0 1.0
1 2.7183
2 7.3891

exp of any number is positive: negative inputs land between 0 and 1. That is why softmax can use it to turn any score into something that can be a share.

Softmax of one row

Softmax turns a row of logits into probabilities: subtract the row’s largest value, take exp of each, divide each by their sum. By hand for logits [2, 0]:

subtract the max 2:   [0, -2]
exp of each:          [1, 0.1353]
sum:                  1.1353
divide by the sum:    [1/1.1353, 0.1353/1.1353] = [0.8808, 0.1192]

The same in NumPy, artifacts/my-work/softmax_row.py:

import numpy as np

logits = np.array([2.0, 0.0])
shifted = logits - logits.max()
e = np.exp(shifted)
probs = e / e.sum()
print(shifted)            # [ 0. -2.]
print(e.round(4))         # [1.     0.1353]
print(e.sum().round(4))   # 1.1353
print(probs.round(6))     # [0.880797 0.119203]
print(probs.sum())        # 0.9999999999999999

np.exp works on every element at once; .max() and .sum() give the largest value and the total; .round(4) only shortens what is printed. The last line is not exactly 1: computers store most decimals slightly inaccurately. That is why the checks below compare within a tolerance, abs(a - b) < 1e-6, and never with ==; 1e-6 is 0.000001.

Shift all scores, or only one

Softmax only sees the differences between scores. artifacts/my-work/shift.py:

import numpy as np


def softmax_row(logits):
    e = np.exp(logits - logits.max())
    return e / e.sum()


logits = np.array([2.0, 0.0])
print(softmax_row(logits).round(6))                          # [0.880797 0.119203]
print(softmax_row(logits + 10000).round(6))                  # the same
print(softmax_row(logits + np.array([0.0, 4.0])).round(6))   # [0.119203 0.880797]

Adding 10,000 to both scores keeps their difference at 2, so nothing changes. Adding 4 to the second score only turns the difference from +2 into −2, and the probabilities swap.

That second case is the edit you made in Unit 0: +4.0 on the score for “m” after a space moved its probability 0.089 → 0.843, and the other 32 characters lost that share. Adding 4.0 to all 33 scores in the row would have changed nothing.

Where 0.843 comes from

Adding 4.0 to one logit multiplies that character’s exp by exp(4) ≈ 54.6 and leaves the other 32 alone. Before the edit “m” had 0.0894 (shown rounded as 0.089) and the others shared 0.9106. After dividing by the new total: 0.0894 × 54.6 / (0.9106 + 0.0894 × 54.6) ≈ 0.843.

Why subtract the maximum

Leave the subtraction out and use big scores. artifacts/my-work/overflow.py:

import numpy as np

big = np.array([1000.0, 1001.0])
e = np.exp(big)
print(e)
print(e / e.sum())

e = np.exp(big - big.max())
print((e / e.sum()).round(4))

Output (paths shortened):

…/overflow.py:4: RuntimeWarning: overflow encountered in exp
  e = np.exp(big)
[inf inf]
…/overflow.py:6: RuntimeWarning: invalid value encountered in divide
  print(e / e.sum())
[nan nan]
[0.2689 0.7311]

A NumPy float holds numbers up to about 1.8 × 10³⁰⁸; exp(1000) is far larger, so it becomes inf (infinity), and inf / inf is nan (“not a number”). NumPy only warns and keeps going, so a program can carry nan on silently. Subtracting the maximum turns the scores into [-1, 0]: the largest exp is then exp(0) = 1, and, as the previous step showed, the probabilities are unchanged.

Cross-entropy is a visible penalty

A class index is the position of the correct alternative in the row, starting at 0. The cross-entropy of one prediction is −ln(p), where p is the probability given to the correct class — the “surprise” from Unit 0. artifacts/my-work/penalty.py:

import math

for p in [0.9, 0.5, 0.25, 0.1, 0.01]:
    print(p, round(-math.log(p), 3))

Predict which line prints the largest number, then run:

0.9 0.105
0.5 0.693
0.25 1.386
0.1 2.303
0.01 4.605

A certain, correct prediction (p = 1) costs −ln(1) = 0. Halving p always adds the same 0.693 (0.5 → 0.693, 0.25 → 1.386). For the logits [2, 0] from the softmax step: if the target is class 0, the loss is −ln(0.880797) = 0.126928; if it is class 1, −ln(0.119203) = 2.126928. The same scores cost little when right and a lot when confidently wrong.

A batch of two rows

A batch holds several rows at once: logits of shape [batch, classes] and one target class index per row. artifacts/my-work/batch_loss.py:

import numpy as np

logits = np.array([[2.0, 0.0],
                   [0.0, 0.0]])
targets = np.array([0, 1])

row_max = logits.max(axis=-1, keepdims=True)
print(row_max)
e = np.exp(logits - row_max)
probs = e / e.sum(axis=-1, keepdims=True)
print(probs.round(6))

chosen = probs[np.arange(2), targets]
print(np.arange(2), chosen.round(6))
losses = -np.log(chosen)
print(losses.round(6))
print(losses.mean().round(6))

Output:

[[2.]
 [0.]]
[[0.880797 0.119203]
 [0.5      0.5     ]]
[0 1] [0.880797 0.5     ]
[0.126928 0.693147]
0.410038
  • axis=-1 works along the last axis, across each row’s classes, so you get one max and one sum per row instead of one for the whole array.
  • keepdims=True keeps the result as a column of shape (2, 1), so F02’s broadcasting subtracts each row’s own max from that row.
  • np.arange(2) is [0, 1], the row numbers. probs[np.arange(2), targets] pairs them with the target columns: probs[0, 0] and probs[1, 1].
  • np.log is ln for every element; .mean() is the average. The batch loss is the average penalty: (0.126928 + 0.693147) / 2 = 0.410038.
What goes wrong without keepdims

logits.max(axis=-1) has shape (2,): [2., 0.]. Broadcasting lines a shape-(2,) array up with the columns, so both rows get [2, 0] subtracted and the probabilities come out as [[0.5, 0.5], [0.119203, 0.880797]] — wrong, and with no error.

Check against the course's own functions

Save as artifacts/my-work/check_reference.py and run it:

from llm_course import cross_entropy_from_logits, stable_softmax

print(stable_softmax([[2.0, 0.0], [0.0, 0.0]]).round(6))
print(round(cross_entropy_from_logits([[2.0, 0.0], [0.0, 0.0]], [0, 1]), 6))

It prints the same probabilities and 0.410038.

Build and break

Work in one file, artifacts/my-work/test_f03.py, laid out like F01’s test_f01.py: imports (numpy, pytest), then two functions, then the tests. Do not import llm_course here.

  • softmax(logits): the batch step’s lines for a [batch, classes] array.
  • cross_entropy(logits, targets): raise ValueError when logits are not 2D, when targets does not have shape (batch,), or when a target is below 0 or not below the number of classes, naming the failing value. Otherwise return the mean −ln of each row’s target probability as a float.

One test function each, comparing within 1e-6. Pass arrays, such as np.array([[2.0, 0.0]]): a plain list has no .max(axis=...) and stops with an AttributeError.

  1. softmax of [[2.0, 0.0]] gives 0.880797 and 0.119203;
  2. every row sums to one, including a row [1000.0, 1001.0];
  3. adding 10000 to every logit changes no probability;
  4. cross_entropy of the two-row batch gives 0.410038;
  5. a class index 2 in a two-class row is rejected;
  6. targets of the wrong length are rejected.

Run python -m pytest artifacts/my-work/test_f03.py; the last line shows 6 passed. Stuck? Open the hint at the bottom.

Then break it: delete the max subtraction from softmax, predict which tests fail, and run again. Expected: 2 failed, 4 passed, 4 warnings, with the overflow warning you saw earlier listed above the summary. Restore the subtraction.

Compare with the reference solution at the bottom only after your tests pass. The course’s version is stable_softmax and cross_entropy_from_logits in src/llm_course/foundations.py; it also rejects non-finite logits and non-integer targets.

Stretch: when even the stable loss is inf

cross_entropy_from_logits([[1000.0, -1000.0]], [1]) returns inf with RuntimeWarning: divide by zero encountered in log: the target’s probability exp(-2000) is too small to store, so it becomes 0, and ln(0) is infinite. The fix, log-softmax, never forms that probability. Because ln(a / b) = ln(a) − ln(b), the log-probabilities are the shifted scores minus ln of their exp-sum (the “log-sum-exp”):

import numpy as np

logits = np.array([[1000.0, -1000.0]])
shifted = logits - logits.max(axis=-1, keepdims=True)
log_probs = shifted - np.log(np.exp(shifted).sum(axis=-1, keepdims=True))
print(log_probs)         # [[    0. -2000.]]
print(-log_probs[0, 1])  # 2000.0

PyTorch’s cross_entropy, which you meet in F06, works this way.

Completion evidence

  • python -m pytest artifacts/my-work/test_f03.py shows 6 passed and ruff check artifacts/my-work passes;
  • your hand results (one row, the penalty, the batch) and your code agree within 1e-6;
  • you can say why adding one constant to every logit changes nothing, and why adding it to one logit moves probability;
  • you have recorded what the break printed.

What’s next

Continue with F04 — Neuron and gradient descent.

Build hint

softmax is the batch step’s three lines. In cross_entropy, check logits.ndim != 2, then targets.shape != (logits.shape[0],) — the comma makes a one-item tuple — then np.any(targets < 0) and np.any(targets >= logits.shape[1]). The rest is the batch step too: probs[np.arange(logits.shape[0]), targets], -np.log(...), .mean(), and float(...) around it. To compare whole arrays, use np.all(np.abs(a - b) < 1e-6); np.abs is abs for every element.

Reference solution
import numpy as np
import pytest


def softmax(logits):
    """Turn each row of scores into probabilities that sum to 1."""
    shifted = logits - logits.max(axis=-1, keepdims=True)
    e = np.exp(shifted)
    return e / e.sum(axis=-1, keepdims=True)


def cross_entropy(logits, targets):
    """Mean of -ln(probability of the target) over the batch."""
    if logits.ndim != 2:
        raise ValueError(f"logits must be [batch, classes], got shape {logits.shape}")
    if targets.shape != (logits.shape[0],):
        raise ValueError(f"targets must have shape ({logits.shape[0]},), got {targets.shape}")
    if np.any(targets < 0) or np.any(targets >= logits.shape[1]):
        raise ValueError(f"target index outside 0..{logits.shape[1] - 1}: {targets.tolist()}")
    probs = softmax(logits)
    chosen = probs[np.arange(logits.shape[0]), targets]
    return float(-np.log(chosen).mean())


def test_softmax_matches_hand_calculation():
    probs = softmax(np.array([[2.0, 0.0]]))
    assert abs(probs[0, 0] - 0.880797) < 1e-6
    assert abs(probs[0, 1] - 0.119203) < 1e-6


def test_rows_sum_to_one():
    probs = softmax(np.array([[2.0, 0.0], [0.0, 0.0], [1000.0, 1001.0]]))
    assert np.all(np.abs(probs.sum(axis=-1) - 1) < 1e-6)


def test_common_constant_changes_nothing():
    logits = np.array([[2.0, 0.0]])
    assert np.all(np.abs(softmax(logits + 10000) - softmax(logits)) < 1e-6)


def test_cross_entropy_matches_hand_calculation():
    logits = np.array([[2.0, 0.0], [0.0, 0.0]])
    assert abs(cross_entropy(logits, np.array([0, 1])) - 0.410038) < 1e-6


def test_rejects_class_index_outside_the_row():
    with pytest.raises(ValueError) as error:
        cross_entropy(np.array([[2.0, 0.0]]), np.array([2]))
    assert "2" in str(error.value)


def test_rejects_targets_of_the_wrong_length():
    with pytest.raises(ValueError) as error:
        cross_entropy(np.array([[2.0, 0.0]]), np.array([0, 1]))
    assert "(2,)" in str(error.value)

← F02 — Tensors & shapes · F04 — Neuron & gradient descent → · Glossary

Was this page helpful?