Unit 0 — First Tiny-LM Run
← Learning guide · Unit 0 troubleshooting → · Glossary
Every session starts in the course folder with .venv activated: your prompt
begins with (.venv) (how).
Learning outcome
You will run a prepared Tiny-LM, generate from your own prompt, inspect token IDs, probabilities, shapes, and loss, change one logit, and explain the measured effect. No prior ML knowledge, learner data, cloud, or accelerator is required.
It takes about 60 minutes if you already program. If you are new to programming, plan 90–120 minutes over two sittings. The times in the section headings show the order of the steps, not deadlines.
You need to be able to type commands in a terminal. Everything else — including every technical word on this page — is defined where it first appears. Each new term is also a forward reference: F03 and T01 build the same ideas properly later. Here you only need them well enough to predict, measure, and explain one change.
0–10 min · Predict and check setup
Before predicting anything, you need something to predict from. This is the whole model in three sentences:
- It has read a small amount of text and remembers, for every character, which characters tended to follow it.
- To write, it looks at the character it just produced and picks the character that followed it most often.
- It feeds that character back in as the new input, and repeats.
That is all. There is nothing else in this model.
Now write down two predictions. You will check both against measurements later in this unit.
- Starting from the prompt
the, will the text the model writes look like English, like random letters, or will it repeat itself? - If you force
mto be the most likely character after a space, will the written text change?
Then check your setup:
python examples/diagnose_unit0.pyYou should see five lines like these:
python pass 3.12.13
backend pass mps
artifacts pass artifacts/unit0-diagnostic
generation pass "the te t"
ready yesEach line names one check, its result, and what it found: your Python version
(3.11 or newer is needed); the backend, the processor that will do the
arithmetic — cpu, mps (an Apple GPU) or cuda (an NVIDIA GPU); the folder
where the prepared files were written; and a first smoke test — the prompt
the plus the four characters the model wrote after it. The last line says whether Unit 0 is ready.
If any line says fail, use the troubleshooting guide.
10–20 min · First generation
python examples/run_unit0.pyThe command prints a ten-line summary. This is the real output on the course
author’s machine. Your backend line may say cpu; with --device cpu every
other line is identical:
backend mps
prompt "the " → token IDs [26, 14, 11, 1]
top 5 next "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline "the te te te te te te te te "
change +4.0 on the score for "m" after " "
changed "the mode mode mode mode mode"
probability "m" after " ": 0.089 → 0.843
corpus loss 1.695 → 1.890 (worse; lower is better)
verdict the probability of "m" rose and the text changed
full report artifacts/unit0/report.jsonRead it line by line. Quotation marks show exactly where a piece of text starts
and ends, so " " is a single space.
backend— which processor ran it, as in the diagnostic.prompt— your prompt, then the same prompt with each character replaced by its number in the model’s alphabet. Those numbers are token IDs:tis 26 and the space is 1.top 5 next— the five characters the model considers most likely right after the prompt’s last character, each with its probability: a number from 0 to 1, where 0.263 means about 26 %. Equal probabilities are listed by token ID, so a tie can straddle the cut:salso has 0.089 and just misses the list.baseline— what the model wrote: your prompt plus 24 new characters.change— every run also makes one deliberate edit to a copy of the model. Here it makesmmore likely after a space; the next section explains what the “score” is.changed— what the edited copy wrote from the same prompt.probability— how likelymis after a space, before → after the edit.corpus loss— one number for how surprised the model is by the text it learned from, before → after the edit; lower is better.verdict— one sentence the script computes from the numbers above; no model writes it.full report— the file with everything the run measured, in more detail. You will open it once, later in this unit.
Compare the baseline line with your first prediction now.
Look at the text it read
The first run wrote two files into artifacts/unit0/: the practice text the
model learned from, tiny_corpus.txt, and the model’s saved numbers,
tiny_bigram.pt. The report sits beside them as report.json, and every run
replaces it.
Open artifacts/unit0/tiny_corpus.txt in a text editor: 16 short sentences.
Take a few lines and note which character follows each space (about 5 minutes).
Across the whole text, 84 spaces are followed by a letter: 24 by t, 16 by
p, 8 each by a, f, m and s, and 4 each by l, n and r. That is
where top 5 next comes from: 24 / 84 ≈ 0.29.
Why 0.263 and not 0.29
The model adds a quarter count to every possible pair, so that even a pair the text never contains is not impossible. It knows 33 characters, so the row after a space gains 33 × 0.25 = 8.25 counts: (24 + 0.25) / (84 + 8.25) = 0.263.
Why te te te? The model looks only at the last character and always takes
its most frequent follower:
| After | The text shows | The model writes |
|---|---|---|
| space | t 24 of 84 times |
t |
t |
e 12 and h 12 of 64 times — a tie |
e: on a tie the lower token ID wins (e is 11, h is 14) |
e |
a space 16 of 68 times | a space — and it starts over |
So this model can never write the: after t it always takes e.
Licence, fingerprints, and why the files are not in Git
The practice text is a 596-byte synthetic text the course generates
itself and dedicates to the public domain (CC0-1.0). The saved numbers are
a checkpoint: a file with a model’s numbers, here computed from the
counts of neighbouring characters. manifest.json records where both
files came from and a SHA-256 fingerprint of each. Both are generated on
your machine because datasets and checkpoints must not be committed to Git.
20–30 min · Trace the model
The model knows 33 different characters. That set is called the vocabulary,
and its size is written V, so V=33.
For the character it just saw, the model holds one score for every character
that could come next — 33 scores. Each score is called a logit: an
unnormalised score, where bigger means more likely, but which is not yet a
percentage. All of them together form a transition matrix with shape
[V,V]: one row per current character, one column per possible next character,
so 33 × 33 = 1089 numbers, and the current character selects exactly one row.
Turning one row of 33 logits into 33 probabilities that add up to 1 is called softmax. Choosing the single highest one and feeding that character back in is called greedy decoding — the loop you predicted from in the first ten minutes.
corpus loss is the cross-entropy. For each character in the corpus, take
the probability p the model gave to the character that actually came next
and turn it into a surprise, −ln(p) (the negative natural logarithm). The
more probability the right character got, the smaller the surprise:
| Probability of the right character | Surprise −ln(p) |
|---|---|
| 1 (certain) | 0 |
0.263 (t after a space) |
1.34 |
| 0.01 | 4.6 |
The corpus loss is the average surprise over the 595 characters that follow another character: 1.695. So lower is better. The glossary lists it under “Loss, NLL, cross-entropy”.
If you already program, check three facts about the report of a four-character
prompt. Save this as artifacts/my-work/check_report.py (create the folder if
it is missing):
from llm_course import run_vertical_slice
report = run_vertical_slice("the ", preferred_device="cpu")
assert len(report["prompt_token_ids"]) == 4
assert len(report["baseline"]["top_next_tokens"]) == 5
assert 0.0 <= report["baseline"]["target_probability"] <= 1.0
print("3 checks passed")Run it from the course folder with python artifacts/my-work/check_report.py.
It prints 3 checks passed; a failing check stops with an AssertionError.
If you do not program yet, do not skip this step — read the three assert
lines as three claims and check them by hand in the summary you already have:
the prompt line shows four token IDs for the four-character prompt, the
top 5 next line lists five candidates, and the first number on the
probability line — the target’s probability before the edit — is between 0
and 1. You will write this code yourself in F01.
30–42 min · Build one controlled change
The reference run adds 4.0 to one transition logit — the “score” on the
change line: after the prompt’s final space, make m more likely. Because the
change is made to a logit and not to a probability, +4.0 does not mean “four
more percent” — softmax decides how much of the probability actually moves, and
it takes it from the other candidates.
In the summary of your first run, read:
- the
probabilityline — the target probability before and after; - the
baselineandchangedlines — whether the generated continuation changed; - the
corpus lossline — whether the corpus loss improved or worsened.
This is where you check your second prediction.
Why mode? The changed text follows the counts too: after m the text most
often has o (8 of 20), after o both d and k appear 8 of 32 times and d
wins the tie by its lower token ID, and after d comes e — so the edited
model writes mode mode.
The full report
The summary is computed from a larger report. Look at it once:
python examples/run_unit0.py --json--json prints the whole report instead of the summary; it is the same content
as artifacts/unit0/report.json. Here it is with the two five-candidate lists
shortened to their first entry and the two fingerprints cut after eight
characters:
{
"backend": "mps",
"model": "character-bigram",
"vocabulary_size": 33,
"assets": {
"artifact_dir": "artifacts/unit0",
"corpus_path": "artifacts/unit0/tiny_corpus.txt",
"checkpoint_path": "artifacts/unit0/tiny_bigram.pt",
"manifest_path": "artifacts/unit0/manifest.json",
"dataset_sha256": "6d10e715[…]",
"checkpoint_sha256": "855912a6[…]"
},
"prompt": "the ",
"prompt_token_ids": [
26,
14,
11,
1
],
"baseline": {
"generation": "the te te te te te te te te ",
"corpus_loss": 1.6954470872879028,
"top_next_tokens": [
{
"token": "t",
"token_id": 26,
"probability": 0.2628726065158844
},
[…]
],
"target_probability": 0.08943088352680206
},
"controlled_change": {
"source_character": " ",
"target_character": "m",
"logit_boost": 4.0,
"generation": "the mode mode mode mode mode",
"corpus_loss": 1.8896712064743042,
"top_next_tokens": [
{
"token": "m",
"token_id": 19,
"probability": 0.8428245186805725
},
[…]
],
"target_probability": 0.8428245186805725
},
"measured_effect": {
"target_probability_delta": 0.7533936351537704,
"generation_changed": true
}
}Read a dotted name as a path into it: baseline.corpus_loss means the
corpus_loss field inside baseline. The blocks, top to bottom:
backend,promptandprompt_token_ids— the first two summary lines.model— the kind of model: a character bigram, one character of context.vocabulary_size—V, the 33 characters from the previous section.assets— provenance: where the corpus, the checkpoint and the manifest are, and a SHA-256 fingerprint of the corpus and of the checkpoint (a 64-character code that changes if a single byte of the file changes). It proves which data and which model produced these numbers. You can skip it on day one.baseline— the unedited model: itsgeneration, itscorpus_loss, itstop_next_tokens(each with its token ID), andtarget_probability, the probability of the target character after the prompt.controlled_change— the same fields for the edited copy, plus which cell was edited:source_character(the row),target_character(the column) andlogit_boost(how much was added).measured_effect— the answer to this section’s question, already computed:target_probability_deltais after minus before (0.843 − 0.089 ≈ 0.753), andgeneration_changedsays whether the two texts differ. Theverdictline is written from these two fields.
Keep the summary for everyday runs; open the report when you need a number the summary rounds or leaves out.
Your own change
First a fixed example with a different prompt and target character:
python examples/run_unit0.py --prompt "a " --target-character t --boost 3Predict the direction before running. The change affects exactly one cell in
the [V,V] matrix, so it can help one transition while harming the overall
corpus loss.
Check your prediction
backend mps
prompt "a " → token IDs [7, 1]
top 5 next "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline "a te te te te te te te te "
change +3.0 on the score for "t" after " "
changed "a te te te te te te te te "
probability "t" after " ": 0.263 → 0.877
corpus loss 1.695 → 1.828 (worse; lower is better)
verdict the probability of "t" rose, but the text did not change
full report artifacts/unit0/report.jsonThe probability of t more than tripled, yet baseline and changed are the
same text, and the verdict says so. Look at top 5 next: t was already the
most likely character after a space. Greedy decoding only asks which character
is highest, and t was highest before and after, so the edit made the model
more certain without changing what it writes. A probability and the pick made
from it are different things: the pick — the position of the highest value —
is called the argmax. Later units build on this, starting with
I01, where sampling draws from the probabilities
instead of always taking the argmax.
Then pick your own prompt and target. This is the model’s whole alphabet, in token-ID order from 0:
"\n !,-.?abcdefghijklmnopqrstuvwxyz"\n (a line break) is 0 and the space is 1, so a is 7 and t is 26: count
along to check. Capitals become lowercase; any other character is read as ?.
Your edit acts on the row of the prompt’s last character: with
--prompt "the" it changes the score for your target after e.
Interactive · Boost one logit
[V,V] · one cell
Boost one logit
The same counted bigram table as python examples/run_unit0.py, rebuilt in your browser. Before you drag: will the probability rise or fall, will the text change, and will the corpus loss get better or worse? Predict, then move the slider. Save the reversed boost (−4) until you have made the next section’s prediction.
What the script would print
- top 5 next
- "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
- change
- +4.0 on the score for "m" after " "
- probability
- "m" after " ": 0.089 → 0.843
- baseline
"the te te te te te te te te "- changed
"the mode mode mode mode mode"- corpus loss
- 1.695 → 1.890 (worse; lower is better)
- verdict
- the probability of "m" rose and the text changed
The text it learned from
"t" a character after " " (84 times) · "m" the target "m" (8 times)
- the tiny model predicts the next token.
- the small model learns from tiny text.
- the tiny model learns a simple pattern.
Show all 16 sentences — every character after " " is marked
- the tiny model predicts the next token.
- the small model learns from tiny text.
- a token follows a simple pattern.
- each pattern reveals a probability.
- the tiny model learns a simple pattern.
- the small model follows a probability.
- a token reveals the next token.
- each pattern predicts from tiny text.
- the tiny model follows the next token.
- the small model reveals from tiny text.
- a token predicts a simple pattern.
- each pattern learns a probability.
- the tiny model reveals a simple pattern.
- the small model predicts a probability.
- a token learns the next token.
- each pattern follows from tiny text.
Why this letter?
How often each character follows " " in that text (84 times in all), and its probability before and after your change:
| next | seen | before | after |
|---|---|---|---|
| "t" | 0.263 | 0.045 | |
| "p" | 0.176 | 0.030 | |
| "a" | 0.089 | 0.015 | |
| "f" | 0.089 | 0.015 | |
| "m"targetpicked | 0.089 | 0.843 | |
| "s" | 0.089 | 0.015 | |
| "l" | 0.046 | 0.008 | |
| "n" | 0.046 | 0.008 | |
| "r" | 0.046 | 0.008 |
before = (seen + 0.25) / (84 + 33 characters × 0.25). Your change is added to the target’s score; softmax then shares the probability out again, so the others shrink. The model picks the character with the highest “after”. Pairs never seen get a quarter count, so they are unlikely, not impossible.
How the text is written
Greedy decoding takes the most likely next character, then looks only at that one character — so the counts (and the one score you changed) decide every step.
- after " " → "t" · 24 of 84
- after "t" → "e" · 12 of 64tie with "h" (also 12) — lower token ID wins: "e" is 11, "h" is 14
- after "e" → " " · 16 of 68
from here it repeats (back to step 1)
- after " " → "m" · 8 of 84boosted: 0.089 → 0.843, now above "t"
- after "m" → "o" · 8 of 20
- after "o" → "d" · 8 of 32tie with "k" (also 8) — lower token ID wins: "d" is 10, "k" is 17
- after "d" → "e" · 8 of 12
- after "e" → " " · 16 of 68
from here it repeats (back to step 1)
42–50 min · Break an assumption
Reverse the intervention:
python examples/run_unit0.py --prompt "a " --target-character t --boost -4The target probability should fall. If your explanation still says the token became more likely, your interpretation—not the model—is broken. Predict whether the text changes as well, then run it.
Check your prediction
backend mps
prompt "a " → token IDs [7, 1]
top 5 next "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline "a te te te te te te te te "
change -4.0 on the score for "t" after " "
changed "a pa pa pa pa pa pa pa pa "
probability "t" after " ": 0.263 → 0.006
corpus loss 1.695 → 1.815 (worse; lower is better)
verdict the probability of "t" fell and the text changed
full report artifacts/unit0/report.jsonThis time the text changes too: with t pushed down, p, second in
top 5 next, becomes the highest. After p, a and r tie at 8 of 20
and a wins (token ID 7 < 24); after a comes a space — so the text
reads pa pa.
Every edit so far made the corpus loss worse, even the boost for t, which was
already the favourite. The scores were computed from the counts in this very
text, so they are already nearly the best fit for it, and almost any single
edit moves away from that. Boosting t takes probability from p, a, m and
the other characters that really do follow a space in the text, and each of
them now costs more surprise.
The exception
Pushing down a pair the text never contains improves the loss by a hair: --prompt x --target-character z --boost -2 gives 1.6954 → 1.6953.
Then try an empty prompt:
python examples/run_unit0.py --prompt ""
run_unit0.py: error: --prompt must contain at least one character (characters outside the vocabulary are read as "?")This is the expected error, not a crash: the script checks its input before it
runs the model, prints one line and stops without writing a report. A model
that predicts the next character needs at least one character to start from;
any character works, and one the model does not know is read as ?. Read the
boundary error instead of removing the validation.
In Windows PowerShell the empty quotes can be dropped; you then see
argument --prompt: expected one argument — the same kind of input check, also
with exit code 2.
50–55 min · Measure reproducibility
Greedy decoding always takes the highest score, so it gives the same answer on any backend. Sampling picks randomly among the candidates instead, so it needs a seed — a fixed starting point for the randomness — to repeat itself:
python examples/run_unit0.py --prompt "the " --sample --seed 11Before running, predict whether the sampled text will look like the greedy
te te te. Run it twice and compare the generations. Accelerator users may
also run --device cpu; the mandatory evidence is always available on CPU.
Check your prediction
backend mps
prompt "the " → token IDs [26, 14, 11, 1]
top 5 next "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline "the aroke fchu.\nty thea tts?"
change +4.0 on the score for "m" after " "
changed "the m mke fchu.\nty thea mob?"
probability "m" after " ": 0.089 → 0.843
corpus loss 1.695 → 1.890 (worse; lower is better)
verdict the probability of "m" rose and the text changed
full report artifacts/unit0/report.jsonSampled text looks nothing like the greedy te te te: it no longer always
takes the top candidate. \n inside the quotes is a line break the model
wrote. The second run prints the same text. Then try --seed 12: the
baseline becomes "the t mhol ty l tilj,cte phx", a different text, but
again the same on every run.
Why the random draws happen on the CPU
The model runs on the device named in the backend line. For each random
draw, only the 33 probabilities are copied to the CPU, so the same seed
follows the same path on every machine.
55–60 min · Explain
Complete these sentences in your lab notes:
- The prompt becomes token IDs by …
- The current character selects one row of scores, and that row has … numbers in it, because …
- Softmax is needed because …
- The controlled change moved probability from … to …
- The generation changed/did not change because …
- Corpus loss changed because the edited transition …
Compare after writing
One acceptable answer each, for the reference run (+4.0 on m):
- … looking up each character’s position in the model’s 33-character
alphabet, after lowercasing:
tis 26, the space is 1. - … 33 numbers, because there is one score for every character in the vocabulary that could come next.
- … logits are only scores; softmax turns them into probabilities from 0 to 1 that add up to 1.
- … the other characters that follow a space (
t,p,a, …) tom: 0.089 → 0.843. - … greedy decoding writes the highest-probability character, and after
a space that is now
minstead oft. (With+3ont,twas already highest, so the text stayed.) - … no longer matches the counts in the text: characters that really follow a space lost probability, so the average surprise rose.
Completion gate
You pass Unit 0 when the diagnostic is ready, your own prompt generates text,
you can point to the token IDs, the probabilities and the loss in your own
report, and your explanation matches the measured intervention. A wrong
prediction is fine: write down what surprised you.
Explaining it in your own words counts; reusing the words on this page does not.
Keep the full report of your own run, artifacts/unit0/report.json, as
evidence. Every run replaces it, so copy it once you have the run you want to
keep.
Course release acceptance additionally requires the documented five-learner pilot.
What’s next
Continue with the diagnostic and learning-path check. It determines which foundations you can safely skip and preserves a re-entry link for every shortcut.