Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

Unit 0 — First Tiny-LM Run

← Learning guide · Unit 0 troubleshooting → · Glossary

Every session starts in the course folder with .venv activated: your prompt begins with (.venv) (how).

Learning outcome

You will run a prepared Tiny-LM, generate from your own prompt, inspect token IDs, probabilities, shapes, and loss, change one logit, and explain the measured effect. No prior ML knowledge, learner data, cloud, or accelerator is required.

It takes about 60 minutes if you already program. If you are new to programming, plan 90–120 minutes over two sittings. The times in the section headings show the order of the steps, not deadlines.

You need to be able to type commands in a terminal. Everything else — including every technical word on this page — is defined where it first appears. Each new term is also a forward reference: F03 and T01 build the same ideas properly later. Here you only need them well enough to predict, measure, and explain one change.

0–10 min · Predict and check setup

Before predicting anything, you need something to predict from. This is the whole model in three sentences:

  1. It has read a small amount of text and remembers, for every character, which characters tended to follow it.
  2. To write, it looks at the character it just produced and picks the character that followed it most often.
  3. It feeds that character back in as the new input, and repeats.

That is all. There is nothing else in this model.

Now write down two predictions. You will check both against measurements later in this unit.

  1. Starting from the prompt the , will the text the model writes look like English, like random letters, or will it repeat itself?
  2. If you force m to be the most likely character after a space, will the written text change?

Then check your setup:

python examples/diagnose_unit0.py

You should see five lines like these:

python      pass  3.12.13
backend     pass  mps
artifacts   pass  artifacts/unit0-diagnostic
generation  pass  "the te t"
ready       yes

Each line names one check, its result, and what it found: your Python version (3.11 or newer is needed); the backend, the processor that will do the arithmetic — cpu, mps (an Apple GPU) or cuda (an NVIDIA GPU); the folder where the prepared files were written; and a first smoke test — the prompt the plus the four characters the model wrote after it. The last line says whether Unit 0 is ready. If any line says fail, use the troubleshooting guide.

10–20 min · First generation

python examples/run_unit0.py

The command prints a ten-line summary. This is the real output on the course author’s machine. Your backend line may say cpu; with --device cpu every other line is identical:

backend      mps
prompt       "the " → token IDs [26, 14, 11, 1]
top 5 next   "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline     "the te te te te te te te te "
change       +4.0 on the score for "m" after " "
changed      "the mode mode mode mode mode"
probability  "m" after " ": 0.089 → 0.843
corpus loss  1.695 → 1.890 (worse; lower is better)
verdict      the probability of "m" rose and the text changed
full report  artifacts/unit0/report.json

Read it line by line. Quotation marks show exactly where a piece of text starts and ends, so " " is a single space.

  • backend — which processor ran it, as in the diagnostic.
  • prompt — your prompt, then the same prompt with each character replaced by its number in the model’s alphabet. Those numbers are token IDs: t is 26 and the space is 1.
  • top 5 next — the five characters the model considers most likely right after the prompt’s last character, each with its probability: a number from 0 to 1, where 0.263 means about 26 %. Equal probabilities are listed by token ID, so a tie can straddle the cut: s also has 0.089 and just misses the list.
  • baseline — what the model wrote: your prompt plus 24 new characters.
  • change — every run also makes one deliberate edit to a copy of the model. Here it makes m more likely after a space; the next section explains what the “score” is.
  • changed — what the edited copy wrote from the same prompt.
  • probability — how likely m is after a space, before → after the edit.
  • corpus loss — one number for how surprised the model is by the text it learned from, before → after the edit; lower is better.
  • verdict — one sentence the script computes from the numbers above; no model writes it.
  • full report — the file with everything the run measured, in more detail. You will open it once, later in this unit.

Compare the baseline line with your first prediction now.

Look at the text it read

The first run wrote two files into artifacts/unit0/: the practice text the model learned from, tiny_corpus.txt, and the model’s saved numbers, tiny_bigram.pt. The report sits beside them as report.json, and every run replaces it.

Open artifacts/unit0/tiny_corpus.txt in a text editor: 16 short sentences. Take a few lines and note which character follows each space (about 5 minutes). Across the whole text, 84 spaces are followed by a letter: 24 by t, 16 by p, 8 each by a, f, m and s, and 4 each by l, n and r. That is where top 5 next comes from: 24 / 84 ≈ 0.29.

Why 0.263 and not 0.29

The model adds a quarter count to every possible pair, so that even a pair the text never contains is not impossible. It knows 33 characters, so the row after a space gains 33 × 0.25 = 8.25 counts: (24 + 0.25) / (84 + 8.25) = 0.263.

Why te te te? The model looks only at the last character and always takes its most frequent follower:

After The text shows The model writes
space t 24 of 84 times t
t e 12 and h 12 of 64 times — a tie e: on a tie the lower token ID wins (e is 11, h is 14)
e a space 16 of 68 times a space — and it starts over

So this model can never write the: after t it always takes e.

Licence, fingerprints, and why the files are not in Git

The practice text is a 596-byte synthetic text the course generates itself and dedicates to the public domain (CC0-1.0). The saved numbers are a checkpoint: a file with a model’s numbers, here computed from the counts of neighbouring characters. manifest.json records where both files came from and a SHA-256 fingerprint of each. Both are generated on your machine because datasets and checkpoints must not be committed to Git.

20–30 min · Trace the model

The model knows 33 different characters. That set is called the vocabulary, and its size is written V, so V=33.

For the character it just saw, the model holds one score for every character that could come next — 33 scores. Each score is called a logit: an unnormalised score, where bigger means more likely, but which is not yet a percentage. All of them together form a transition matrix with shape [V,V]: one row per current character, one column per possible next character, so 33 × 33 = 1089 numbers, and the current character selects exactly one row.

Turning one row of 33 logits into 33 probabilities that add up to 1 is called softmax. Choosing the single highest one and feeding that character back in is called greedy decoding — the loop you predicted from in the first ten minutes.

corpus loss is the cross-entropy. For each character in the corpus, take the probability p the model gave to the character that actually came next and turn it into a surprise, −ln(p) (the negative natural logarithm). The more probability the right character got, the smaller the surprise:

Probability of the right character Surprise −ln(p)
1 (certain) 0
0.263 (t after a space) 1.34
0.01 4.6

The corpus loss is the average surprise over the 595 characters that follow another character: 1.695. So lower is better. The glossary lists it under “Loss, NLL, cross-entropy”.

If you already program, check three facts about the report of a four-character prompt. Save this as artifacts/my-work/check_report.py (create the folder if it is missing):

from llm_course import run_vertical_slice

report = run_vertical_slice("the ", preferred_device="cpu")
assert len(report["prompt_token_ids"]) == 4
assert len(report["baseline"]["top_next_tokens"]) == 5
assert 0.0 <= report["baseline"]["target_probability"] <= 1.0
print("3 checks passed")

Run it from the course folder with python artifacts/my-work/check_report.py. It prints 3 checks passed; a failing check stops with an AssertionError.

If you do not program yet, do not skip this step — read the three assert lines as three claims and check them by hand in the summary you already have: the prompt line shows four token IDs for the four-character prompt, the top 5 next line lists five candidates, and the first number on the probability line — the target’s probability before the edit — is between 0 and 1. You will write this code yourself in F01.

30–42 min · Build one controlled change

The reference run adds 4.0 to one transition logit — the “score” on the change line: after the prompt’s final space, make m more likely. Because the change is made to a logit and not to a probability, +4.0 does not mean “four more percent” — softmax decides how much of the probability actually moves, and it takes it from the other candidates.

In the summary of your first run, read:

  • the probability line — the target probability before and after;
  • the baseline and changed lines — whether the generated continuation changed;
  • the corpus loss line — whether the corpus loss improved or worsened.

This is where you check your second prediction.

Why mode? The changed text follows the counts too: after m the text most often has o (8 of 20), after o both d and k appear 8 of 32 times and d wins the tie by its lower token ID, and after d comes e — so the edited model writes mode mode.

The full report

The summary is computed from a larger report. Look at it once:

python examples/run_unit0.py --json

--json prints the whole report instead of the summary; it is the same content as artifacts/unit0/report.json. Here it is with the two five-candidate lists shortened to their first entry and the two fingerprints cut after eight characters:

{
  "backend": "mps",
  "model": "character-bigram",
  "vocabulary_size": 33,
  "assets": {
    "artifact_dir": "artifacts/unit0",
    "corpus_path": "artifacts/unit0/tiny_corpus.txt",
    "checkpoint_path": "artifacts/unit0/tiny_bigram.pt",
    "manifest_path": "artifacts/unit0/manifest.json",
    "dataset_sha256": "6d10e715[…]",
    "checkpoint_sha256": "855912a6[…]"
  },
  "prompt": "the ",
  "prompt_token_ids": [
    26,
    14,
    11,
    1
  ],
  "baseline": {
    "generation": "the te te te te te te te te ",
    "corpus_loss": 1.6954470872879028,
    "top_next_tokens": [
      {
        "token": "t",
        "token_id": 26,
        "probability": 0.2628726065158844
      },
      […]
    ],
    "target_probability": 0.08943088352680206
  },
  "controlled_change": {
    "source_character": " ",
    "target_character": "m",
    "logit_boost": 4.0,
    "generation": "the mode mode mode mode mode",
    "corpus_loss": 1.8896712064743042,
    "top_next_tokens": [
      {
        "token": "m",
        "token_id": 19,
        "probability": 0.8428245186805725
      },
      […]
    ],
    "target_probability": 0.8428245186805725
  },
  "measured_effect": {
    "target_probability_delta": 0.7533936351537704,
    "generation_changed": true
  }
}

Read a dotted name as a path into it: baseline.corpus_loss means the corpus_loss field inside baseline. The blocks, top to bottom:

  • backend, prompt and prompt_token_ids — the first two summary lines.
  • model — the kind of model: a character bigram, one character of context. vocabulary_size — V, the 33 characters from the previous section.
  • assets — provenance: where the corpus, the checkpoint and the manifest are, and a SHA-256 fingerprint of the corpus and of the checkpoint (a 64-character code that changes if a single byte of the file changes). It proves which data and which model produced these numbers. You can skip it on day one.
  • baseline — the unedited model: its generation, its corpus_loss, its top_next_tokens (each with its token ID), and target_probability, the probability of the target character after the prompt.
  • controlled_change — the same fields for the edited copy, plus which cell was edited: source_character (the row), target_character (the column) and logit_boost (how much was added).
  • measured_effect — the answer to this section’s question, already computed: target_probability_delta is after minus before (0.843 − 0.089 ≈ 0.753), and generation_changed says whether the two texts differ. The verdict line is written from these two fields.

Keep the summary for everyday runs; open the report when you need a number the summary rounds or leaves out.

Your own change

First a fixed example with a different prompt and target character:

python examples/run_unit0.py --prompt "a " --target-character t --boost 3

Predict the direction before running. The change affects exactly one cell in the [V,V] matrix, so it can help one transition while harming the overall corpus loss.

Check your prediction
backend      mps
prompt       "a " → token IDs [7, 1]
top 5 next   "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline     "a te te te te te te te te "
change       +3.0 on the score for "t" after " "
changed      "a te te te te te te te te "
probability  "t" after " ": 0.263 → 0.877
corpus loss  1.695 → 1.828 (worse; lower is better)
verdict      the probability of "t" rose, but the text did not change
full report  artifacts/unit0/report.json

The probability of t more than tripled, yet baseline and changed are the same text, and the verdict says so. Look at top 5 next: t was already the most likely character after a space. Greedy decoding only asks which character is highest, and t was highest before and after, so the edit made the model more certain without changing what it writes. A probability and the pick made from it are different things: the pick — the position of the highest value — is called the argmax. Later units build on this, starting with I01, where sampling draws from the probabilities instead of always taking the argmax.

Then pick your own prompt and target. This is the model’s whole alphabet, in token-ID order from 0:

"\n !,-.?abcdefghijklmnopqrstuvwxyz"

\n (a line break) is 0 and the space is 1, so a is 7 and t is 26: count along to check. Capitals become lowercase; any other character is read as ?. Your edit acts on the row of the prompt’s last character: with --prompt "the" it changes the score for your target after e.

Interactive · Boost one logit

[V,V] · one cell

Boost one logit

The same counted bigram table as python examples/run_unit0.py, rebuilt in your browser. Before you drag: will the probability rise or fall, will the text change, and will the corpus loss get better or worse? Predict, then move the slider. Save the reversed boost (−4) until you have made the next section’s prediction.

Boost: +4.0

What the script would print

top 5 next
"t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
change
+4.0 on the score for "m" after " "
probability
"m" after " ": 0.089 → 0.843
baseline
"the te te te te te te te te "
changed
"the mode mode mode mode mode"
corpus loss
1.695 → 1.890 (worse; lower is better)
verdict
the probability of "m" rose and the text changed

The text it learned from

"t" a character after " " (84 times) · "m" the target "m" (8 times)

  1. the tiny model predicts the next token.
  2. the small model learns from tiny text.
  3. the tiny model learns a simple pattern.
Show all 16 sentences — every character after " " is marked
  1. the tiny model predicts the next token.
  2. the small model learns from tiny text.
  3. a token follows a simple pattern.
  4. each pattern reveals a probability.
  5. the tiny model learns a simple pattern.
  6. the small model follows a probability.
  7. a token reveals the next token.
  8. each pattern predicts from tiny text.
  9. the tiny model follows the next token.
  10. the small model reveals from tiny text.
  11. a token predicts a simple pattern.
  12. each pattern learns a probability.
  13. the tiny model reveals a simple pattern.
  14. the small model predicts a probability.
  15. a token learns the next token.
  16. each pattern follows from tiny text.

Why this letter?

How often each character follows " " in that text (84 times in all), and its probability before and after your change:

nextseenbeforeafter
"t"24 of 840.2630.045
"p"16 of 840.1760.030
"a"8 of 840.0890.015
"f"8 of 840.0890.015
"m"targetpicked8 of 840.0890.843
"s"8 of 840.0890.015
"l"4 of 840.0460.008
"n"4 of 840.0460.008
"r"4 of 840.0460.008

before = (seen + 0.25) / (84 + 33 characters × 0.25). Your change is added to the target’s score; softmax then shares the probability out again, so the others shrink. The model picks the character with the highest “after”. Pairs never seen get a quarter count, so they are unlikely, not impossible.

How the text is written

Greedy decoding takes the most likely next character, then looks only at that one character — so the counts (and the one score you changed) decide every step.

Before the change
  1. after " " → "t" · 24 of 84
  2. after "t" → "e" · 12 of 64tie with "h" (also 12) — lower token ID wins: "e" is 11, "h" is 14
  3. after "e" → " " · 16 of 68

from here it repeats (back to step 1)

After the change
  1. after " " → "m" · 8 of 84boosted: 0.089 → 0.843, now above "t"
  2. after "m" → "o" · 8 of 20
  3. after "o" → "d" · 8 of 32tie with "k" (also 8) — lower token ID wins: "d" is 10, "k" is 17
  4. after "d" → "e" · 8 of 12
  5. after "e" → " " · 16 of 68

from here it repeats (back to step 1)

42–50 min · Break an assumption

Reverse the intervention:

python examples/run_unit0.py --prompt "a " --target-character t --boost -4

The target probability should fall. If your explanation still says the token became more likely, your interpretation—not the model—is broken. Predict whether the text changes as well, then run it.

Check your prediction
backend      mps
prompt       "a " → token IDs [7, 1]
top 5 next   "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline     "a te te te te te te te te "
change       -4.0 on the score for "t" after " "
changed      "a pa pa pa pa pa pa pa pa "
probability  "t" after " ": 0.263 → 0.006
corpus loss  1.695 → 1.815 (worse; lower is better)
verdict      the probability of "t" fell and the text changed
full report  artifacts/unit0/report.json

This time the text changes too: with t pushed down, p, second in top 5 next, becomes the highest. After p, a and r tie at 8 of 20 and a wins (token ID 7 < 24); after a comes a space — so the text reads pa pa.

Every edit so far made the corpus loss worse, even the boost for t, which was already the favourite. The scores were computed from the counts in this very text, so they are already nearly the best fit for it, and almost any single edit moves away from that. Boosting t takes probability from p, a, m and the other characters that really do follow a space in the text, and each of them now costs more surprise.

The exception

Pushing down a pair the text never contains improves the loss by a hair: --prompt x --target-character z --boost -2 gives 1.6954 → 1.6953.

Then try an empty prompt:

python examples/run_unit0.py --prompt ""

run_unit0.py: error: --prompt must contain at least one character (characters outside the vocabulary are read as "?")

This is the expected error, not a crash: the script checks its input before it runs the model, prints one line and stops without writing a report. A model that predicts the next character needs at least one character to start from; any character works, and one the model does not know is read as ?. Read the boundary error instead of removing the validation.

In Windows PowerShell the empty quotes can be dropped; you then see argument --prompt: expected one argument — the same kind of input check, also with exit code 2.

50–55 min · Measure reproducibility

Greedy decoding always takes the highest score, so it gives the same answer on any backend. Sampling picks randomly among the candidates instead, so it needs a seed — a fixed starting point for the randomness — to repeat itself:

python examples/run_unit0.py --prompt "the " --sample --seed 11

Before running, predict whether the sampled text will look like the greedy te te te. Run it twice and compare the generations. Accelerator users may also run --device cpu; the mandatory evidence is always available on CPU.

Check your prediction
backend      mps
prompt       "the " → token IDs [26, 14, 11, 1]
top 5 next   "t" 0.263 · "p" 0.176 · "a" 0.089 · "f" 0.089 · "m" 0.089
baseline     "the aroke fchu.\nty thea tts?"
change       +4.0 on the score for "m" after " "
changed      "the m mke fchu.\nty thea mob?"
probability  "m" after " ": 0.089 → 0.843
corpus loss  1.695 → 1.890 (worse; lower is better)
verdict      the probability of "m" rose and the text changed
full report  artifacts/unit0/report.json

Sampled text looks nothing like the greedy te te te: it no longer always takes the top candidate. \n inside the quotes is a line break the model wrote. The second run prints the same text. Then try --seed 12: the baseline becomes "the t mhol ty l tilj,cte phx", a different text, but again the same on every run.

Why the random draws happen on the CPU

The model runs on the device named in the backend line. For each random draw, only the 33 probabilities are copied to the CPU, so the same seed follows the same path on every machine.

55–60 min · Explain

Complete these sentences in your lab notes:

  1. The prompt becomes token IDs by …
  2. The current character selects one row of scores, and that row has … numbers in it, because …
  3. Softmax is needed because …
  4. The controlled change moved probability from … to …
  5. The generation changed/did not change because …
  6. Corpus loss changed because the edited transition …
Compare after writing

One acceptable answer each, for the reference run (+4.0 on m):

  1. … looking up each character’s position in the model’s 33-character alphabet, after lowercasing: t is 26, the space is 1.
  2. … 33 numbers, because there is one score for every character in the vocabulary that could come next.
  3. … logits are only scores; softmax turns them into probabilities from 0 to 1 that add up to 1.
  4. … the other characters that follow a space (t, p, a, …) to m: 0.089 → 0.843.
  5. … greedy decoding writes the highest-probability character, and after a space that is now m instead of t. (With +3 on t, t was already highest, so the text stayed.)
  6. … no longer matches the counts in the text: characters that really follow a space lost probability, so the average surprise rose.

Completion gate

You pass Unit 0 when the diagnostic is ready, your own prompt generates text, you can point to the token IDs, the probabilities and the loss in your own report, and your explanation matches the measured intervention. A wrong prediction is fine: write down what surprised you. Explaining it in your own words counts; reusing the words on this page does not. Keep the full report of your own run, artifacts/unit0/report.json, as evidence. Every run replaces it, so copy it once you have the run you want to keep.

Course release acceptance additionally requires the documented five-learner pilot.

What’s next

Continue with the diagnostic and learning-path check. It determines which foundations you can safely skip and preserves a re-entry link for every shortcut.

← Learning guide · Unit 0 troubleshooting → · Glossary

Was this page helpful?