T01 — Bigram language-model baseline
← F06 — PyTorch & checkpoint · T02 — Byte-level BPE → · Glossary
Learning outcome
In 75–90 minutes, you will build and compare a count-based and a neural character bigram model. You will trace the complete next-token training loop, measure negative log-likelihood on isolated data, sample reproducibly, and use fixed evidence to decide whether training improved the baseline. The supplied CC0 corpus is generated locally; you need no data of your own.
Predict · What can one character know?
A bigram model sees only the current character when predicting the next one. Before running the code, predict:
- which tensor shapes result from a batch
[B,T]; - whether a learned table can beat uniform probability
1/33; - why it still cannot remember the word before the current character.
Then run the CPU reference:
python examples/run_bigram_baseline.py --device cpuKeep the JSON output. It records the configuration, split hashes, losses, fixed prompts, probabilities, and seeded samples for both models.
Trace · Split before making windows
split_corpus shuffles 16 unique lines with seed 7 and creates 12 training,
2 validation, and 2 test lines. Only then does CharacterWindowDataset create
overlapping next-character examples. This order prevents nearly identical
windows from the same line leaking into multiple partitions.
For block_size=16 and batch_size=8, a batch has:
| Value | Shape | Meaning |
|---|---|---|
inputs |
[8,16] |
current character IDs |
targets |
[8,16] |
the same windows shifted by one |
logits |
[8,16,33] |
one score per possible next character |
The validation split chooses experiments. The test split is read only for the final report; repeatedly tuning on it would turn it into training information.
Build · Count probabilities
The count model records how often target character y follows source
character x. Additive smoothing gives every transition non-zero probability:
Here V=33 and alpha=0.25. Inspect fit_count_bigram: counts become
probabilities, then log-probabilities become the fixed transition logits.
There is no optimizer and no gradient descent in this model.
Use bigram_probabilities(model, "the ") and verify that all 33 values are
finite, non-negative, and sum to one.
Build · Learn the same table with gradients
TinyBigramLM contains one trainable [33,33] transition table. An input ID
selects a row. Cross-entropy compares that row with the observed next ID.
For targets , mean negative log-likelihood is
For a one-hot next-character target, this is exactly cross-entropy. Trace one
iteration in train_neural_bigram in this order:
- load a seeded batch and move it to the selected device;
- clear old gradients;
- compute logits and cross-entropy;
- run backward and reject non-finite gradients;
- update parameters with AdamW;
- measure validation loss without gradients.
Explain what would go wrong if step 2 happened only once before the loop.
Break · Expose weak evidence
Run an intentionally under-trained comparison:
python examples/run_bigram_baseline.py --device cpu --steps 1One training batch is not evidence that an experiment generalizes. A falling batch loss alone is also insufficient. Keep the split, seed, and prompts fixed, then restore 60 steps:
python examples/run_bigram_baseline.py --device cpu --steps 60Now change only the learning rate to 0.02. Predict the result, run it, and
record whether the validation loss—not merely training loss—improved.
Measure · Compare like with like
The reference configuration is seed 7, 60 steps, learning rate 0.2,
block_size=16, and batch_size=8. On the course environment, the count model
has validation loss about 1.861; the neural model moves from about 3.497 to
1.547 and therefore beats both its initial state and the count baseline.
Minor floating-point differences are acceptable; the direction and evidence
contract are not.
The prompts "the ", "a ", and "each " are fixed. Each report includes
the three most likely next characters and a CPU-seeded sample. Generated text
will remain locally plausible but incoherent because a bigram has one-character
context. That limitation is a result, not a sampling bug.
See the versioned reference baseline for hashes, thresholds, and the reproduction command.
Explain · Your model card paragraph
Write one paragraph that names:
- the input, target, parameter, and logits shapes;
- how counts and gradient descent estimate the same conditional distribution;
- why cross-entropy is negative log-likelihood here;
- which partition supports model selection and which supports final reporting;
- the measured validation comparison;
- the one-character context limitation.
Completion gate
You pass when you can recreate the full data → tokens → batches → logits → probabilities → loss → gradients → update → validation → sampling loop without looking at the diagram, and your saved report proves whether one controlled change improved the fixed baseline. Required evidence:
- disjoint split counts and SHA-256 hashes;
- asserted
[B,T]and[B,T,V]shapes; - finite normalized probabilities;
- initial, final, validation, and test losses;
- all three fixed prompts with the same seed;
- a conclusion based on validation loss, including a result that did not help.
Run the automated contract checks:
pytest tests/test_bigram.pyWhat’s next
The character vocabulary is fixed and cannot represent useful subwords. Continue with the byte-level BPE tokenizer and measure how vocabulary and sequence length change the same modelling problem.
← F06 — PyTorch & checkpoint · T02 — Byte-level BPE → · Glossary