T02 — Byte-level BPE tokenizer from scratch
← T01 — Bigram baseline · T03 — Data pipeline & embeddings → · Glossary
Learning outcome
In 90–120 minutes, you will train, encode, and decode with your own byte-level Byte Pair Encoding tokenizer. You will preserve arbitrary Unicode, make special tokens explicit, visualize merge decisions and token lengths, and compare the small course vocabulary with a production tokenizer. No learner-owned data, cloud, or accelerator is required.
Predict · Characters are not bytes
Predict the UTF-8 byte counts of A, ü, 你, and 👩🏽💻. Then check:
for text in ("A", "ü", "你", "👩🏽💻"):
print(text, list(text.encode("utf-8")))A Python string contains Unicode text; storage and models need integers.
UTF-8 maps every valid string to bytes 0..255. BPE starts from those 256 byte
IDs and learns larger pieces. Therefore ordinary text never needs an unknown
token, even when its language or emoji never appeared in training.
Run the complete reference:
python examples/run_tokenizer_lab.pyIt writes only ignored local evidence under artifacts/tokenizer/:
report.json— corpus hash, vocabulary, merge trace, token pieces and lengths;merge-trace.svg— the first ordered merge decisions;token-lengths.svg— UTF-8 bytes versus course and production token counts.
Trace · One merge with concrete numbers
For abababab, UTF-8 starts as eight IDs:
97 98 97 98 97 98 97 98The pair (97,98) occurs four times. Merge it into new ID 256:
256 256 256 256Now adjacent pair (256,256) has frequency three; left-to-right replacement is
non-overlapping and produces two ID 257 tokens. Pair counting and replacement
are deliberately separate operations.
In the real run, train_byte_bpe repeats this loop:
- count every adjacent pair in the current training sequence;
- choose the highest frequency, breaking ties by the smaller ID pair;
- assign the next ID and replace non-overlapping occurrences;
- stop at the vocabulary budget or minimum frequency.
The deterministic tie-break, fixed corpus generator, and SHA-256 hash make the merge order reproducible.
Build · Encode and decode
The reference vocabulary has 300 IDs:
| Range | Purpose |
|---|---|
0..255 |
complete byte fallback |
256..295 |
40 learned merge tokens |
296..299 |
<|pad|>, <|bos|>, <|eos|>, <|unk|> |
Encoding first converts ordinary text to UTF-8 bytes. It repeatedly applies the currently available merge with the earliest training rank. Decoding expands merge IDs back to bytes and performs one strict UTF-8 decode. Test the invariant:
text = "Grüezi 你好 👩🏽💻 cafe\u0301"
ids = tokenizer.encode(text)
assert tokenizer.decode(ids) == textThe composed é and decomposed e plus combining accent have different byte
sequences. The tokenizer does not silently normalize either one; both roundtrip.
Build · Treat control tokens deliberately
Registered special tokens become atomic only when explicitly allowed:
ids = tokenizer.encode(
"<|bos|>hello<|eos|>",
allowed_special={"<|bos|>", "<|eos|>"},
)Without allowed_special, the same visible text is ordinary UTF-8 content.
This distinction prevents an untrusted document from silently becoming a model
control sequence. <|unk|> is reserved for interoperability; byte fallback
means ordinary Unicode input never emits it.
Break · Find the trade-off
Keep the corpus and minimum frequency fixed, then run:
python examples/run_tokenizer_lab.py --vocabulary-size 280
python examples/run_tokenizer_lab.py --vocabulary-size 340Predict the direction first. A larger vocabulary usually shortens familiar
training-like sequences, but enlarges the model’s embedding and output matrices.
For hidden size C, adding 60 tokens adds 60 × C embedding parameters and,
without tied weights, another 60 × C output parameters.
Now encode a script absent from training. Token count may approach its UTF-8 byte count, yet decoding still succeeds. This separates efficiency from OOV: unfamiliar text can be inefficient without being unrepresentable.
Measure · Compare with a production tokenizer
The lab imports tiktoken only after your implementation is trained and uses
cl100k_base as a production reference. The library is not used to implement
or validate the course BPE algorithm.
With vocabulary 300, the reference run produces:
| Evaluation text | UTF-8 bytes | Course BPE | cl100k_base |
|---|---|---|---|
| English sentence | 37 | 19 | 8 |
| German/Latin accents | 29 | 25 | 11 |
| CJK and emoji | 29 | 25 | 14 |
| composed/decomposed accents | 37 | 29 | 11 |
| Total | 132 | 98 | 44 |
The production tokenizer wins on length because it has a much larger vocabulary trained on vastly more varied data. The small course tokenizer remains useful: every merge is inspectable, its entire training procedure fits in one module, and all inputs still roundtrip.
See the versioned BPE baseline for the exact hash and acceptance contract.
Explain · Make the engineering decision
In your lab notes, answer:
- Why does byte fallback remove ordinary-text OOV failures?
- Why can an OOV-free tokenizer still produce long sequences?
- Which parameter matrices grow with the vocabulary?
- Why are merge rank and deterministic tie-breaking part of the model contract?
- Why must control tokens be distinguished from identical text in documents?
- Which vocabulary size would you choose for this corpus, and what evidence supports it?
Completion gate
You pass when all Roundtrip and edge-case tests succeed and you can discuss the three-way trade-off among vocabulary size, sequence length, and OOV behavior. Submit:
- your
report.jsonwith the expected training hash; - both SVG visualizations;
- one Unicode and one special-token Roundtrip;
- a 280-versus-300-or-340 controlled comparison;
- a short decision that mentions sequence cost and embedding/output size.
Run the contract checks:
pytest tests/test_tokenizer.pyWhat’s next
Next, the tokenizer enters a streaming data pipeline with manifests, contamination checks, deterministic shuffling, padding masks, and token batches for training.
← T01 — Bigram baseline · T03 — Data pipeline & embeddings → · Glossary