Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

T02 — Byte-level BPE tokenizer from scratch

← T01 — Bigram baseline · T03 — Data pipeline & embeddings → · Glossary

Learning outcome

In 90–120 minutes, you will train, encode, and decode with your own byte-level Byte Pair Encoding tokenizer. You will preserve arbitrary Unicode, make special tokens explicit, visualize merge decisions and token lengths, and compare the small course vocabulary with a production tokenizer. No learner-owned data, cloud, or accelerator is required.

Predict · Characters are not bytes

Predict the UTF-8 byte counts of A, ü, 你, and 👩🏽‍💻. Then check:

for text in ("A", "ü", "你", "👩🏽‍💻"):
    print(text, list(text.encode("utf-8")))

A Python string contains Unicode text; storage and models need integers. UTF-8 maps every valid string to bytes 0..255. BPE starts from those 256 byte IDs and learns larger pieces. Therefore ordinary text never needs an unknown token, even when its language or emoji never appeared in training.

Run the complete reference:

python examples/run_tokenizer_lab.py

It writes only ignored local evidence under artifacts/tokenizer/:

  • report.json — corpus hash, vocabulary, merge trace, token pieces and lengths;
  • merge-trace.svg — the first ordered merge decisions;
  • token-lengths.svg — UTF-8 bytes versus course and production token counts.

Trace · One merge with concrete numbers

For abababab, UTF-8 starts as eight IDs:

97 98 97 98 97 98 97 98

The pair (97,98) occurs four times. Merge it into new ID 256:

256 256 256 256

Now adjacent pair (256,256) has frequency three; left-to-right replacement is non-overlapping and produces two ID 257 tokens. Pair counting and replacement are deliberately separate operations.

In the real run, train_byte_bpe repeats this loop:

  1. count every adjacent pair in the current training sequence;
  2. choose the highest frequency, breaking ties by the smaller ID pair;
  3. assign the next ID and replace non-overlapping occurrences;
  4. stop at the vocabulary budget or minimum frequency.

The deterministic tie-break, fixed corpus generator, and SHA-256 hash make the merge order reproducible.

Build · Encode and decode

The reference vocabulary has 300 IDs:

Range Purpose
0..255 complete byte fallback
256..295 40 learned merge tokens
296..299 <|pad|>, <|bos|>, <|eos|>, <|unk|>

Encoding first converts ordinary text to UTF-8 bytes. It repeatedly applies the currently available merge with the earliest training rank. Decoding expands merge IDs back to bytes and performs one strict UTF-8 decode. Test the invariant:

text = "Grüezi 你好 👩🏽‍💻 cafe\u0301"
ids = tokenizer.encode(text)
assert tokenizer.decode(ids) == text

The composed é and decomposed e plus combining accent have different byte sequences. The tokenizer does not silently normalize either one; both roundtrip.

Build · Treat control tokens deliberately

Registered special tokens become atomic only when explicitly allowed:

ids = tokenizer.encode(
    "<|bos|>hello<|eos|>",
    allowed_special={"<|bos|>", "<|eos|>"},
)

Without allowed_special, the same visible text is ordinary UTF-8 content. This distinction prevents an untrusted document from silently becoming a model control sequence. <|unk|> is reserved for interoperability; byte fallback means ordinary Unicode input never emits it.

Break · Find the trade-off

Keep the corpus and minimum frequency fixed, then run:

python examples/run_tokenizer_lab.py --vocabulary-size 280
python examples/run_tokenizer_lab.py --vocabulary-size 340

Predict the direction first. A larger vocabulary usually shortens familiar training-like sequences, but enlarges the model’s embedding and output matrices. For hidden size C, adding 60 tokens adds 60 × C embedding parameters and, without tied weights, another 60 × C output parameters.

Now encode a script absent from training. Token count may approach its UTF-8 byte count, yet decoding still succeeds. This separates efficiency from OOV: unfamiliar text can be inefficient without being unrepresentable.

Measure · Compare with a production tokenizer

The lab imports tiktoken only after your implementation is trained and uses cl100k_base as a production reference. The library is not used to implement or validate the course BPE algorithm.

With vocabulary 300, the reference run produces:

Evaluation text UTF-8 bytes Course BPE cl100k_base
English sentence 37 19 8
German/Latin accents 29 25 11
CJK and emoji 29 25 14
composed/decomposed accents 37 29 11
Total 132 98 44

The production tokenizer wins on length because it has a much larger vocabulary trained on vastly more varied data. The small course tokenizer remains useful: every merge is inspectable, its entire training procedure fits in one module, and all inputs still roundtrip.

See the versioned BPE baseline for the exact hash and acceptance contract.

Explain · Make the engineering decision

In your lab notes, answer:

  1. Why does byte fallback remove ordinary-text OOV failures?
  2. Why can an OOV-free tokenizer still produce long sequences?
  3. Which parameter matrices grow with the vocabulary?
  4. Why are merge rank and deterministic tie-breaking part of the model contract?
  5. Why must control tokens be distinguished from identical text in documents?
  6. Which vocabulary size would you choose for this corpus, and what evidence supports it?

Completion gate

You pass when all Roundtrip and edge-case tests succeed and you can discuss the three-way trade-off among vocabulary size, sequence length, and OOV behavior. Submit:

  • your report.json with the expected training hash;
  • both SVG visualizations;
  • one Unicode and one special-token Roundtrip;
  • a 280-versus-300-or-340 controlled comparison;
  • a short decision that mentions sequence cost and embedding/output size.

Run the contract checks:

pytest tests/test_tokenizer.py

What’s next

Next, the tokenizer enters a streaming data pipeline with manifests, contamination checks, deterministic shuffling, padding masks, and token batches for training.

← T01 — Bigram baseline · T03 — Data pipeline & embeddings → · Glossary

Was this page helpful?