---
title: "T02 — Byte-level BPE tokenizer from scratch"
sidebar:
  label: "T02 — Byte-level BPE"
---

<span id="t02-byte-level-bpe-tokenizer-from-scratch" />


[← T01 — Bigram baseline](/bigram-baseline) · [T03 — Data pipeline & embeddings →](/data-pipeline) · [Glossary](/glossary)

## Learning outcome [#learning-outcome]

In 90–120 minutes, you will train, encode, and decode with your own byte-level
Byte Pair Encoding tokenizer. You will preserve arbitrary Unicode, make special
tokens explicit, visualize merge decisions and token lengths, and compare the
small course vocabulary with a production tokenizer. No learner-owned data,
cloud, or accelerator is required.

## Predict · Characters are not bytes [#predict-characters-are-not-bytes]

Predict the UTF-8 byte counts of `A`, `ü`, `你`, and `👩🏽‍💻`. Then check:

```
for text in ("A", "ü", "你", "👩🏽‍💻"):
    print(text, list(text.encode("utf-8")))
```

A Python string contains Unicode text; storage and models need integers.
UTF-8 maps every valid string to bytes `0..255`. BPE starts from those 256 byte
IDs and learns larger pieces. Therefore ordinary text never needs an unknown
token, even when its language or emoji never appeared in training.

Run the complete reference:

```
python examples/run_tokenizer_lab.py
```

It writes only ignored local evidence under `artifacts/tokenizer/`:

- `report.json` — corpus hash, vocabulary, merge trace, token pieces and lengths;
- `merge-trace.svg` — the first ordered merge decisions;
- `token-lengths.svg` — UTF-8 bytes versus course and production token counts.

## Trace · One merge with concrete numbers [#trace-one-merge-with-concrete-numbers]

For `abababab`, UTF-8 starts as eight IDs:

```
97 98 97 98 97 98 97 98
```

The pair `(97,98)` occurs four times. Merge it into new ID `256`:

```
256 256 256 256
```

Now adjacent pair `(256,256)` has frequency three; left-to-right replacement is
non-overlapping and produces two ID `257` tokens. Pair counting and replacement
are deliberately separate operations.

In the real run, `train_byte_bpe` repeats this loop:

1. count every adjacent pair in the current training sequence;
2. choose the highest frequency, breaking ties by the smaller ID pair;
3. assign the next ID and replace non-overlapping occurrences;
4. stop at the vocabulary budget or minimum frequency.

The deterministic tie-break, fixed corpus generator, and SHA-256 hash make the
merge order reproducible.

## Build · Encode and decode [#build-encode-and-decode]

The reference vocabulary has 300 IDs:

| Range | Purpose |
| --- | --- |
| `0..255` | complete byte fallback |
| `256..295` | 40 learned merge tokens |
| `296..299` | &lt;&#124;pad&#124;&gt;, &lt;&#124;bos&#124;&gt;, &lt;&#124;eos&#124;&gt;, &lt;&#124;unk&#124;&gt; |

Encoding first converts ordinary text to UTF-8 bytes. It repeatedly applies the
currently available merge with the earliest training rank. Decoding expands
merge IDs back to bytes and performs one strict UTF-8 decode. Test the invariant:

```
text = "Grüezi 你好 👩🏽‍💻 cafe\u0301"
ids = tokenizer.encode(text)
assert tokenizer.decode(ids) == text
```

The composed `é` and decomposed `e` plus combining accent have different byte
sequences. The tokenizer does not silently normalize either one; both roundtrip.

## Build · Treat control tokens deliberately [#build-treat-control-tokens-deliberately]

Registered special tokens become atomic only when explicitly allowed:

```
ids = tokenizer.encode(
    "<|bos|>hello<|eos|>",
    allowed_special={"<|bos|>", "<|eos|>"},
)
```

Without `allowed_special`, the same visible text is ordinary UTF-8 content.
This distinction prevents an untrusted document from silently becoming a model
control sequence. `<|unk|>` is reserved for interoperability; byte fallback
means ordinary Unicode input never emits it.

## Break · Find the trade-off [#break-find-the-trade-off]

Keep the corpus and minimum frequency fixed, then run:

```
python examples/run_tokenizer_lab.py --vocabulary-size 280
python examples/run_tokenizer_lab.py --vocabulary-size 340
```

Predict the direction first. A larger vocabulary usually shortens familiar
training-like sequences, but enlarges the model's embedding and output matrices.
For hidden size `C`, adding 60 tokens adds `60 × C` embedding parameters and,
without tied weights, another `60 × C` output parameters.

Now encode a script absent from training. Token count may approach its UTF-8
byte count, yet decoding still succeeds. This separates efficiency from OOV:
unfamiliar text can be inefficient without being unrepresentable.

## Measure · Compare with a production tokenizer [#measure-compare-with-a-production-tokenizer]

The lab imports `tiktoken` only after your implementation is trained and uses
`cl100k_base` as a production reference. The library is not used to implement
or validate the course BPE algorithm.

With vocabulary 300, the reference run produces:

| Evaluation text | UTF-8 bytes | Course BPE | `cl100k_base` |
| --- | ---: | ---: | ---: |
| English sentence | 37 | 19 | 8 |
| German/Latin accents | 29 | 25 | 11 |
| CJK and emoji | 29 | 25 | 14 |
| composed/decomposed accents | 37 | 29 | 11 |
| **Total** | **132** | **98** | **44** |

The production tokenizer wins on length because it has a much larger vocabulary
trained on vastly more varied data. The small course tokenizer remains useful:
every merge is inspectable, its entire training procedure fits in one module,
and all inputs still roundtrip.

See the versioned [BPE baseline](https://github.com/minigraphx/llm-engineering-course/blob/main/docs/baselines/bpe-v1.md)
for the exact hash and acceptance contract.

## Explain · Make the engineering decision [#explain-make-the-engineering-decision]

In your lab notes, answer:

1. Why does byte fallback remove ordinary-text OOV failures?
2. Why can an OOV-free tokenizer still produce long sequences?
3. Which parameter matrices grow with the vocabulary?
4. Why are merge rank and deterministic tie-breaking part of the model contract?
5. Why must control tokens be distinguished from identical text in documents?
6. Which vocabulary size would you choose for this corpus, and what evidence supports it?

## Completion gate [#completion-gate]

You pass when all Roundtrip and edge-case tests succeed and you can discuss the
three-way trade-off among vocabulary size, sequence length, and OOV behavior.
Submit:

- your `report.json` with the expected training hash;
- both SVG visualizations;
- one Unicode and one special-token Roundtrip;
- a 280-versus-300-or-340 controlled comparison;
- a short decision that mentions sequence cost and embedding/output size.

Run the contract checks:

```
pytest tests/test_tokenizer.py
```

## What's next [#whats-next]

Next, the tokenizer enters a streaming data pipeline with manifests,
contamination checks, deterministic shuffling, padding masks, and token batches
for training.

[← T01 — Bigram baseline](/bigram-baseline) · [T03 — Data pipeline & embeddings →](/data-pipeline) · [Glossary](/glossary)
