Byte Pair Encoding

What to observe

1. At zero merges the vocabulary is just the alphabet, and every word is one token per letter. The sentence at the bottom costs as many tokens as it has characters. That is the worst tokenizer that still works.
2. Step once. The pair panel shows why that pair won: it is simply the one with the most occurrences across the whole text, weighted by how often each word appears. Nothing linguistic is consulted, only counting.
3. Keep stepping and watch common short words get swallowed whole:▁t + h becomes ▁th, then ▁the. Frequent words end up as single tokens while rare ones stay in pieces, which is exactly the trade a tokenizer is for.
4. Watch the sentence at the bottom as merges accumulate. The same characters keep collapsing into fewer tokens, and characters per token climbs. That number is what a tokenizer is judged on: it is how much text fits in a context window.
5. The test sentence contains words the training text never had. They still tokenize, because whatever is left over falls back on shorter pieces, down to single letters. A BPE tokenizer can never be stumped by a new word, only made inefficient by it.
6. Type a word with a letter that does not appear in the training text at all and it goes red. That is the one real failure mode, and why real tokenizers work on bytes: with 256 byte values in the base vocabulary there is nothing left to be unknown.
7. Edit the training text and the vocabulary is rebuilt from it. Feed it one topic and the tokenizer becomes efficient at that topic and clumsy at everything else, which is why tokenizers trained on English cut other languages into far more tokens per word.