Generative AI
Byte Pair Encoding
A tokenizer is built by counting: glue the commonest pair of neighbouring symbols into one, and repeat. Then watch a new sentence get cut up.
the pair about to mergethe symbol a merge just madea learned tokennot in the vocabulary
▁ marks the start of a word · drag Merges applied to scrub
Vocabulary size |V|—
—
Values
distinct words—
starting alphabet—
merges learned—
vocabulary size—
sentence tokens—
characters per token—
What to observe
- At zero merges the vocabulary is the alphabet and the sentence costs one token per character.
- Each merge glues the commonest pair, counted over the whole text: first a + t, then ▁t + h and ▁th + e, so "the" is now one token.
- After every merge the sentence drops from 27 tokens to 11. flatnever appears in training, yet it still tokenizes, as ▁f, l, at.
- A letter the training text never had goes red: it is not in the vocabulary. That is why real tokenizers start from the 256 bytes.
Shortcuts:space run/pause · s step · r reset · f fullscreen