Step by Step
B
Begin with characters — the starting vocabulary
BPE starts with a vocabulary of individual characters, the smallest possible units, before any merging has occurred.
Example: starting with a vocabulary containing just individual letters like "a," "b," "c," and so on.
P
Pair merging — combining the most frequent pair repeatedly
The algorithm repeatedly counts all adjacent character (or token) pairs in the training text and merges the single most frequent pair into a new combined token, repeating this process many times.
Example: if "t" and "h" appear adjacent to each other extremely frequently, they get merged into a new single "th" token.
E
End at target vocabulary size — common words as whole tokens
This merging process repeats until reaching a target vocabulary size (commonly 50,000-100,000 tokens for GPT models), resulting in common whole words becoming single tokens while rare words get split into smaller subword units.
Example: a common word like "the" likely becomes a single token, while a rare, unusual word might get split into two or three subword pieces.
⭐
No true out-of-vocabulary words — subwords handle anything
Because any word can ultimately be broken down into some combination of its learned subword units (down to individual characters if necessary), there's no true out-of-vocabulary problem — any input text can always be tokenized, even words never seen during training.
Example: even a brand-new made-up word can still be tokenized by breaking it down into familiar subword pieces or, in the worst case, individual characters.
Applied Walkthrough
1
A tokenizer trained with BPE encounters a word it has genuinely never seen before during training.
2
Ask: does this cause an out-of-vocabulary error, the way older tokenization schemes might? No — BPE can always break the unfamiliar word down into smaller subword pieces (or, worst case, individual characters) that it has seen before.
3
This means there's no true out-of-vocabulary problem with BPE-based tokenization, unlike older whole-word-only vocabulary approaches.
4
This is exactly why modern LLMs, using BPE-style subword tokenization, can handle genuinely novel words, typos, and even made-up terms without failing outright.
Exam Application
Exams test whether you understand the core BPE algorithm (starting from characters, repeatedly merging the most frequent pair, stopping at a target vocabulary size) and whether you know its key benefit — eliminating true out-of-vocabulary words. Also expect a note that token counts vary by language, with English roughly at 750 words per 1000 tokens.
⚠ Common Trap
The most common trap is assuming tokens always correspond exactly to whole words. Common words often do become single tokens, but rare or unusual words get split into multiple subword tokens — meaning token count and word count are not the same thing, which matters for cost, speed, and context window calculations.
✓ Quick Self-Check
1. What vocabulary does BPE start with, before any merging?
Individual characters.
Tap to reveal / hide
2. What does the BPE algorithm repeatedly do to build up its vocabulary?
Counts all adjacent character/token pairs and merges the most frequent pair into a new combined token, repeating until reaching a target vocabulary size.
Tap to reveal / hide
3. Why does BPE eliminate the true out-of-vocabulary word problem?
Because any word can be broken down into some combination of learned subword units, even down to individual characters if necessary.
Tap to reveal / hide
4. What is a typical target vocabulary size for GPT-style models?
50,000 to 100,000 tokens.
Tap to reveal / hide
5. Is 750 English words approximately equal to 750 tokens, or more?
More — approximately 1000 tokens, since common words often become one token but many words split into multiple subword tokens.
Tap to reveal / hide