Tokenization converts raw text into the integer sequences that transformers process.
Byte Pair Encoding (BPE):
- Start with individual characters as tokens
- Find the most frequent pair of adjacent tokens in the corpus
- Merge that pair into a new token
- Repeat until vocabulary reaches target size (typically 32K–100K)
Example: "lower" → after merges → ["low", "er"] (common subwords become single tokens)
Why subword tokenization:
- Character-level: sequences too long, vocabulary too small
- Word-level: huge vocabulary, can't handle new/rare words
- BPE: balanced vocabulary size and coverage. Common words = single token; rare words = split into known subwords
Key properties:
- Tokenizer is trained separately, before the model (on the same or similar corpus)
- The tokenizer defines the model's "alphabet" — the model sees token IDs, not text
- Different tokenizers → different token counts for the same text
- Tokens ≠ words: "tokenization" might be ["token", "ization"] or ["tok", "en", "ization"]
Impact on cost/performance:
- Context window is measured in tokens, not words (~1 token ≈ 0.75 words in English)
- Efficient tokenization = shorter sequences = faster training + longer effective context
- SentencePiece (used by LLaMA) is a common alternative to BPE
See also: Pretraining, Autoregressive Generation