Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 37 ยท Tokens and Embeddings

Byte-Pair Encoding in Practice

Train a byte-pair encoding tokenizer from scratch in Python: count pairs, learn merges, encode and decode text, and see how vocabulary size changes token counts.

Byte-pair encoding (BPE) is the algorithm behind the tokenizers of most GPT-style models. It was originally a data-compression trick: repeatedly replace the most frequent pair of adjacent symbols with a new symbol. Applied to text, it learns a subword vocabulary from data. Frequent words end up as single tokens, and rare words are spelled from smaller pieces, down to single bytes.

The training algorithm

  1. Start with a base vocabulary of single symbols (characters or, in byte-level BPE, the 256 byte values).
  2. Count every adjacent pair of symbols in the training text.
  3. Merge the most frequent pair into one new symbol and add it to the vocabulary. Record the merge.
  4. Repeat until the vocabulary reaches the target size.

The ordered list of merges is the tokenizer. To encode new text, split into base symbols and apply the merges in the order they were learned.

Python Train BPE on a tiny corpus

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Watch how e + s, then es + t, then est + </w> appear early: the suffix est is frequent in newest and widest, so it becomes one token. The word low grows into a single symbol too. Meaning-bearing pieces emerge from nothing but frequency.

Encoding new text

Once you have the ordered merges, encoding is replaying them on the new text. Unseen words still work because they decompose into known pieces.

Python Encode with learned merges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

slowest never appeared in the training text, yet it encodes into familiar pieces (slow and est), which is the point of subwords. zzz has no merges at all, so it falls back to single characters.

From characters to bytes

The version above works on characters, so a character the training data never contained would be unknown. Byte-level BPE starts from the 256 possible byte values of the UTF-8 encoding. Every string, in any script, including emoji and code, is then representable with zero unknown tokens. Merges still learn frequent byte sequences like " the" or "ing". The cost is that non-Latin characters take several bytes, so rarely seen scripts break into more tokens.

Python Bytes are the universal alphabet

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Vocabulary size trade-offs

A bigger vocabulary means more merges, so text becomes fewer tokens (shorter sequences, cheaper attention, more text per context window). But the embedding table and output layer both grow with the vocabulary, and rare tokens get little training signal. Typical production vocabularies range from about 32,000 to over 200,000 entries.

Python More merges, fewer tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The count drops fast at first and then flattens: after the common pieces are merged, each extra merge saves little. That diminishing return is how vocabulary size is chosen.

Other algorithms

  • WordPiece (used by BERT) is similar but picks merges that maximize training-data likelihood rather than raw frequency.
  • Unigram (used by SentencePiece) starts with a large vocabulary and prunes it, and can produce several possible segmentations.
  • SentencePiece is a library that treats the text as a raw stream (spaces included), so it needs no language-specific pre-splitting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In each BPE training step, what is merged?

    BPE repeatedly merges the most frequent adjacent pair into a new symbol.
  2. 2. How does a trained BPE tokenizer encode new text?

    The ordered merge list is the tokenizer; encoding applies it to the new text.
  3. 3. What advantage does byte-level BPE have?

    All text is a sequence of bytes, and the 256 byte values are in the base vocabulary.
  4. 4. What is a cost of a very large vocabulary?

    The embedding and output layers grow with vocabulary size and rare tokens are poorly trained.

Summary

  • BPE learns a subword vocabulary by repeatedly merging the most frequent adjacent pair.
  • The ordered list of merges is the tokenizer; encoding replays them.
  • Byte-level BPE guarantees coverage of any text.
  • More merges mean fewer tokens per text, with diminishing returns and larger embedding tables.
  • WordPiece and Unigram are alternatives; SentencePiece is a widely used toolkit.

Next: Vocabulary, Special Tokens and Chat Templates