Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 38 ยท Hub, Models and Tokenizers

Subword Tokenizers: BPE and WordPiece

Learn how Byte-Pair Encoding learns merges from a corpus and how WordPiece splits words greedily, with runnable Python implementations you can edit.

Modern tokenizers learn their vocabulary from data. The two most important families are Byte-Pair Encoding (BPE), used by GPT-2, Llama-style and RoBERTa tokenizers, and WordPiece, used by BERT. A third, Unigram (used with SentencePiece by T5 and many multilingual models), prunes a large vocabulary instead of growing a small one. This lesson implements the first two so you can see exactly what they do.

Byte-Pair Encoding: learning merges

BPE starts with every character as a token. It then repeatedly counts adjacent pairs of tokens in the corpus, merges the most frequent pair into a new token and records the merge rule. After a chosen number of merges (or when the vocabulary reaches a target size) training stops. To tokenize new text you split it into characters and apply the recorded merges in the learned order.

The runnable code below is a stand-in: a compact BPE trainer on a toy corpus, small enough to read. Real tokenizers do the same on gigabytes of text, with byte-level handling and speed tricks.

Python Stand-in: train a tiny BPE tokenizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the output: frequent pieces such as est_ (the end of "newest" and "widest") and low appear as single tokens. The unseen word "lowest" is built from known pieces instead of becoming [UNK]. Try adding more merges or new corpus words and watch the vocabulary grow.

Byte-level BPE

GPT-2 and many successors run BPE on bytes rather than characters. The base vocabulary is the 256 possible byte values, so any text, in any language, emoji included, can be encoded and nothing is ever unknown. The price is that rare characters take several tokens.

WordPiece: greedy longest match

BERT's WordPiece produces a vocabulary in which continuation pieces are marked with ##. To tokenize a word it takes the longest prefix found in the vocabulary, then continues from the next position, always looking for the longest ##-prefixed piece. If no piece matches at some position the whole word becomes [UNK].

(Training chooses merges by a likelihood score rather than raw frequency, but the tokenization step is this greedy longest-match search.)

Python Stand-in: WordPiece greedy matching

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

"playing" works because play and ##ing are in the vocabulary, "unbelievable" becomes un, ##believ, ##able, and "xyz" fails entirely, which is exactly how BERT behaves for a word it cannot cover.

Unigram and SentencePiece

SentencePiece is a toolkit that treats text as a raw stream, including spaces (shown as โ–), so no pre-splitting on whitespace is needed. This suits languages without spaces, such as Japanese and Chinese. Its Unigram model starts with a large candidate vocabulary and removes the pieces that hurt the corpus likelihood least until the target size is reached, then picks the most probable segmentation for each text.

Algorithm Starts from Used by
BPE characters or bytes, adds merges GPT-2, Llama, RoBERTa
WordPiece characters, adds by likelihood BERT, DistilBERT, Electra
Unigram (SentencePiece) big vocabulary, prunes T5, ALBERT, XLM-R

Training your own tokenizer

The tokenizers library trains these algorithms very fast (read-only):

Python
from tokenizers import Tokenizer, models, trainers, pre_tokenizers

tokenizer = Tokenizer(models.BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=8000, special_tokens=["[UNK]", "[PAD]"])
tokenizer.train(files=["corpus.txt"], trainer=trainer)
tokenizer.save("my-tokenizer.json")

Train a new tokenizer only when your text (for example source code, chemistry or a rare language) is very different from what existing tokenizers saw. Remember: a new tokenizer needs a model pretrained with it.

flowchart LR
  A["Corpus"] --> B["Count pairs"]
  B --> C["Merge the best pair"]
  C --> D{"Vocabulary full?"}
  D -->|"no"| B
  D -->|"yes"| E["Save vocab and merges"]

BPE training loop.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does BPE do at each training step?

    Each merge adds one token and one rule to the merge list.
  2. 2. What does the ## prefix mean in WordPiece output?

    marks a piece that does not start a word.

  3. 3. Why can byte-level BPE avoid unknown tokens?

    Any text can be broken down to bytes, so there is always a valid encoding.
  4. 4. When is it sensible to train a new tokenizer?

    A tokenizer and its model belong together, so a new tokenizer implies pretraining a new model.

Summary

  • BPE learns merge rules from pair frequencies; WordPiece tokenizes by greedy longest match; Unigram prunes a big vocabulary.
  • Byte-level BPE can encode any text without unknowns.
  • Vocabularies and merge lists belong to a specific pretrained model.
  • The tokenizers library trains custom tokenizers quickly.

Next: AutoTokenizer in Practice