Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 63 ยท Cleaning and Tokenization

Subword Tokenization: BPE and WordPiece

Understand why modern language models split words into pieces. Train a byte-pair encoding tokenizer from scratch in Python and compare it with WordPiece and SentencePiece.

A word-level vocabulary has a hard limit: it can only represent words it has seen. Give it unbelievably, a typo, a new brand name or a word in a rich language such as Finnish, and it falls back to a single [UNK] (unknown) token that throws the word away. You could grow the vocabulary to millions of entries, but then every model gets huge and rare words still have almost no training data.

Subword tokenization is the fix used by nearly every modern language model, from BERT to GPT. Instead of whole words, text is split into frequent pieces: common words stay whole, rare words break into several known parts, and in the worst case into single characters. In this lesson you will train the best-known algorithm, byte-pair encoding (BPE), from scratch. You should know about tokenization basics and how counts work from word frequencies.

The out-of-vocabulary problem

Three ways to cut text

Characters Tiny vocabulary, very long sequences.
Words Short sequences, but unknown words break it.
Subwords Frequent pieces, a good middle.

Subwords trade a small vocabulary for slightly longer text.

1 / 2

Subword units also capture structure: a model that has seen walk, walked and walking can read jumped if it knows jump and ed. The next question is how to choose the pieces. BPE does it from data with one simple loop.

How byte-pair encoding works

BPE starts with each word split into characters, and an end-of-word marker so the model can tell er inside a word from er at the end. Then it repeats one step:

  1. Count how often each adjacent pair of symbols occurs, weighted by word frequency.
  2. Merge the most frequent pair into one new symbol, and record the merge.
  3. Repeat until you have made the number of merges you want.

The list of merges, in order, is the tokenizer. The vocabulary size is the number of starting characters plus the number of merges, so you control it with one number.

Python Training BPE from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at how the words are rebuilt. After 12 merges, newer is a single symbol because it was frequent, while lower is still lo we r</w>, three symbols, because its parts were merged but the whole word never became popular enough. The tokenizer was built purely from counts: nothing told it that -er or -est are English endings, yet pieces like r</w> and st</w> appear because those pairs are frequent.

Ties are broken by taking the larger pair alphabetically. Any consistent rule works, but a tokenizer must always break ties the same way, or the same text would produce different tokens after retraining.

Encoding new words

To tokenize new text, replay the learned merges in the same order on the new word's characters. Words that were seen in training collapse into few tokens; words never seen still tokenize, because the worst case is single characters. Compare this with a word-level vocabulary that has no choice but [UNK].

Python Tokenizing unseen words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

slowest was never in the training data, yet BPE reads it as s plus the familiar lo we st</w>, so a model keeps most of the meaning. zebra shares nothing with the training words and falls all the way to single characters, which is slow and carries little information, but it is still representable. That is the guarantee subwords give: nothing is out of vocabulary.

Choosing the vocabulary size

More merges make a bigger vocabulary and shorter token sequences. Fewer merges give a small vocabulary and long sequences. Real models pick something in between: BERT's vocabulary has about 30,000 entries and many GPT-style models use 50,000 to 200,000, trained on gigabytes of text, not eight words.

Python Merges versus sequence length

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weighted average falls as merges are added, and reaches 1 when every word has become its own token, which is simply a word-level vocabulary again. The two ends of this table are the character-level and word-level tokenizers from the first slide. Subword tokenization lets you pick the point in between.

WordPiece, SentencePiece and byte-level BPE

BPE has relatives that you will meet in practice. They are shown read-only here because the real libraries need to download trained vocabularies, which the browser runner cannot do.

  • WordPiece (BERT) also grows a vocabulary by merging, but chooses the pair that most increases the likelihood of the training data, not simply the most frequent one. It marks pieces that continue a word with a ## prefix and tokenizes by taking the longest matching piece first from the left.
  • SentencePiece works directly on raw text, treating the space as an ordinary symbol (shown as โ–), so it needs no pre-tokenizer and works for languages written without spaces. It offers BPE and a unigram model that starts with a large vocabulary and prunes it.
  • Byte-level BPE (GPT-2 and later) starts from the 256 possible bytes instead of characters. Every string, emoji and script is representable, so there is no unknown token at all.
Python
from transformers import AutoTokenizer

bert = AutoTokenizer.from_pretrained("bert-base-uncased")      # WordPiece
bert.tokenize("unbelievably")        # pieces such as 'un', '##bel', '##ie', '##va', '##bly'

gpt2 = AutoTokenizer.from_pretrained("gpt2")                   # byte-level BPE
gpt2.tokenize(" unbelievably")       # pieces with a leading-space marker

The exact pieces depend on each model's learned vocabulary, so run the code yourself to see them. What matters is the pattern: frequent words come out whole, rare ones in a few parts.

Practical consequences

  • Tokens, not words, are the unit. Model limits and prices are counted in tokens, and English averages somewhat more than one token per word, with rarer languages and code often needing more.
  • Train the tokenizer on text like yours. A vocabulary built on English news wastes tokens on medical terms or another language.
  • The tokenizer is part of the model. Never swap it after training; token IDs would no longer mean the same thing. Use the one that ships with the model.
  • Odd behavior has token causes. Models can struggle to count letters or reverse words partly because they see pieces, not characters.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does subword tokenization mainly solve?

    Any unseen word can be built from known pieces or characters, so nothing is out of vocabulary.
  2. 2. In each BPE training step, which pair of symbols is merged?

    BPE repeatedly merges the most frequent adjacent pair, and the ordered list of merges becomes the tokenizer.
  3. 3. How do you tokenize a new word with a trained BPE model?

    Merges apply in the order they were learned, so frequent combinations form first.
  4. 4. What happens to the vocabulary size and tokens per word as you add more BPE merges?

    Each merge adds one symbol and shortens some sequences. At the extreme, every word is one token.
  5. 5. Why does byte-level BPE never need an unknown token?

    Any string is a sequence of bytes, and all byte values are in the starting vocabulary.

Summary

  • Word-level vocabularies fail on unseen words; subwords split rare words into frequent pieces.
  • BPE starts from characters and repeatedly merges the most frequent adjacent pair; the ordered merges are the tokenizer.
  • Encoding replays the merges, and worst case falls back to characters, so there are no unknowns.
  • The number of merges sets the vocabulary size and trades against sequence length.
  • WordPiece scores merges by likelihood and uses ##; SentencePiece works on raw text; byte-level BPE starts from bytes.

Next, put the whole chapter together in Building a Reusable Preprocessing Pipeline.