Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 38 ยท Hub, Models and Tokenizers

Tokenization Basics

Learn why models need tokenizers, how text becomes token ids, and the trade-offs between word, character and subword tokenization, with runnable examples.

A neural network works on numbers, so before any model sees your text, a tokenizer must convert it into a sequence of integers. The model then looks each integer up in an embedding table. The quality of this first step shapes everything after it: vocabulary size, how rare words are handled, how long sequences become and even which languages a model handles well.

The process has two directions. Encoding turns a string into token ids. Decoding turns ids back into a string. A tokenizer is always tied to a model: the ids a model learned during pretraining only make sense with the exact same vocabulary, so never mix tokenizers and models from different families.

Three granularities

Word tokenization splits on spaces and punctuation. It is easy to understand, but the vocabulary explodes (every form of every word), and any unseen word becomes an [UNK] token that carries no information.

Character tokenization uses single characters. The vocabulary is tiny and nothing is unknown, but sequences become very long and each token carries little meaning, which makes learning harder and slower.

Subword tokenization is the compromise used by every modern transformer. Frequent words stay whole, rare words are split into frequent pieces: tokenization might become token + ization. The vocabulary stays at 30,000 to 250,000 entries, and unseen words can still be represented from pieces.

Three ways to split text

Words Short sequences, huge vocabulary, unknowns.
Characters Tiny vocabulary, very long sequences.
1 / 2

Build the simplest tokenizer

The following runnable code is a stand-in: a hand-made word and character tokenizer with a vocabulary built from a tiny corpus. It is not what BERT uses, but it shows the encode and decode cycle and the unknown-token problem.

Python Stand-in: word and character tokenizers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

"chased" is not in the vocabulary, so it becomes [UNK] (id 1) and the information is lost. A subword tokenizer avoids this by splitting the unseen word into known pieces; the next lesson builds one.

Special tokens

Besides ordinary pieces, vocabularies contain special tokens with agreed roles:

Token Role
[PAD] / <pad> fills short sequences up to a common length
[UNK] / <unk> a piece the vocabulary cannot represent
[CLS] / <s> start of input; its final vector often summarises the text
[SEP] / </s> marks the end or the boundary between two texts
[MASK] the hidden word in masked language modelling

The exact names differ between models, which is another reason to always use the matching tokenizer.

The real library

With the real library one call encodes text (read-only):

Python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
enc = tok("Tokenizers are fun!")
print(enc["input_ids"])                       # integers, starting with the [CLS] id
print(tok.convert_ids_to_tokens(enc["input_ids"]))
print(tok.decode(enc["input_ids"]))

The result is a dictionary-like object (BatchEncoding) with input_ids and attention_mask. The exact pieces depend on the model, so this course does not print them here.

Why tokens matter in practice

  • Length limits are measured in tokens, not words or characters. A model with a 512-token limit may fit only 300 to 400 English words.
  • Cost and speed of generation scale with token counts.
  • Languages that the vocabulary covers poorly produce many more tokens per sentence.
  • Numbers and code often split into several pieces, which can hurt arithmetic.
Python Stand-in: how splitting changes length

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem does a word-level tokenizer have with unseen words?

    Words outside the vocabulary map to [UNK].
  2. 2. Which approach do modern transformers use?

    Subwords keep the vocabulary manageable while representing rare words from pieces.
  3. 3. Why must a model be used with its own tokenizer?

    Embedding rows were learned for specific ids, so a different vocabulary gives nonsense.
  4. 4. What is the role of [PAD]?

    Padding makes sequences rectangular; the attention mask tells the model to ignore it.

Summary

  • Tokenizers turn text into integer ids (encode) and back (decode).
  • Words give unknowns, characters give long sequences, subwords balance both.
  • Special tokens such as padding, unknown, class and separator have fixed roles.
  • Limits, speed and cost are all measured in tokens.

Next: Subword Tokenizers: BPE and WordPiece