Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 53 ยท Tokens and Embeddings

Tokens and Tokenization

Learn why language models read tokens instead of words, how subword vocabularies work, what special tokens do, and why numbers and spaces behave oddly.

A language model does not see letters or words. It sees a sequence of integers, and each integer stands for a token: a chunk of text such as the, cat, ing or a single punctuation mark. Before any text reaches the model, a program called the tokenizer cuts it into tokens and looks up each one's ID. After the model answers with IDs, the tokenizer turns them back into text.

Tokenization looks like a boring preprocessing step, but it explains many things that confuse beginners: why models struggle to count letters, why prices and limits are quoted in tokens, and why the same sentence costs more in some languages. In this lesson you will build a toy tokenizer and see these quirks yourself. If you have not yet seen how a model produces text one piece at a time, read How LLMs Are Trained first.

Text becomes numbers

Split The tokenizer cuts text into pieces.
Look up Each piece has an integer ID.
Model The network only ever sees the IDs.
Decode Output IDs are turned back into text.

The tokenizer is fixed before training and never changes afterward.

1 / 2

Three ways to cut text

There are three obvious choices for what a "piece" should be.

Characters. The vocabulary is tiny (a few hundred symbols), and nothing is ever unknown. But sequences become very long, and the model must learn from scratch that c, a, t together mean something.

Whole words. Sequences are short and each token carries meaning. But the vocabulary must contain every word, including run, runs, running and every typo and name. Any word not in the list becomes an "unknown" token and its meaning is lost.

Subwords. The middle path used by nearly all modern models. Common words get their own token, rare words are split into familiar pieces (un + believ + able), and in the worst case a word falls back to single characters or bytes. The vocabulary stays at a manageable size (typically 30,000 to 250,000 entries) and nothing is ever unknown.

Run this example to compare the three on one sentence.

Python Characters, words and subwords

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Characters give the longest sequence, words the shortest, and subwords land in between while still handling unusual words.

The vocabulary and the ID table

The tokenizer's vocabulary is a fixed table from token string to integer ID. It is built once, by running an algorithm over a large text sample (the most common one is byte-pair encoding), and then frozen. The model is trained with those exact IDs, so you can never swap tokenizers on a trained model.

Two details matter in practice:

  • Spaces belong to tokens. Most English tokenizers attach the leading space to the word that follows, so cat and cat are different tokens with different IDs.
  • Case matters. Hello, hello and HELLO are usually different tokens, and a rare capitalization may take several pieces.

Special tokens

Besides ordinary text pieces, vocabularies reserve a few special tokens that carry structure instead of language:

Token (example names) Purpose
<bos> / <s> Marks the beginning of a sequence
<eos> / </s> Marks the end; the model emits it to say "I'm done"
<pad> Fills shorter sequences so a batch has equal length
<unk> Stands for an unknown piece (rare in byte-level tokenizers)
role markers Separate system, user and assistant turns in a chat

Names differ from model to model. Chat models wrap every message in role markers, which is how the same network knows who is speaking. You normally never type these yourself: the API or the library's chat template adds them. This is also a security topic: if user text can contain a literal special token string, it might be interpreted as structure, so well-built tokenizers treat such strings as plain text unless told otherwise.

Token quirks you will meet

Because the model sees chunks, not letters, certain tasks are harder than they look.

  • Counting letters. To a model, strawberry may be two or three tokens. It never "sees" the individual r characters unless it has learned the spelling of each token.
  • Numbers. Many tokenizers split digits into groups, such as 12345 into 123 and 45. Arithmetic across those boundaries is harder to learn, and the same number can tokenize differently depending on the surrounding text.
  • Whitespace. Runs of spaces, tabs and newlines have their own tokens. Indented code uses more tokens than the same code without indentation.
  • Other languages. A vocabulary built mostly from English text has few long pieces for, say, Thai or Hindi. Those sentences split into many small pieces, so they cost more and fill the context window faster. Counting Tokens and Estimating Cost measures this effect.
  • Typos and rare names. A misspelled word is split into odd pieces, which can change how the model treats it.

The next example is a toy tokenizer with a small vocabulary, special tokens, an ID table, and a decode step. Watch how the number and the unfamiliar word behave.

Python A toy tokenizer with IDs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice three things in the output. Everything round-trips: decoding the IDs gives the original text back. Words in the vocabulary take one token each (the, cat, mat), while zzz qq shatters into single characters and tokenizers starts with five single letters because the vocabulary only has token with a leading space. And 12345 becomes 1, 2, 3, 45: pieces that follow the vocabulary, not the human idea of "a number".

Why vocabulary size is a trade-off

A bigger vocabulary means shorter sequences, so each request is faster and cheaper and fits more text in the context window. But the embedding table grows (one vector per token, see Embeddings), and rare tokens are seen so seldom in training that their vectors stay poorly learned. A smaller vocabulary has the opposite problem: long sequences and more computation per sentence. Real models settle between 32,000 and a few hundred thousand entries, and multilingual models lean toward the larger end.

Practical tips

  • Always count tokens with the tokenizer that matches your model. Estimates from word counts are only a first guess.
  • Keep prompts compact: extra whitespace, repeated boilerplate and verbose JSON all cost tokens.
  • Don't ask a model to do character-level work (reverse a word, count letters) without expecting mistakes. Let code do it, or have the model call a tool.
  • When two outputs differ only in spacing or capitalization, remember they are different token sequences to the model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a language model actually receive as input?

    The tokenizer converts text into token IDs, and the model works only with those integers.
  2. 2. Why do modern tokenizers use subwords instead of whole words?

    Subwords cover rare words by splitting them into known pieces, so there is no unknown word and the vocabulary stays manageable.
  3. 3. Why is counting the letters in a word hard for many LLMs?

    A word is usually one to three tokens, so the individual letters are not directly visible to the network.
  4. 4. What is the role of a special token such as the end-of-sequence token?

    Special tokens carry structure (start, end, padding, roles) rather than ordinary text.

Summary

  • Models read sequences of token IDs, not letters or words; a tokenizer converts in both directions.
  • Subword vocabularies balance sequence length against vocabulary size and never meet an unknown word.
  • A leading space is usually part of a token, and case changes the token.
  • Special tokens mark the start, end, padding and chat roles.
  • Quirks such as weak letter counting, split numbers and costly non-English text all come from tokenization.

Next: learn how a vocabulary is actually built in Byte-Pair Encoding from Scratch.