Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Tokens and Embeddings

Tokenization Basics

See why LLMs split text into tokens, compare character, word and subword tokenizers in Python, and learn how tokens affect cost, context and odd model behavior.

A neural network works with numbers, so text must be converted before the model sees it. Tokenization is the step that splits text into pieces called tokens and maps each piece to an integer id. Everything the model knows is expressed in tokens: its context limit, its price, and many of its quirks (such as struggling to count letters in a word) come from this step.

Three ways to split text

There are three obvious choices, and each has a flaw.

  • Characters. The vocabulary is tiny (a few hundred symbols) and nothing is ever unknown. But sequences become very long, so the model spends its limited context window on individual letters and must learn spelling-to-meaning from scratch.
  • Words. Each token is meaningful and sequences are short. But the vocabulary must contain every word form (run, runs, running, ran), any unseen word is out of vocabulary, and typos, names, and new words break it.
  • Subwords. Frequent words stay whole; rare words are split into reusable pieces like un + break + able. The vocabulary stays moderate (tens of thousands of entries), sequences stay reasonably short, and nothing is unknown because any text can be spelled with smaller pieces. Nearly every modern LLM uses subwords.
Python Characters, words and a toy subword split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The greedy longest-match splitter is a simplification (real tokenizers such as BPE or Unigram use learned rules, see Byte-Pair Encoding in Practice), but it shows the principle: known pieces are used when possible and single characters are the safety net.

Token ids and the vocabulary

After splitting, each piece is looked up in a table that maps it to an integer. Decoding is the reverse lookup followed by string concatenation. Encoding followed by decoding should give back exactly the original text.

Python Encode and decode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the last line: a character the vocabulary cannot spell becomes <unk> and the round trip loses information. Real tokenizers avoid this by working at the byte level, so any text in any language, emoji included, can always be represented.

Why subword tokenization has side effects

Because the model sees tokens, not letters, tasks that depend on letters are harder. Counting the r characters in "strawberry" requires the model to know the spelling of tokens it only ever sees as single units. Arithmetic is awkward when a number is cut at arbitrary places. A model's context window and your bill are both counted in tokens, not words, and languages that were rare in the tokenizer's training data need more tokens per sentence, so they are effectively more expensive and fit less in context.

Python Tokens per word differs by text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This stand-in is not a real tokenizer, but the trend is real: common English words are about one token each, while rare words, long numbers, and unusual text fragment into many.

Choosing a token unit

Characters Tiny vocabulary, very long sequences.
Words Short sequences, endless vocabulary.
1 / 2

Practical consequences

  1. Count tokens, not words, when you budget for context or cost. A rough English rule is about three quarters of a word per token, but always measure with the model's own tokenizer.
  2. Whitespace and case matter. " the" (with a leading space) and "the" are usually different tokens.
  3. Never mix tokenizers. Ids only mean something with the vocabulary they came from. Using the wrong tokenizer with a model produces nonsense.
  4. Tokenizers are fixed after training. Changing the vocabulary means retraining or at least careful surgery on the embedding table.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main drawback of word-level tokenization?

    Every word form needs an entry and new or misspelled words have none.
  2. 2. Why do most LLMs use subword tokenization?

    Subwords give a middle ground: common words are single tokens and rare words are built from pieces.
  3. 3. Why can a model struggle to count letters in a word?

    A multi-letter token is a single unit to the model, so letter-level facts must be learned indirectly.
  4. 4. What does byte-level tokenization guarantee?

    Every string is a sequence of bytes, so the byte fallback can always represent it.
  5. 5. You pass ids from tokenizer A to a model trained with tokenizer B. What happens?

    Ids are indices into one specific vocabulary and mean nothing elsewhere.

Summary

  • Tokenization turns text into integer ids; the model only ever sees ids.
  • Characters give long sequences, words give an unbounded vocabulary, subwords balance both.
  • Byte-level fallback removes unknown tokens.
  • Context windows, prices, and letter-level weaknesses are all consequences of tokens.
  • Always use the exact tokenizer that belongs to the model.

Next: Byte-Pair Encoding in Practice