Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 37 ยท Tokens and Embeddings

Vocabulary, Special Tokens and Chat Templates

Learn what special tokens like BOS, EOS and padding do, how chat templates turn messages into one token stream, and why template mistakes hurt output quality.

A tokenizer vocabulary holds more than ordinary text pieces. It also contains special tokens: reserved entries that carry structure rather than words. They mark where a document starts and ends, fill empty space in a batch, and, in chat models, separate the system prompt, the user, and the assistant. Getting them wrong is one of the most common reasons a model that "should work" gives poor answers.

Common special tokens

Token role Typical purpose
BOS (beginning of sequence) Marks the start of a text; gives the first real token something to attend to
EOS (end of sequence) The model emits this to say "I am finished"; generation stops when it appears
PAD Fills shorter sequences in a batch up to equal length; ignored by attention and loss
UNK Stands for an unknown piece (rare with byte-level tokenizers)
Role markers Delimit system, user, assistant, and tool messages in chat models
Control tokens Mark things like images, tool calls, or the start of reasoning

The exact spellings differ by model family, so never hard-code them. Read them from the tokenizer's own configuration.

Python A tiny tokenizer with special tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mask tells attention and the loss which positions are real. The unseen word banana became <unk> here; a byte-level tokenizer would split it into known pieces instead.

Why EOS matters for generation

A model outputs one token at a time and has no other way to say it is done. If it produces the EOS token, the generation loop stops. If your code ignores EOS or never includes it in training data, a model will happily continue past the end of its answer and start inventing the next user turn. When you fine-tune, make sure every training example ends with the EOS token so the model learns to stop.

Python Stop on EOS

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Chat templates

A chat model is trained on conversations flattened into a single stream of tokens. The API you use takes a list of messages, but the model receives one long string in which role markers separate each message. The conversion rule is the model's chat template. Here is a made-up template to show the idea (real ones use different markers):

TEXT
<|system|>You are a helpful assistant.<|end|>
<|user|>What is 2 + 2?<|end|>
<|assistant|>

At the end the template adds the marker that opens the assistant turn; the model then generates the answer and finishes with its end-of-turn token. During fine-tuning the same template is applied to every example, including the assistant answer, so the model learns exactly where to speak.

Python Render a chat template

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mistakes that quietly hurt quality

  • Wrong template for the model. Feeding a model a format it was never trained on lowers answer quality and can break stopping. Use the template shipped with the model's tokenizer.
  • Double BOS. If the template already adds a BOS token and your code adds another, the model sees an unusual prefix.
  • Missing generation prompt. Without the opening assistant marker, the model may continue the user's message instead of answering.
  • Special tokens in user text. If a user types the literal string <|end|> and the tokenizer converts it into the real special token, they can forge conversation structure. Good tokenizers treat user text as plain text, and applications should escape or reject control strings. This is a root cause of some prompt-injection tricks.
  • Padding side and EOS confusion. For batched generation, decoder models usually need left padding so the real tokens sit next to the generation point, and the pad token should not be the same as EOS unless you mask it carefully.

One conversation, one stream

System Sets behavior and rules.
User Asks the question.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main job of the EOS token during generation?

    The model emits EOS to end its answer; the generation loop stops when it appears.
  2. 2. Why are PAD tokens masked out of attention and the loss?

    Padding only equalizes lengths in a batch; real tokens are marked with the mask.
  3. 3. What is a chat template?

    Chat models are trained on a specific flattened format, and the template reproduces it.
  4. 4. Why is it a security issue if user text is converted into real special tokens?

    If control strings become real role tokens, input can impersonate other roles.

Summary

  • Vocabularies include special tokens for structure: BOS, EOS, PAD, UNK, role markers, and control tokens.
  • EOS lets a model stop; always train examples that end with it.
  • A padding mask keeps filler from affecting attention and loss.
  • A chat template flattens messages into a single stream and must match the model.
  • Template mistakes, double BOS, missing generation prompts and forged control strings are common sources of bad output and security problems.

Next: Token Embeddings