Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 38 ยท Hub, Models and Tokenizers

AutoTokenizer in Practice

Use AutoTokenizer for batches: padding, truncation, attention masks, token type ids, offsets, pairs of texts and chat templates, with runnable stand-ins.

AutoTokenizer.from_pretrained(name) reads the repository's tokenizer files and returns the right tokenizer class for that model. In everyday work you call the tokenizer object directly, as if it were a function, and pass options for batching, length limits and tensor output. This lesson covers the options you will use constantly.

The basic call

Read-only real code:

Python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")

batch = tok(
    ["Short one.", "A somewhat longer second sentence in the batch."],
    padding=True,         # pad to the longest sequence in this batch
    truncation=True,      # cut sequences longer than max_length
    max_length=16,
    return_tensors="pt",  # PyTorch tensors; use "np" for NumPy
)
print(batch.keys())       # input_ids, attention_mask

The result contains:

  • input_ids: the integer ids, with special tokens added.
  • attention_mask: 1 for real tokens, 0 for padding.
  • token_type_ids (for some models such as BERT): 0 for the first text and 1 for the second when you pass a pair.

Padding, truncation and the attention mask

Neural networks process a rectangular tensor, so all sequences in a batch need the same length. Shorter ones are padded with the [PAD] id. The attention mask tells the model to ignore those positions. Without it, padding would change the model's output.

truncation=True cuts anything beyond max_length (or the model's own maximum, often 512). Information past the cut is lost, so for long documents split them into overlapping chunks instead of silently truncating.

Python Stand-in: pad and mask a batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ids here are just numbers chosen for the demo. The point is the mechanism: padding on the right with mask zeros, and truncation applied before padding.

Padding side

Encoders pad on the right. Decoder-only models used for generation usually need left padding (tok.padding_side = "left"), because the next token is generated right after the last position; padding on the right would put padding between the prompt and the new text. Many decoder tokenizers also have no pad token, so you often set tok.pad_token = tok.eos_token.

Pairs of texts

Tasks like natural language inference or question answering take two texts. Pass them as two arguments:

Python
enc = tok("Who wrote it?", "The report was written by Ana.", truncation="only_second")
print(tok.decode(enc["input_ids"]))
# [CLS] who wrote it? [SEP] the report was written by ana. [SEP]

truncation="only_second" shortens the context without cutting the question. token_type_ids mark which tokens belong to which text.

Offsets and word ids (fast tokenizers)

Most tokenizers on the Hub are fast tokenizers (written in Rust). They can tell you where every token came from in the original string and which word it belongs to. That is essential for tagging tasks, where labels are per word but the model works per token.

Python
enc = tok("Hugging Face is in Paris", return_offsets_mapping=True)
print(enc.word_ids())             # e.g. [None, 0, 0, 1, 2, 3, 4, None]
print(enc["offset_mapping"])      # (start, end) character span of each token

Special tokens get None; the pieces of one word share the same word id.

Python Stand-in: align word labels to subword tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The value -100 is the index PyTorch's loss function ignores, so special tokens and continuation pieces do not affect training.

Special tokens, chat templates and saving

  • tok.encode(text, add_special_tokens=False) returns ids without [CLS]/[SEP].
  • tok.decode(ids, skip_special_tokens=True) returns clean text.
  • tok.vocab_size and len(tok) give the vocabulary size; after adding tokens with tok.add_tokens([...]) you must call model.resize_token_embeddings(len(tok)).
  • Chat models carry a chat template: tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) formats a list of {"role", "content"} messages exactly as the model was trained.
  • tok.save_pretrained("folder") writes the tokenizer files so you can reload them offline or upload them.
Python
messages = [
    {"role": "system", "content": "You answer briefly."},
    {"role": "user", "content": "What is a token?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

Never format chat prompts by hand; a wrong template silently degrades a model.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the attention mask contain for padding positions?

    0 tells the model to ignore those positions.
  2. 2. Which padding side do decoder-only generation models usually need?

    Left padding keeps the real prompt right next to the position where new tokens are generated.
  3. 3. What is the label -100 used for in token classification?

    PyTorch's cross-entropy loss ignores the index -100 by default.
  4. 4. Why use word_ids() with a fast tokenizer?

    Word ids let you align per-word labels with per-token model outputs.
  5. 5. After adding new tokens to a tokenizer, what must you do to the model?

    The embedding table needs rows for the new ids: model.resize_token_embeddings(len(tok)).

Summary

  • Call the tokenizer with padding, truncation, max_length and return_tensors to build model-ready batches.
  • attention_mask hides padding; token_type_ids separate paired texts.
  • Fast tokenizers give word ids and offsets, which align word-level labels with subword tokens.
  • Use apply_chat_template for chat models and save_pretrained to keep tokenizer files with the model.

Next: AutoModel, Configs and Heads