Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 38 ยท Fine-tuning with Trainer

Data Collators and Dynamic Padding

Understand how data collators batch tokenized examples: dynamic padding, attention masks, label padding, masked language modeling and length grouping, with runnable stand-ins.

A neural network processes tensors, and a tensor in a batch must be rectangular. Real sentences have different lengths, so before a batch can be fed to the model the short ones must be padded. The component that turns a list of individual examples into one rectangular batch is called a data collator. It runs at the last moment, on the fly, for every batch.

Choosing padding well is one of the cheapest speed-ups in fine-tuning: padding every example to the model's maximum length can waste more than half of your compute. This lesson shows how collators work, how to measure the waste, and how masked-language-modeling collators create their training labels.

Transformers and PyTorch do not run inside this site. The plain code blocks are real, read-only code. The TryIt blocks are labelled stand-ins in plain Python that behave like the real collators so you can experiment.

Static versus dynamic padding

There are two places to pad. Static padding happens in the tokenizing step: you pass padding="max_length", max_length=512 and every example becomes 512 tokens long, forever. Dynamic padding postpones it: you tokenize without padding and let the collator pad each batch only up to the length of its longest member.

Python
from transformers import AutoTokenizer, DataCollatorWithPadding

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    # no padding here, only truncation
    return tokenizer(batch["text"], truncation=True, max_length=256)

tokenized = dataset.map(tokenize, batched=True)
collator = DataCollatorWithPadding(tokenizer=tokenizer)

trainer = Trainer(model=model, args=args,
                  train_dataset=tokenized["train"],
                  eval_dataset=tokenized["test"],
                  data_collator=collator)

If you pass a tokenizer to Trainer and no collator, it creates DataCollatorWithPadding for you. The collator also builds the attention mask: ones for real tokens and zeros for padding, so the model ignores the padded positions.

What a collator really does

A collator is simply a callable: it takes a list of dictionaries (features of single examples) and returns a dictionary of tensors. This stand-in does what DataCollatorWithPadding does, using lists instead of tensors.

Python Stand-in: a padding collator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three details matter. The padding id comes from the tokenizer (tokenizer.pad_token_id; it is not always 0). The attention mask marks which positions are real. And pad_to_multiple_of=8 rounds the length up so tensor cores on modern GPUs run faster, at the price of a few extra padded tokens.

Measuring the waste

How much does dynamic padding save? Write the arithmetic down. Fake a dataset with realistic length variety (most reviews are short, a few are long) and compare the three strategies: pad everything to a fixed maximum, pad each batch to its longest example, and sort by length first so batches contain similar lengths.

Python Stand-in: padding waste per strategy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With this skewed length distribution, static padding wastes more than 80% of the compute. Dynamic padding in random order helps only a little, because one long example in a batch of 32 forces the whole batch to its length. Sorting by length first brings the waste down to a few percent. The real Trainer implements the last idea as group_by_length=True in TrainingArguments: it still shuffles, but in buckets of similar length. For evaluation, where order does not matter, sorting is free.

Where padding can happen

Static Every example padded to max_length when tokenizing.
Dynamic Collator pads each batch to its longest example.
Grouped Similar lengths share a batch, so almost nothing is padded.

Dynamic padding is the default recommendation for training.

1 / 2

Padding the labels: why -100?

For classification, labels is one integer per example and needs no padding. For token classification (named entities) or language modeling, there is one label per token, so the labels must be padded too. But padded positions must not contribute to the loss. PyTorch's cross-entropy loss skips any target equal to -100 (its ignore_index), so collators such as DataCollatorForTokenClassification pad labels with -100. The same value is used for special tokens you want to exclude.

Masked language modeling collators

Some collators also create labels. DataCollatorForLanguageModeling(tokenizer, mlm=True, mlm_probability=0.15) implements BERT-style masked language modeling: it picks 15% of the tokens as prediction targets and, of those, replaces 80% with the [MASK] token, 10% with a random token and leaves 10% unchanged. Labels are -100 everywhere except at the picked positions, where the label is the original token. With mlm=False it builds causal language-modeling labels: a copy of the input ids with padding set to -100 (the model shifts them internally).

Python Stand-in: masked language modeling collator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because masking happens in the collator, every epoch sees a different masking of the same text. That is free data augmentation and it is why you tokenize once and mask per batch rather than masking in advance.

Writing your own collator

Write a custom collator when your batch needs something the built-in ones do not do: tabular side features, image plus text, per-example weights, or truncating to the batch's longest example. The contract is small: accept a list of feature dicts, return a dict of tensors. Keep it fast, since it runs in the data-loading workers on every step.

Python
import torch

def my_collator(features):
    longest = max(len(f["input_ids"]) for f in features)
    ids = torch.full((len(features), longest), tokenizer.pad_token_id)
    mask = torch.zeros((len(features), longest), dtype=torch.long)
    for i, f in enumerate(features):
        n = len(f["input_ids"])
        ids[i, :n] = torch.tensor(f["input_ids"])
        mask[i, :n] = 1
    return {"input_ids": ids, "attention_mask": mask,
            "labels": torch.tensor([f["label"] for f in features])}

Also remember: models accept a limited number of positions (512 for BERT, for example). Always truncate when tokenizing so the collator never has to build a batch longer than the model allows.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does dynamic padding do?

    Padding is postponed to the collator, so short batches stay short and less compute is wasted.
  2. 2. Why are padded label positions set to -100?

    -100 is the ignore_index of the loss, so those positions add nothing to the loss or the gradient.
  3. 3. In BERT-style masking with probability 0.15, what happens to the chosen tokens?

    Mixing replacement kinds stops the model from learning that only [MASK] positions matter.
  4. 4. What does group_by_length=True in TrainingArguments do?

    Similar lengths in a batch mean fewer padded tokens, so training is faster.

Summary

  • A data collator turns a list of examples into one rectangular batch at the last moment.
  • Dynamic padding (tokenize without padding, collate with padding) saves a lot of compute compared with padding to the maximum length; grouping by length saves more.
  • The attention mask tells the model which positions are real; labels for padded positions are -100.
  • Language-modeling collators create labels (masking 80/10/10 for MLM) freshly every batch.
  • A custom collator is just a function from a list of dicts to a dict of tensors.

Next: Custom training loops with Accelerate