Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 38 ยท Datasets and Preprocessing

Preprocessing with Dataset.map

Tokenize datasets with Dataset.map, use batched processing, remove columns, handle long texts with overflowing chunks and align token labels.

Dataset.map applies a function to every row (or every batch of rows) and returns a new dataset with the changes. It is the workhorse for tokenization, cleaning, feature creation and label alignment. Because it is vectorised over batches, runs in parallel on request and caches its result, it is far faster and safer than a hand-written Python loop.

Tokenizing a dataset

The standard pattern, read-only real code:

Python
from datasets import load_dataset
from transformers import AutoTokenizer

ds = load_dataset("imdb")
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    return tok(batch["text"], truncation=True, max_length=256)

tokenized = ds.map(tokenize, batched=True, remove_columns=["text"])
tokenized = tokenized.rename_column("label", "labels")

Four details matter:

  1. batched=True sends a dict of lists (many rows) to your function, which lets the fast tokenizer process them together. Without it the function receives one row at a time and is many times slower.
  2. The function returns a dict of new or updated columns. They are added to the dataset.
  3. No padding here. Pad later, per batch, with a data collator (next lessons). Padding everything to a fixed length wastes memory and time.
  4. remove_columns drops the raw text so the training loop does not choke on strings.

What map does, step by step

The stand-in below is a plain-Python imitation of map. It supports both row mode and batched mode, so you can compare the shapes your function sees. The tokenizer is a toy word hasher, not a trained one.

Python Stand-in: row mode versus batched mode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In batched mode your function receives lists and must return lists. A batched function may even return more rows than it received, which is how you split long documents into chunks.

Cleaning and feature columns

map is not only for tokenizing. Use it to strip HTML, normalise whitespace, create new columns or compute lengths:

Python
def clean(batch):
    return {
        "text": [t.replace("<br />", " ").strip() for t in batch["text"]],
        "n_words": [len(t.split()) for t in batch["text"]],
    }

ds = ds.map(clean, batched=True)
ds = ds.filter(lambda row: row["n_words"] > 3)

Return the same column name (text) to overwrite it, a new name to add one.

Long texts: overflowing chunks

Texts longer than the model limit would be truncated and lose information. With fast tokenizers you can split them into overlapping windows:

Python
enc = tok(
    batch["text"],
    truncation=True, max_length=256, stride=32,
    return_overflowing_tokens=True, return_offsets_mapping=False,
)
sample_map = enc.pop("overflow_to_sample_mapping")   # which original row each chunk came from

One long review becomes several training rows; overflow_to_sample_mapping tells you the source row so you can copy its label to every chunk. Question answering and long-document classification both use this pattern.

Python Stand-in: split a long token list into overlapping windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Neighbouring windows share stride tokens so that a sentence cut at a boundary appears whole in at least one window.

Aligning labels for token classification

For NER the labels belong to words, not tokens. Use word_ids() (see AutoTokenizer in practice):

Python
def tokenize_and_align(batch):
    enc = tok(batch["tokens"], is_split_into_words=True, truncation=True)
    all_labels = []
    for i, labels in enumerate(batch["ner_tags"]):
        word_ids = enc.word_ids(batch_index=i)
        prev, aligned = None, []
        for wid in word_ids:
            aligned.append(-100 if wid is None or wid == prev else labels[wid])
            prev = wid
        all_labels.append(aligned)
    enc["labels"] = all_labels
    return enc

Performance and caching

  • num_proc=4 runs map in four processes (helpful for CPU-heavy functions; keep it off for tiny ones).
  • Results are cached in a fingerprinted Arrow file, so rerunning is instant if function and data are unchanged. Functions that depend on outside state should be deterministic or you will get stale caches.
  • batch_size controls how many rows each call handles (default 1000).
  • For huge data use streaming so nothing is stored.
  • ds.with_transform(fn) applies a function on the fly when rows are read instead of precomputing, handy for augmentation.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does batched=True change in Dataset.map?

    Batched processing lets fast tokenizers handle many texts per call and is much faster.
  2. 2. Why usually leave padding out of the map function?

    Dynamic padding at batch time wastes far less than padding everything to the maximum.
  3. 3. What does return_overflowing_tokens=True give you?

    Long inputs are split into windows, with overflow_to_sample_mapping pointing to the source row.
  4. 4. Which value marks label positions the loss should ignore?

    PyTorch's cross-entropy ignores -100 by default.

Summary

  • Dataset.map(fn, batched=True) is the fast, cached way to tokenize and transform data.
  • Return a dict of columns; use remove_columns and rename_column to match the model's expected names.
  • Split long texts into overlapping windows and align word labels with word_ids().
  • Leave padding to the collator and use num_proc, caching and with_transform wisely.

Next: Data Collators and Dynamic Padding