Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 38 ยท Fine-tuning with Trainer

Fine-tuning for Text Classification

Fine-tune a pretrained transformer for text classification: classification head, label mappings, logits and softmax, freezing layers and a complete Trainer script, with stand-ins.

Text classification is the most common first fine-tuning task: given a piece of text, pick one label. Sentiment (positive or negative), topic, spam detection, intent detection and toxicity filtering all have this shape. The recipe is always the same: load a pretrained encoder, put a small classification head on top, tokenize your examples, and train the whole thing with a small learning rate for a few epochs.

This lesson explains what the head is, how logits become probabilities, how to map label names to numbers, when to freeze the body, and shows a complete script you can adapt.

The real library and a GPU are not available on this site, so the full script is read-only. The TryIt blocks are stand-ins: they use numpy and fixed random "embeddings" instead of a real pretrained model, to let you run the head, the loss and the freezing idea yourself.

What AutoModelForSequenceClassification builds

Loading AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=3) gives you the pretrained body (embeddings plus transformer layers) and a freshly initialized head: usually one or two linear layers applied to a pooled representation of the text (the [CLS] token's final vector for BERT-like models). You will see a warning that some weights were "newly initialized"; that is the head, and it is expected: the pretrained model never saw your labels.

flowchart LR
  T["text"] --> K["tokenizer"] --> B["pretrained body<br/>transformer layers"]
  B --> P["pooled vector<br/>(768 numbers)"]
  P --> H["classification head<br/>linear: 768 to num_labels"]
  H --> L["logits"] --> S["softmax"] --> R["probabilities"]

Only the head starts from random weights. The body already understands language.

Logits, softmax and cross-entropy

The head outputs one raw score per label, called a logit. Logits can be any real number. Softmax turns them into probabilities that sum to one. The training loss is cross-entropy: the negative log of the probability the model gave to the correct label. A confident correct answer gives a loss near 0; a confident wrong one gives a large loss.

Python Stand-in: logits to probabilities to loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Subtracting the maximum logit before exponentiating is the standard trick for numerical stability; the result is mathematically identical. Predicting a class is argmax over the logits; you do not need softmax for that, only when you want calibrated-looking probabilities.

Label names: id2label and label2id

Models work with integers. Store the mapping in the model's config so that saved models, the Hub widget and pipelines show readable names instead of LABEL_0.

Python
id2label = {0: "negative", 1: "neutral", 2: "positive"}
label2id = {v: k for k, v in id2label.items()}

model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=3,
    id2label=id2label,
    label2id=label2id,
)

If your dataset has a ClassLabel feature, dataset.features["label"].names already contains the names in id order. Also decide what kind of problem you have: problem_type="single_label_classification" (one correct label, cross-entropy), "multi_label_classification" (several can be true, a sigmoid per label and binary cross-entropy, labels given as floats) or "regression" (one number, mean squared error). Transformers infers this from num_labels and label dtype, but stating it avoids surprises.

A runnable head on frozen features

The simplest honest way to see "fine-tuning a head" without a GPU is to keep the body fixed and train only the head. Here the "body" is a fixed table of random word vectors with mean pooling, and the head is a softmax layer trained with gradient descent. A real body would of course give much better features.

Python Stand-in: training only the classification head

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In real fine-tuning you usually also update the body ("full fine-tuning"), with a very small learning rate so the pretrained knowledge is nudged rather than destroyed. Training the head only is faster and cheaper, and sometimes enough when you have very little data, but it generally performs worse than updating the body.

Freezing layers

You can freeze any parameters by switching off their gradients:

Python
for param in model.distilbert.parameters():   # the body
    param.requires_grad = False
# only model.pre_classifier and model.classifier are trained now
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(trainable)

Freezing is the idea behind parameter-efficient methods such as LoRA, which you will meet in LoRA explained. A popular middle path is gradual unfreezing: train the head first, then unlock the top layers, then the rest.

The complete script

Python
import numpy as np
from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DataCollatorWithPadding, Trainer, TrainingArguments)

ds = load_dataset("imdb")
ckpt = "distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(ckpt)

def prep(batch):
    return tok(batch["text"], truncation=True, max_length=256)

ds = ds.map(prep, batched=True)

model = AutoModelForSequenceClassification.from_pretrained(
    ckpt, num_labels=2,
    id2label={0: "NEGATIVE", 1: "POSITIVE"},
    label2id={"NEGATIVE": 0, "POSITIVE": 1})

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {"accuracy": float((preds == labels).mean())}

args = TrainingArguments("out/imdb", learning_rate=2e-5, num_train_epochs=2,
                         per_device_train_batch_size=16, eval_strategy="epoch",
                         save_strategy="epoch", load_best_model_at_end=True,
                         group_by_length=True)

trainer = Trainer(model=model, args=args,
                  train_dataset=ds["train"], eval_dataset=ds["test"],
                  data_collator=DataCollatorWithPadding(tok),
                  compute_metrics=compute_metrics)
trainer.train()
trainer.save_model("out/imdb/final")

Things that go wrong

  • Label mismatch. Labels must be integers in 0..num_labels-1. A label of 3 with num_labels=3 crashes with an index error deep inside the loss.
  • Too long a learning rate. 1e-3 that suits a small network destroys a pretrained body; stay near 2e-5.
  • Truncating the wrong end. For long documents the first 256 tokens may not contain the answer; consider head+tail truncation or chunking.
  • Imbalanced classes. Accuracy hides a model that always predicts the majority class. Look at precision and recall per class: see classification metrics.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which part of AutoModelForSequenceClassification starts from random weights?

    The body is pretrained. Only the head, which maps to your labels, is newly initialized.
  2. 2. What is a logit?

    Logits are raw real-valued scores; softmax converts them into probabilities.
  3. 3. Why set id2label and label2id?

    The mapping is stored in config.json and used by pipelines and the Hub widget.
  4. 4. A typical learning rate for fully fine-tuning a BERT-like model is

    Small steps adjust the pretrained weights without wiping out what they already know.
  5. 5. Which problem_type fits tagging an article with several topics at once?

    Several labels can be true together, so each label gets its own sigmoid and binary cross-entropy.

Summary

  • Fine-tuning for classification = pretrained body + new head, trained end to end with a small learning rate.
  • The head produces logits; softmax gives probabilities; cross-entropy is the loss; argmax is the prediction.
  • Store id2label and label2id in the config; choose the correct problem_type.
  • Freezing the body trains only the head: cheap, but usually weaker than full fine-tuning.
  • Check class balance and per-class metrics, not just accuracy.

Next: Checkpoints, callbacks and early stopping