Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 38 ยท Hub, Models and Tokenizers

AutoModel, Configs and Task Heads

Understand AutoConfig, AutoModel and the AutoModelFor classes: body versus head, logits, softmax, label mappings, and how from_pretrained builds a model.

A pretrained transformer has two parts. The body (the base model) turns token ids into one hidden vector per token. A head is a small extra network that turns those vectors into the output you want: class scores, token tags, answer-span positions or next-token probabilities. The Auto classes let you choose body plus head with one name and they figure out the concrete architecture from the repository's config.json.

The Auto classes

Class Returns Typical use
AutoConfig only the settings inspect or change architecture before loading
AutoModel body only: hidden states embeddings, feature extraction
AutoModelForSequenceClassification body + classifier sentiment, topic, spam
AutoModelForTokenClassification body + per-token classifier NER, part-of-speech
AutoModelForQuestionAnswering body + span head extractive QA
AutoModelForCausalLM decoder + language-model head text generation
AutoModelForSeq2SeqLM encoder-decoder translation, summarization
AutoModelForMaskedLM encoder + mask head fill-mask, domain pretraining

Read-only real code:

Python
from transformers import AutoConfig, AutoModel, AutoModelForSequenceClassification

config = AutoConfig.from_pretrained("distilbert-base-uncased")
print(config.hidden_size, config.num_hidden_layers, config.vocab_size)

body = AutoModel.from_pretrained("distilbert-base-uncased")
clf = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=3
)

For the classifier the library loads the pretrained body and creates a new randomly initialized head. You see a warning such as "Some weights ... were not initialized from the model checkpoint ... You should probably TRAIN this model". That is expected: the head is what fine-tuning will learn.

Configuration

config.json holds the architecture: number of layers, hidden size, attention heads, vocabulary size, maximum positions and, for classifiers, the label mappings id2label and label2id. You can override values when loading:

Python
clf = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=3,
    id2label={0: "negative", 1: "neutral", 2: "positive"},
    label2id={"negative": 0, "neutral": 1, "positive": 2},
)

Giving real label names makes pipeline outputs readable after you push the model to the Hub.

You can also create a model with random weights from a config alone (AutoModel.from_config(config)), which is how you pretrain from scratch.

From hidden states to a prediction

A sequence classifier does three small things after the body:

  1. It takes one summary vector for the text (for BERT-like models, the first token's vector, [CLS]; others average or take the last token).
  2. A linear layer maps that vector to one number per class: the logits.
  3. Softmax converts logits to probabilities that sum to 1.

Logits are raw scores and can be any real number. The class with the largest logit also has the largest probability, so argmax works on either.

The code below is a stand-in: the body is replaced by made-up vectors and the head is a hand-written linear layer in NumPy. It reproduces steps 2 and 3 exactly as a real head would do them.

Python Stand-in: a classification head

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The head is tiny compared with the body: here 21 numbers. In a real DistilBERT classifier the body has about 66 million parameters and the head only a few thousand.

Model outputs and the loss

Calling a model returns an output object. With labels supplied, it also computes the loss (cross-entropy for classification) which the trainer minimises:

Python
import torch

inputs = tokenizer(["great", "awful"], padding=True, return_tensors="pt")
labels = torch.tensor([2, 0])
out = clf(**inputs, labels=labels)
print(out.loss, out.logits.shape)       # logits shape: (2, 3)

For inference use model.eval() and torch.no_grad() so layers like dropout are off and no gradient memory is used.

Python Stand-in: cross-entropy loss by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the model gives a high probability to the true class, the loss is small; wrong confident predictions are punished heavily. This is the number that fine-tuning drives down.

Precision, devices and saving

  • torch_dtype=torch.float16 or "auto" loads weights in lower precision to save memory.
  • device_map="auto" spreads a large model over available devices (needs accelerate).
  • model.to("cuda") moves a normal model to a device; inputs must go to the same device (inputs.to(model.device)).
  • model.save_pretrained("folder") writes config.json and weights; from_pretrained("folder") reads them back.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does AutoModelForSequenceClassification.from_pretrained(name, num_labels=3) do with the classifier head?

    The body comes from the checkpoint, the head is new and must be trained.
  2. 2. What are logits?

    Softmax turns logits into probabilities.
  3. 3. Which class returns hidden states only, with no task head?

    AutoModel gives the body, useful for embeddings and feature extraction.
  4. 4. Why call model.eval() and use torch.no_grad() for inference?

    Evaluation mode makes outputs deterministic and no_grad saves memory.

Summary

  • Models are a pretrained body plus a task head; AutoModelFor... picks the right head class.
  • config.json describes the architecture and label mappings; you can override it at load time.
  • A classification head maps a summary vector to logits, softmax gives probabilities, cross-entropy gives the loss.
  • Use eval(), no_grad(), matching devices and save_pretrained for clean inference and saving.

Next: Pipelines for Text Tasks