Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 73 ยท Frameworks and Production

Hugging Face Transformers Pipelines

Learn the Hugging Face transformers workflow: pipelines, tokenizers, AutoModel classes and a fine-tuning loop, with runnable NumPy stand-ins for each step.

Training a transformer from scratch takes enormous data and compute. Almost nobody does it. Instead, people download a pretrained model that someone else has trained and either use it directly or fine-tune it on a small data set of their own. The center of that world is Hugging Face, a platform and a set of Python libraries. The Hub hosts hundreds of thousands of models and datasets, and the transformers library gives all of them one common interface on top of PyTorch (and, for many models, TensorFlow or JAX).

This lesson walks through the workflow in the order you will meet it: the one-line pipeline, the tokenizer that turns text into numbers, the Auto model classes, and a fine-tuning loop. The Hugging Face code is read-only here, because the library and its model downloads cannot run in the browser. Two things you can run: a tokenizer built from scratch, and a fine-tuning loop on a frozen stand-in encoder that has exactly the same shape as the real thing. Be clear about which is which: the stand-ins show the mechanics honestly, but they are not real pretrained models.

The Hugging Face stack

Hub Models, datasets and demos to download.
transformers One API for thousands of models.
datasets Load and map large data sets efficiently.
tokenizers Fast subword tokenization.
1 / 2

Pipelines: the one-line start

A pipeline bundles three steps: preprocessing (tokenizing), the model call and postprocessing (turning raw scores into a readable answer). You name a task and, ideally, a model, and get a callable object back:

Python
# Read-only: needs `pip install transformers torch` and downloads the model on first use
from transformers import pipeline

classify = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")
classify(["I really enjoyed this film", "The plot made no sense"])
# a list with one dict per text, shaped like {'label': 'POSITIVE', 'score': 0.99}

generate = pipeline("text-generation", model="gpt2")
generate("Deep learning is", max_new_tokens=20, do_sample=True, temperature=0.8)

ner = pipeline("ner", aggregation_strategy="simple")          # find people, places, organizations
qa = pipeline("question-answering")                           # answer from a given paragraph
zero_shot = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
zero_shot("The new GPU doubles training speed", candidate_labels=["hardware", "cooking", "sports"])

There are pipelines for text classification, token classification (named entities), question answering, summarization, translation, fill-in-the-blank, text generation, feature extraction (embeddings), and also for images (classification, detection, segmentation), audio (speech recognition) and multimodal tasks. Always pass a model name in real work: the default for a task is convenient for a demo, but it can change between library versions and may not suit your language or domain. Check a model's page on the Hub for what it was trained on, its license and its size before you depend on it.

Under the hood the three stages are exactly what you would write by hand. The program in the fine-tuning section builds a toy version of this, so you can see that a pipeline is not magic.

Tokenizers: text to numbers

A model never sees text. A tokenizer turns a string into a list of integer ids, and the ids index into the model's embedding table (see Embeddings). Transformers use subword tokenization: common words are single tokens, rare words are split into familiar pieces, and anything can be represented, so there are no unknown words. The most common algorithms are byte-pair encoding (BPE), used by the GPT family, and WordPiece, used by BERT. BPE starts from single characters and repeatedly merges the most frequent neighboring pair into a new token, then applies those merges in the same order to new text.

A real call returns more than ids:

Python
# Read-only
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
batch = tok(["the actor was good", "unbelievable story"], padding=True, truncation=True,
            max_length=128, return_tensors="pt")
# batch["input_ids"]       token ids, with [CLS] at the start and [SEP] at the end
# batch["attention_mask"]  1 for real tokens, 0 for padding
tok.convert_ids_to_tokens(batch["input_ids"][1])

Three details cause most bugs. Padding makes all texts in a batch the same length, and the attention mask tells the model to ignore the padding. Truncation cuts long texts to the model's limit (often 512 tokens for BERT-style models), silently discarding the rest. Special tokens such as [CLS] and [SEP] are added for you, and must be the ones the model was trained with, so always load the tokenizer that matches the model. The next example trains a tiny BPE vocabulary from a short corpus and then batches text the same way.

Python A BPE tokenizer with padding and attention masks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The merges the program learns first are the most common letter pairs, like word endings. A word that was frequent in the corpus (unbelievably) became a single token, believable split into believabl plus e</w>, acted into ac, te and d</w>, and a word the corpus never contained (zebra) falls back to single characters, so nothing is ever unknown. In the batch, the shorter second sentence is padded with three zeros and the attention mask marks exactly those three positions with 0. A real vocabulary has 30,000 to 100,000 tokens and is learned from gigabytes of text, but the algorithm is this one.

Auto classes: choosing the model head

A pretrained transformer body produces a vector for each token. A head on top turns it into the output your task needs. The Auto classes load the right architecture and head from a model name:

Python
# Read-only
from transformers import AutoModel, AutoModelForSequenceClassification, AutoModelForCausalLM

body = AutoModel.from_pretrained("bert-base-uncased")                      # vectors only (embeddings)
clf = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
lm = AutoModelForCausalLM.from_pretrained("gpt2")                          # next-token prediction

out = clf(**batch)                      # out.logits has shape (batch_size, 2)
out = clf(**batch, labels=labels)       # adding labels also returns out.loss

When you load bert-base-uncased for classification, the library warns that the classification head is newly initialized. That is expected: the body is pretrained, the small head on top starts random, and fine-tuning teaches it. Other heads exist for token classification, question answering, masked-language modeling and more.

Fine-tuning

Fine-tuning continues training a pretrained model on your own labeled examples. Because the body already understands language, a few thousand examples and a few epochs are often enough, with a small learning rate (for example 2e-5 for the whole model) so that you adjust the pretrained weights instead of destroying them. The usual ingredients are the ones from Momentum, RMSProp and Adam and Learning Rate Schedules: the AdamW optimizer with a short warmup and then a linear decay of the learning rate.

The Trainer class wraps the loop. The datasets library loads and tokenizes data efficiently:

Python
# Read-only
from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DataCollatorWithPadding, Trainer, TrainingArguments)

name = "distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
ds = load_dataset("imdb")                                            # train / test splits of movie reviews
ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=256), batched=True)

model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)
args = TrainingArguments(output_dir="out", num_train_epochs=3, per_device_train_batch_size=16,
                         learning_rate=2e-5, weight_decay=0.01, warmup_ratio=0.1,
                         eval_strategy="epoch", save_strategy="epoch")
trainer = Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["test"],
                  data_collator=DataCollatorWithPadding(tok),        # pads each batch to its own longest text
                  compute_metrics=lambda p: {"accuracy": (p.predictions.argmax(-1) == p.label_ids).mean()})
trainer.train()
trainer.save_model("out/final")

Prefer writing the loop yourself when you need something unusual. It is the loop you already know from PyTorch Essentials:

Python
# Read-only: the same fine-tuning as an explicit PyTorch loop
from torch.optim import AdamW
from transformers import get_linear_schedule_with_warmup

optimizer = AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=100, num_training_steps=total_steps)
for batch in train_loader:
    out = model(**batch)                 # batch has input_ids, attention_mask and labels
    out.loss.backward()
    optimizer.step(); scheduler.step(); optimizer.zero_grad()

The program below runs the same structure end to end on a toy review data set. Its encoder is frozen and random (a table of token vectors averaged over the text), standing in for a pretrained body, so only the classification head is trained. Everything around it is the real recipe: mini-batches, cross-entropy, AdamW with decoupled weight decay, warmup then linear decay, per-epoch evaluation. At the end, a function with the same three stages as a pipeline returns results in the familiar {label, score} shape.

Python A fine-tuning loop and a pipeline stand-in

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before training, the head is random and the accuracy on 200 test reviews is 46.0%, no better than a coin flip. After the first epoch it is 84.5%, and it settles around 90% (89.5% after epoch 6). That curve is what fine-tuning usually looks like: the biggest gain arrives in the first epoch, and later epochs polish. The stand-in encoder cannot read negation or context, so the ceiling is low; a real pretrained model would handle "not great" correctly, and would also update its body, not just the head. The final lines show the pipeline pattern: text in, a list of dictionaries out, with the label chosen from the larger probability and the score being that probability.

Practical notes

  • Start small. Try a distilled model (DistilBERT, a small T5) first. It trains in minutes and tells you whether your data and labels are the problem.
  • Use a validation set, and watch for overfitting: with a large model and little data it appears after only a few epochs (see Overfitting and Regularization).
  • Memory. Large models need a GPU. Mixed precision (fp16 or bf16), smaller batches, gradient accumulation and parameter-efficient fine-tuning (LoRA through the peft library, which trains only small added matrices) cut memory sharply. See Model Compression for quantization.
  • Reproducibility. Pin the model name and a specific revision, and set the seed; hub models can be updated.
  • Safety and licensing. Read the model card. Some models have restricted licenses or known biases (see Ethics, Bias and Safety). Do not load code-bearing model repositories with trust_remote_code=True unless you trust the author, since it runs their Python on your machine.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What are the three stages bundled into a Hugging Face pipeline?

    A pipeline tokenizes the input, runs the model and turns the raw outputs into a readable result such as a label with a score.
  2. 2. What is the attention mask for in a tokenized batch?

    Texts in a batch are padded to a common length. The mask tells attention not to look at the padded positions.
  3. 3. Why is a small learning rate such as 2e-5 typical when fine-tuning a pretrained transformer?

    Pretrained weights are a valuable starting point. A large step can wipe out that knowledge, so fine-tuning uses small steps.
  4. 4. A subword tokenizer meets a word that was never in its training text. What happens?

    BPE and WordPiece vocabularies include small pieces and characters, so any text can be represented without unknown words.
  5. 5. What does the warning about a 'newly initialized' classification head mean?

    Pretraining does not include your labels, so the output layer for your task has to be trained during fine-tuning.

Summary

  • Hugging Face provides the Hub of pretrained models and datasets, and the transformers library with one interface for loading and running them.
  • A pipeline bundles tokenization, the model and postprocessing for common tasks; always specify the model you want.
  • Tokenizers use subword algorithms such as BPE and WordPiece, and produce input_ids and an attention_mask; padding, truncation and special tokens must match the model.
  • Auto classes load a pretrained body with the head your task needs; the new head starts random.
  • Fine-tuning uses a small learning rate, AdamW with warmup and decay, and a validation set, either through Trainer or with the explicit loop you already know.
  • Parameter-efficient methods such as LoRA and mixed precision make fine-tuning large models practical on modest hardware.

Next: Experiments and Reproducibility.