Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 38 ยท Fine-tuning with Trainer

The Trainer and TrainingArguments

Learn how the Hugging Face Trainer runs the training loop for you: TrainingArguments, steps and epochs, evaluation, logging, learning-rate schedules, with runnable stand-ins.

Fine-tuning means taking a model that was already trained on a huge amount of text and training it a little more on your own task. Writing that loop by hand is easy to get subtly wrong: batching, shuffling, moving tensors to the GPU, gradient accumulation, learning-rate schedules, evaluation, logging, checkpoints. The Trainer class in the Transformers library wraps all of it so that you describe what you want and it handles how.

The real Transformers library, PyTorch and a GPU are not available inside this website. The code in plain code blocks below is real and read-only. Each TryIt is a clearly labelled stand-in written in plain Python that you can run: it re-creates the ideas (steps, schedules, evaluation) so you understand what the real class is doing. Numbers printed by stand-ins are not benchmarks of any real model.

The two objects you always create

A Trainer needs a model, a configuration object called TrainingArguments, and datasets. This is the smallest complete example:

Python
from transformers import (AutoModelForSequenceClassification,
                          Trainer, TrainingArguments)

model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2
)

args = TrainingArguments(
    output_dir="out/sentiment",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    logging_steps=50,
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    compute_metrics=compute_metrics,
)
trainer.train()
print(trainer.evaluate())

Notice that the dataset, the model and the metric function are passed in, while every knob about the training procedure lives in TrainingArguments. That split is the design: arguments are plain, serializable settings, so you can log them, share them and reproduce a run.

The evaluation argument was called evaluation_strategy in older releases and is eval_strategy in current ones. If you copy code from an old tutorial and get an "unexpected keyword" error, check the argument name first.

What the important arguments mean

Argument What it controls
num_train_epochs How many full passes over the training set.
per_device_train_batch_size Examples processed together on one device.
gradient_accumulation_steps Run several small batches before one optimizer update, to fake a bigger batch.
learning_rate Peak step size. Fine-tuning usually uses small values such as 2e-5 to 5e-5.
lr_scheduler_type Shape of the learning rate over time (linear, cosine, constant...).
warmup_ratio / warmup_steps How long the learning rate ramps up from zero.
weight_decay Mild regularization that shrinks weights.
eval_strategy, save_strategy When to evaluate and write checkpoints: "no", "steps" or "epoch".
fp16 / bf16 Mixed precision, covered in memory math and mixed precision.
seed Seeds shuffling and initialization of new layers.

The effective batch size is per_device_train_batch_size x gradient_accumulation_steps x number_of_devices. Learning rates are tuned for an effective batch size, so when you change one of those numbers, think about the others.

Counting steps

A common surprise is how many optimizer updates a run really performs. A step is one optimizer update, not one batch. This stand-in computes the numbers the Trainer would report before it starts.

Python Stand-in: how many steps will training take?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two runs have the same effective batch size of 16, so they perform the same number of updates, even though the second only keeps 4 examples in memory at once. The third uses four devices and therefore does a quarter as many (but larger) steps. This is why "I changed to four GPUs and my results got worse" is often a learning-rate question, not a bug.

A tiny Trainer you can run

The best way to understand Trainer is to write a miniature one. This stand-in trains a logistic-regression classifier with the same shape of API: an arguments object, a train loop with shuffling and batching, a linear warmup-and-decay schedule, periodic logging, and a compute_metrics callback at the end of each epoch.

Python Stand-in: a miniature Trainer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Match each part to the real class. Args is TrainingArguments. The lr_at method is the scheduler. The compute_metrics function receives predictions and labels, exactly as in the real Trainer (where it receives an EvalPrediction with predictions as logits and label_ids). The real Trainer also handles devices, mixed precision, distributed training, resuming and callbacks, but the heart of it is this loop.

Metrics and the logged output

During training the Trainer prints a table of loss every logging_steps and, at each evaluation, eval_loss plus whatever your compute_metrics returned (prefixed with eval_). trainer.state.log_history holds the same records as a list of dictionaries, which is perfect for plotting a learning curve afterwards. Watch two curves: training loss should decrease steadily; evaluation loss should decrease and then flatten. When evaluation loss starts to rise while training loss keeps falling, the model is overfitting and you should stop or regularize (see checkpoints, callbacks and early stopping).

flowchart LR
  A["TrainingArguments"] --> T["Trainer"]
  M["Model"] --> T
  D["Datasets + collator"] --> T
  C["compute_metrics"] --> T
  T --> L["Training loop<br/>forward, loss, backward, update"]
  L --> E["Evaluate + log + save"]
  E --> L

The Trainer receives four ingredients and owns the loop.

Common mistakes

  • Forgetting labels. The model only returns a loss when the batch contains labels. If your dataset column is called label, the default collator renames it, but custom columns must match the model's expected names.
  • Leftover text columns. Columns such as text cannot be turned into tensors. The Trainer removes columns the model's forward does not accept (remove_unused_columns=True), which can silently drop a column you wanted; set it to False only when you know why.
  • No evaluation set. Without eval_dataset, eval_strategy cannot do anything and you will not see overfitting.
  • Changing batch size without the learning rate. Remember the effective batch size.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two runs use batch 16 with no accumulation, and batch 4 with gradient_accumulation_steps=4. Which statement is true?

    4 x 4 = 16, so both runs update the weights after seeing 16 examples. Accumulation only lowers peak memory.
  2. 2. Where does the learning-rate schedule (for example linear decay with warmup) get configured?

    Schedule shape and warmup are plain training settings and live in TrainingArguments.
  3. 3. What does compute_metrics receive in the real Trainer?

    You turn logits into predicted classes yourself (for example with argmax) and return a dictionary of named metrics.
  4. 4. Training loss keeps falling but evaluation loss rises after epoch 2. This most likely means

    The model is memorizing training examples instead of generalizing. Stop earlier, regularize, or add data.

Summary

  • Trainer runs the loop; TrainingArguments holds every setting about the procedure; the model, datasets and compute_metrics are passed in separately.
  • Effective batch size = per-device batch x accumulation x devices; a step is one optimizer update.
  • A warmup-then-decay schedule is the usual default for fine-tuning with small learning rates.
  • Watch training versus evaluation loss to spot overfitting.
  • The stand-ins here show the logic; the real class adds devices, mixed precision, resuming and callbacks.

Next: Data collators and dynamic padding