Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 38 ยท Fine-tuning with Trainer

Checkpoints, Callbacks and Early Stopping

Save, resume and manage training checkpoints, use Trainer callbacks and EarlyStoppingCallback, keep the best model and avoid overfitting, with runnable stand-ins.

Training can take hours or days. Machines crash, jobs get pre-empted and you often want the model from the best moment of a run, not the last one. Checkpoints solve the first problem, callbacks let you react to events during training, and early stopping ends a run when it stops improving. Together they make long runs safe and economical.

The real Trainer needs PyTorch and is not available on this site. Plain code blocks are real and read-only; each TryIt is a labelled stand-in in plain Python that reproduces the logic.

What is inside a checkpoint

With save_strategy="steps" (and save_steps=500) or "epoch", the Trainer writes a folder output_dir/checkpoint-<step>/. It contains the model weights (model.safetensors), the config, and everything needed to resume: optimizer.pt (Adam keeps two extra numbers per weight, so this file is large), scheduler.pt, rng_state.pth, and trainer_state.json with the step counter, the logged losses and the name of the best checkpoint so far.

Python
args = TrainingArguments(
    output_dir="out/run1",
    save_strategy="steps", save_steps=500,
    eval_strategy="steps", eval_steps=500,
    save_total_limit=2,              # keep only the newest 2 (and the best)
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)
trainer = Trainer(...)
trainer.train()                              # fresh run
# after a crash:
trainer.train(resume_from_checkpoint=True)   # picks the latest checkpoint in output_dir

save_total_limit deletes old checkpoints so a 100-checkpoint run does not fill the disk. With load_best_model_at_end=True the Trainer reloads the best weights when training ends; for this to work, save_strategy and eval_strategy must use the same schedule. metric_for_best_model says which logged number decides "best"; greater_is_better tells it whether bigger is better (accuracy) or smaller (loss).

Simulating checkpoint rotation

The bookkeeping is easy to get wrong, so run it. This stand-in keeps the newest save_total_limit checkpoints and always protects the best one.

Python Stand-in: save_total_limit with a best checkpoint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at step 2000: loss 0.45 is the best, so checkpoint-2000 is never deleted even when newer checkpoints arrive. That is why you may see three folders when the limit is two.

Callbacks: reacting to training events

A callback is an object with methods that the Trainer calls at fixed moments: on_train_begin, on_step_end, on_evaluate, on_save, on_log, on_train_end and others. Built-in callbacks print progress, send metrics to TensorBoard or Weights & Biases (report_to), and stop early. Your own callback subclasses TrainerCallback:

Python
from transformers import TrainerCallback

class LossSpikeGuard(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if logs and logs.get("loss", 0) > 20:
            print("loss exploded at step", state.global_step)
            control.should_training_stop = True
        return control

trainer = Trainer(..., callbacks=[LossSpikeGuard()])

The control object is how a callback talks back: set should_training_stop, should_save, should_evaluate or should_log. The state object tells you where you are (global_step, epoch, best_metric).

Early stopping

Overfitting shows up as evaluation loss that stops falling (or starts rising) while training loss keeps dropping. EarlyStoppingCallback watches the metric you named in metric_for_best_model and stops the run when it has not improved for early_stopping_patience consecutive evaluations.

Python
from transformers import EarlyStoppingCallback

args = TrainingArguments(
    output_dir="out/early", num_train_epochs=20,
    eval_strategy="epoch", save_strategy="epoch",
    load_best_model_at_end=True, metric_for_best_model="eval_loss",
    greater_is_better=False,
)
trainer = Trainer(..., callbacks=[EarlyStoppingCallback(early_stopping_patience=3)])

Set a generous num_train_epochs; early stopping will cut the run short. Early stopping needs evaluation and saving on the same schedule, load_best_model_at_end=True, and a metric_for_best_model. Forgetting any of them raises an error at start.

Early stopping you can run

This stand-in implements the patience logic with a minimum improvement threshold (early_stopping_threshold in the real callback).

Python Stand-in: EarlyStoppingCallback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The best epoch was 4; the run continued for three more evaluations before stopping. That is the price of patience: more patience tolerates noisy metrics, less patience saves compute.

A safe long run

Save often Checkpoint every few hundred steps or each epoch.
Rotate save_total_limit keeps disk use bounded.
Keep the best load_best_model_at_end restores the winner.
Stop early Patience ends runs that stopped improving.

Add resume_from_checkpoint for crash recovery.

1 / 2

Resuming correctly

resume_from_checkpoint restores model, optimizer, scheduler and the data position, so the learning-rate curve continues as if nothing happened. Loading only the weights with from_pretrained("out/run1/checkpoint-500") and starting a new run restarts the optimizer and the schedule: that is warm-starting, not resuming. Both are legitimate, but they are not the same. Use the same TrainingArguments and the same seed when resuming, or the results will drift.

Logging and reading history

trainer.state.log_history is a list of dicts such as {"loss": 0.41, "step": 100} and {"eval_loss": 0.38, "epoch": 1.0}. Training-loss entries and evaluation entries are separate rows, so filter by key before plotting. Set report_to="tensorboard" (or "wandb") to visualize curves without extra code, and report_to="none" to disable external reporting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does save_total_limit=2 do?

    Older checkpoints are deleted to bound disk usage; the best one is protected so it can be restored.
  2. 2. Which settings must be present for EarlyStoppingCallback to work?

    The callback compares the chosen metric across evaluations and the best model needs a saved checkpoint to restore.
  3. 3. A callback wants to end training. How does it tell the Trainer?

    The control object carries flags the Trainer reads after each callback.
  4. 4. What is the difference between resuming and warm-starting?

    Resuming continues the same run exactly; warm-starting begins a new run from old weights.

Summary

  • A checkpoint holds weights plus optimizer, scheduler, RNG and trainer state so a run can resume exactly.
  • save_total_limit bounds disk use; load_best_model_at_end with metric_for_best_model restores the best moment.
  • Callbacks react to events and steer the Trainer through the control object.
  • EarlyStoppingCallback stops after early_stopping_patience evaluations without improvement.
  • Resume with resume_from_checkpoint; loading weights only is a warm start.

Next: Parameter-efficient fine-tuning: the big picture