Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 38 ยท Fine-tuning with Trainer

Custom Training Loops with Accelerate

Write your own fine-tuning loop when Trainer is not enough: gradient accumulation, clipping, schedulers and Hugging Face Accelerate for multi-GPU, with runnable stand-ins.

The Trainer covers most fine-tuning jobs, but sometimes you need control over every step: a custom loss, two models trained together, a reinforcement-learning style update, or simply a desire to see what happens. In those cases you write the loop yourself in PyTorch. The Accelerate library from Hugging Face then removes the pain of running that same loop on one GPU, several GPUs, a TPU or the CPU.

This lesson walks through the anatomy of a training loop, proves with runnable code why gradient accumulation works, and shows where Accelerate plugs in.

PyTorch, Accelerate and GPUs are not available on this website. Code in plain blocks is real and read-only. The TryIt stand-ins reproduce the mathematics with plain Python so you can run and change them.

The anatomy of one step

Every training step does the same five things:

  1. Take a batch from the data loader.
  2. Forward pass: run the model, get predictions and a loss.
  3. Backward pass: compute the gradient of the loss with respect to every parameter.
  4. Clip and update: optionally limit the gradient size, let the optimizer change the weights, move the learning-rate schedule forward.
  5. Zero the gradients so they do not pile up into the next step.
Python
import torch
from torch.utils.data import DataLoader
from transformers import (AutoModelForSequenceClassification,
                          DataCollatorWithPadding, get_scheduler)

model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2)
loader = DataLoader(tokenized["train"], shuffle=True, batch_size=16,
                    collate_fn=DataCollatorWithPadding(tokenizer))

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
epochs = 3
total_steps = epochs * len(loader)
scheduler = get_scheduler("linear", optimizer=optimizer,
                          num_warmup_steps=int(0.1 * total_steps),
                          num_training_steps=total_steps)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.train()
for epoch in range(epochs):
    for batch in loader:
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)          # forward; loss included because labels are present
        outputs.loss.backward()           # backward
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()                  # update
        scheduler.step()
        optimizer.zero_grad()             # reset

Do not forget model.train(): it enables dropout. Use model.eval() together with torch.no_grad() when you evaluate, or you will waste memory and get noisy predictions.

Gradient accumulation, proven

When the batch you want does not fit in memory, you split it into micro-batches, run forward and backward for each and only call optimizer.step() after N of them. Gradients are summed by PyTorch automatically, so the trick only needs the loss divided by N so the sum equals an average. This stand-in proves, with a one-parameter linear model, that accumulated micro-batches give the same gradient as one big batch.

Python Stand-in: accumulation equals a big batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The difference is zero because every micro-batch has the same size. With uneven micro-batches (for example, token-level losses over different-length sequences) the equality is only approximate, and careful loops normalize by the total number of tokens instead of the number of micro-batches.

In code, accumulation looks like this:

Python
accum = 4
for step, batch in enumerate(loader):
    loss = model(**batch).loss / accum
    loss.backward()
    if (step + 1) % accum == 0:
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

Warmup, decay and gradient clipping

A scheduler changes the learning rate while training. The default for fine-tuning is linear warmup followed by linear decay to zero: a small learning rate at first protects the pretrained weights from large, noisy updates; the decay lets the model settle. Gradient clipping rescales the whole gradient vector if its norm exceeds a limit (1.0 is typical), which prevents one odd batch from wrecking the model. This stand-in implements both.

Python Stand-in: schedule and gradient clipping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that clipping keeps the direction of the gradient and only shrinks its length. A gradient that is already small passes through untouched.

Enter Accelerate

Hand-written loops usually contain device logic: .to(device), DistributedDataParallel, sampler seeds, mixed-precision context managers. Accelerate abstracts all of that. You create an Accelerator, pass your objects to prepare, and replace loss.backward() with accelerator.backward(loss).

Python
from accelerate import Accelerator

accelerator = Accelerator(mixed_precision="bf16", gradient_accumulation_steps=4)
model, optimizer, loader, scheduler = accelerator.prepare(
    model, optimizer, loader, scheduler)

for epoch in range(epochs):
    for batch in loader:
        with accelerator.accumulate(model):
            loss = model(**batch).loss
            accelerator.backward(loss)
            accelerator.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()
            scheduler.step()
            optimizer.zero_grad()

accelerator.wait_for_everyone()
accelerator.save_state("checkpoints/last")

You launch the same script in any setup with accelerate launch train.py, after answering the questions of accelerate config once. Notice what vanished: no .to(device) (the loader and model are placed for you), no manual division of the loss by the number of accumulation steps and no if (step + 1) % accum test.

flowchart TD
  S["train.py"] --> A["accelerate launch"]
  A --> G1["1 GPU"]
  A --> G2["Several GPUs (DDP)"]
  A --> G3["TPU / CPU"]
  A --> G4["DeepSpeed / FSDP"]

One script, many hardware setups.

Trainer or custom loop?

Use Trainer when your task is a standard supervised one. It gives you logging, checkpoints, evaluation, resuming and distributed training for free. Write your own loop when you need unusual control flow: multiple losses, adversarial training, custom sampling, or when you are teaching yourself. You can also customize Trainer by subclassing it and overriding compute_loss, which keeps the rest of its features while changing the objective.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a manual loop, why is the loss divided by the number of accumulation steps?

    Gradients from each micro-batch are added together. Dividing by N turns that sum into the mean.
  2. 2. What does gradient clipping preserve?

    Clipping by norm rescales the whole vector, so it keeps the direction and limits the length.
  3. 3. Which call replaces loss.backward() in an Accelerate loop?

    accelerator.backward handles mixed-precision scaling and distributed details before calling backward.
  4. 4. Why call optimizer.zero_grad()?

    Gradients accumulate by default; without zeroing them each update would include stale gradients.

Summary

  • A training step is: forward, backward, (clip), optimizer step, scheduler step, zero the gradients.
  • Gradient accumulation sums micro-batch gradients after dividing the loss, giving the same update as a big batch.
  • Linear warmup with decay and gradient clipping make fine-tuning stable.
  • Accelerate keeps the loop identical across CPU, GPU, multi-GPU and TPU.
  • Prefer Trainer for standard tasks; subclass it or write a loop for special objectives.

Next: Fine-tuning for text classification