Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 45 ยท Sequences and Transformers

Fine-Tuning, PEFT and LoRA

Learn how to adapt large pretrained models cheaply: full fine-tuning, parameter-efficient methods, a LoRA layer written in PyTorch, adapters and prompt tuning.

Training a large model from scratch takes huge data and compute. Fine-tuning starts from a model someone has already trained and continues training it on your smaller, specific dataset, so it keeps its general knowledge and learns your task. You met the idea for images in Transfer Learning. This lesson goes further: the whole fine-tuning workflow, and the parameter-efficient methods (PEFT) such as LoRA and prompt tuning that make it possible to adapt a model with billions of weights on a single GPU.

PyTorch cannot run in the browser, so PyTorch code is read-only. The live editors use NumPy to run the real LoRA arithmetic: they train a low-rank update and count the parameters.

The fine-tuning workflow

  1. Select a pretrained model that matches your data type and size budget (a vision backbone, a language model, an audio model).
  2. Prepare your data in the format the model expects: the same preprocessing, tokenizer and normalization it was pretrained with.
  3. Adapt the head. Replace the output layer if your classes or vocabulary differ.
  4. Choose what to train: everything, only the head, or a small set of added weights.
  5. Train with a small learning rate. Pretrained weights are already good; large steps destroy them ("catastrophic forgetting"). Typical values are 1e-5 to 5e-5, often lower for the backbone than for a new head (discriminative learning rates).
  6. Evaluate and refine. Watch validation metrics, stop early, and compare against the zero-shot or frozen-backbone baseline.

Full fine-tuning updates every weight. It is the most flexible, but you need memory for the weights, gradients and optimizer state of the whole model (about four times the weight size with Adam, see The Deep Learning Landscape) and you must store a complete copy of the model for every task.

Parameter-efficient fine-tuning

PEFT freezes the pretrained weights and trains only a tiny number of new or selected parameters, often under 1 percent. The benefits are lower memory (no gradients or optimizer state for the frozen part), faster training, small files to share (a few megabytes per task instead of gigabytes), and less forgetting. The main families:

Method What is trained
Adapters Small bottleneck layers inserted inside each block
LoRA Low-rank matrices added in parallel to existing weight matrices
Prompt tuning A few learned "soft prompt" vectors prepended to the input embeddings
Prefix tuning Learned vectors prepended to the attention keys and values in every layer
BitFit Only the bias terms

LoRA: low-rank adaptation

The idea behind LoRA is that the change a fine-tuned model needs is much simpler than the weights themselves. Instead of updating a big matrix W0 (shape d_out x d_in), keep it frozen and learn a low-rank update:

Text
W = W0 + (alpha / r) * B @ A        with A: (r x d_in), B: (d_out x r), r much smaller than d

The rank r (often 4 to 64) controls capacity, and alpha is a scaling factor. B starts at zero, so at the beginning the model behaves exactly like the pretrained one. Parameters drop from d_out * d_in to r * (d_in + d_out): for a 4,096 by 4,096 matrix with r = 8, from 16.8 million to 65,536 (0.4 percent).

A LoRA layer in plain PyTorch (read-only):

Python
import math
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                    # freeze the pretrained weights
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))   # zero: starts as a no-op
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T

Only A and B receive gradients, so the optimizer is built from the trainable parameters:

Python
trainable = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.AdamW(trainable, lr=2e-4)
print(sum(p.numel() for p in trainable), "trainable of", sum(p.numel() for p in model.parameters()))

After training you can merge the update into the base weights, W = W0 + scale * B @ A, so inference has no extra cost, or keep several small LoRA files and switch between tasks on the same base model.

Python Training only a low-rank update

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the true change had rank 4, a rank-4 update recovers it essentially exactly. What happens when the rank is too small?

Python How much rank do you need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ranks below the true complexity of the change leave a residual error; ranks at or above it fit perfectly. Real models often need only a small rank, which is why LoRA works. In practice you pick r by validation score, and apply LoRA to the attention projection matrices first (query and value), widening to the other linear layers if quality is short.

QLoRA

QLoRA keeps the frozen base weights in a 4-bit quantized format and trains LoRA matrices in higher precision on top. It cuts memory enough to fine-tune very large models on one consumer GPU.

Using the peft library

In real projects you rarely write LoRALinear yourself. Hugging Face's peft library wraps a model for you (read-only; the packages are not available on this site):

Python
from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)
config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_lin", "v_lin"], lora_dropout=0.05)
model = get_peft_model(model, config)
model.print_trainable_parameters()      # a fraction of a percent

Training then uses the ordinary training loop; saving stores only the adapter weights (see Saving and Loading Models).

Prompt tuning

Prompt tuning leaves the whole model frozen and learns a handful of vectors, the soft prompt, that are prepended to the input embeddings. Unlike a text prompt, these vectors are not words; they are free parameters shaped like embeddings and trained by gradient descent. A task needs only num_tokens * embedding_dim parameters, for example 20 x 768 = 15,360.

Python
class SoftPrompt(nn.Module):
    def __init__(self, num_tokens, dim):
        super().__init__()
        self.prompt = nn.Parameter(torch.randn(num_tokens, dim) * 0.02)

    def forward(self, token_embeddings):                   # (batch, seq, dim)
        batch = token_embeddings.size(0)
        prefix = self.prompt.unsqueeze(0).expand(batch, -1, -1)
        return torch.cat([prefix, token_embeddings], dim=1)  # (batch, num_tokens + seq, dim)

Prompt tuning gets close to full fine-tuning on very large models, but is less effective on small ones. Prefix tuning is the deeper variant that adds such vectors in every layer.

Choosing a method

Situation Reasonable choice
Small model, plenty of GPU, one task Full fine-tuning
Very similar task, tiny dataset Train only the head (feature extraction)
Large model, limited GPU, many tasks LoRA or QLoRA
Huge frozen model, tiny budget Prompt tuning

Limitations. PEFT can lag full fine-tuning when the new task is far from the pretraining data, hyperparameters such as r and alpha need tuning, and you still need good, clean task data. Always evaluate against a baseline.

Tip: Count trainable parameters before training. If you meant to freeze the backbone and the count is in the millions, something still has requires_grad=True.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does fine-tuning do?

    Fine-tuning starts from pretrained weights and adapts them to a specific task.
  2. 2. In LoRA, why is matrix B initialized to zeros?

    With B = 0 the update B @ A is zero, so the adapted layer initially equals the frozen layer.
  3. 3. A 1,000 x 1,000 weight matrix is adapted with LoRA rank 8. How many LoRA parameters are trained?

    r * (d_in + d_out) = 8 * 2,000 = 16,000, which is 1.6 percent of the 1,000,000 original weights.
  4. 4. What is trained in prompt tuning?

    The base model stays frozen; only the soft prompt vectors are optimized.
  5. 5. Why use a small learning rate when fine-tuning?

    The pretrained weights are already good, so gentle updates preserve what was learned.

Summary

  • Fine-tuning adapts a pretrained model to your task with a small learning rate and the right preprocessing.
  • PEFT trains only a small number of parameters: adapters, LoRA, prompt tuning, prefix tuning, BitFit.
  • LoRA freezes W0 and learns (alpha / r) * B @ A; it starts as a no-op (B = 0), trains well under 1 percent of weights and can be merged for inference.
  • QLoRA adds 4-bit quantization of the frozen weights for even lower memory.
  • Prompt tuning learns soft prompt vectors on a fully frozen model.

Next: see how trained models are stored and shared in Saving and Loading Models.