Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท Fine-tuning and Alignment

LoRA and Parameter-Efficient Fine-Tuning

See how LoRA adapts a frozen weight matrix with two small low-rank matrices, count the trainable parameters, train one in NumPy, merge it, and learn about QLoRA.

Full fine-tuning updates every weight and needs optimizer state for all of them. Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a tiny number of extra parameters. The most popular method is LoRA, low-rank adaptation. A LoRA adapter for a 7B model is often only tens of megabytes, and one base model can serve many adapters.

The idea

Take a frozen weight matrix W of shape d_out x d_in. Full fine-tuning would learn an update dW of the same shape. LoRA assumes that useful updates have low rank, so it writes

Text
W' = W + (alpha / r) * B @ A

where A is r x d_in, B is d_out x r, and the rank r is small (4, 8, 16, 64). Only A and B are trained. B starts at zero, so at the beginning W' = W and the model behaves exactly like the base.

flowchart LR
    X["input x"] --> W["frozen W"]
    X --> A["A (r x d_in)"]
    A --> B["B (d_out x r)"]
    W --> S(("+"))
    B --> S
    S --> Y["output"]

The adapter path runs beside the frozen path and the two outputs are added.

Counting parameters

Python How small is a LoRA adapter?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At rank 8 the adapter holds well under 1% of the parameters of the matrices it modifies. Because gradients and optimizer states exist only for those parameters, training memory drops sharply.

Training a LoRA adapter in NumPy

We will mimic the situation at miniature scale. A frozen layer W produces outputs. The "task" needs a slightly different mapping, which differs from W by a low-rank matrix. We train only A and B by gradient descent on squared error.

Python Train A and B, keep W frozen

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls close to zero using 256 trainable numbers instead of 1024, because the true update really is rank 4. If you try r = 1 the loss stalls higher: the adapter does not have enough capacity. This is the central trade-off of LoRA. Real tasks are not exactly low rank, but empirically the needed update is often low-dimensional.

Merging and serving

After training you can fold the adapter into the base weights, so inference costs exactly the same as the original model.

Python Merge the adapter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The difference is floating-point noise. Keeping adapters separate has its own advantage: you can hold one base model in memory and swap adapters per customer or task.

Choices when using LoRA

  • Rank r: 8 to 64 for most tasks. Higher rank costs more and can overfit; start small.
  • alpha: a scale factor. The effective multiplier is alpha / r; many people set alpha = 2r and tune the learning rate instead.
  • Target modules: attention projections only is cheapest. Adding the feed-forward matrices usually helps quality.
  • Learning rate: higher than full tuning, often 1e-4 to 3e-4.
  • Dropout on the adapter path (0.05 to 0.1) regularizes small datasets.

QLoRA

QLoRA combines LoRA with a quantized base model. The frozen weights are stored in 4 bits (see Weight Quantization) and dequantized on the fly, while the adapter stays in 16 bits. This lets a 7B model be tuned on a single consumer GPU. Only the adapter receives gradients, so quantization error in the frozen part is partly compensated by training.

Other PEFT methods

Method What is trained
Adapters (bottleneck) small MLPs inserted after layers
Prefix and prompt tuning learned vectors prepended to the input or keys and values
IA3 per-channel scaling vectors
BitFit only the bias terms
DoRA splits weight into magnitude and direction, applies LoRA to direction

Pitfalls

  • Using a rank that is too small and blaming the data.
  • Forgetting the adapter is tied to its base model version: changing the base invalidates it.
  • Merging an adapter into a quantized base without care, which adds rounding error.
  • Comparing to full tuning at equal learning rate; the best rates differ.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In LoRA, why is B initialized to zero?

    With B = 0 the product B @ A is zero, so W' equals W at step 0 and training starts from the pretrained behavior.
  2. 2. A weight matrix is 4096 x 4096. How many trainable parameters does a rank-8 LoRA adapter add?

    A is 8 x 4096 and B is 4096 x 8, so 2 x 8 x 4096 = 65,536 parameters, instead of about 16.8 million.
  3. 3. What does merging a LoRA adapter achieve?

    W + (alpha/r) B A is a single matrix of the original shape, so inference runs at the original speed.
  4. 4. What is stored in 4 bits in QLoRA?

    The base is quantized and frozen; the small adapter is trained in higher precision.

Summary

  • LoRA freezes W and learns a low-rank update B @ A, scaled by alpha / r.
  • It cuts trainable parameters and optimizer memory by orders of magnitude, and adapters are small to store and swap.
  • Rank limits capacity; merge for zero-overhead inference.
  • QLoRA adds a 4-bit frozen base to fit large models on small GPUs.

Next: RLHF and DPO