Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 38 · Efficient Fine-tuning and Quantization

LoRA Explained: Low-Rank Adaptation

Understand LoRA step by step: the low-rank update W + BA, rank and alpha, target modules, zero initialization and merging, with numpy demos and the peft code you would use.

LoRA stands for Low-Rank Adaptation. It is the most widely used parameter-efficient method for large models, and its core idea fits in one sentence: instead of changing a big weight matrix W directly, learn a small correction ΔW and express that correction as the product of two thin matrices, ΔW = B · A. Only A and B are trained; W stays frozen.

The bet behind LoRA is that the change needed to adapt a pretrained model to a new task has low intrinsic rank: it can be described with far fewer numbers than the full matrix contains. Experiments on language models showed this bet pays off remarkably often.

PyTorch and the peft library do not run on this site. The TryIt blocks use numpy to demonstrate the mathematics, which is the same math the library applies to real weights. The peft code is shown read-only.

The mathematics

Let W be a matrix of shape d_out x d_in, say 4096 x 4096 (16.8 million numbers). LoRA chooses a small rank r (for example 8) and creates two matrices: A with shape r x d_in and B with shape d_out x r. The adapted layer computes

h = W x + (alpha / r) · B (A x)

Count the trainable numbers: r · d_in + d_out · r = 8 · 4096 + 4096 · 8 = 65,536, which is 0.4% of the original matrix. The scaling factor alpha / r controls how strongly the update is applied; alpha is a hyper-parameter you set (often r, 2r).

Initialization matters. A starts with small random values and B starts at zero. Therefore B · A = 0 at step zero, the adapted model behaves exactly like the pretrained model, and training begins from a known good point.

flowchart LR
  X["input x"] --> W["frozen W<br/>d_out x d_in"]
  X --> A["A<br/>r x d_in (trainable)"] --> B["B<br/>d_out x r (trainable)"]
  W --> P(("+"))
  B -- "x alpha/r" --> P
  P --> Y["output h"]

The frozen path and the low-rank path run side by side; their outputs are added.

LoRA in numpy

First, the shapes and the zero initialization, then the equivalence of "side path" and "merged weight".

Python Stand-in: LoRA shapes, zero init and merging

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The merge in step 3 is why LoRA adds no inference latency once training is done: you add (alpha/r)·B·A into W and serve a normal model. You can also keep adapters separate and swap them, which is useful when one base model serves several customers or tasks.

Learning a low-rank change

Does a rank-2 correction really have enough capacity? This stand-in builds a "task" whose true weight differs from the pretrained weight by a rank-2 matrix, then trains A and B with plain gradient descent to find it, keeping W frozen.

Python Stand-in: training only A and B

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls close to zero with a fraction of the numbers, because the needed change was low rank. In real models the change is not exactly low rank, so a higher r buys a little more quality; the benefit usually saturates quickly.

The hyper-parameters

  • r (rank). Capacity of the update. Common values are 4, 8, 16 and 64. Start at 8 or 16; raise it only if training loss stays high.
  • lora_alpha. Scale. The effective multiplier is alpha / r. Keeping alpha = 2r is a frequent default. If you double r but keep alpha, the update's strength halves.
  • target_modules. Which linear layers get adapters. The original paper adapted the attention query and value projections. Modern practice targets all linear layers (q_proj, k_proj, v_proj, o_proj and the MLP projections) for better quality at a higher parameter count. Names depend on the architecture; target_modules="all-linear" selects them automatically.
  • lora_dropout. Dropout applied on the adapter input, 0.05 to 0.1 for regularization.
  • bias. Whether to train biases too ("none", "all", "lora_only").
  • use_rslora, use_dora. Variants: rank-stabilized scaling (alpha / sqrt(r)) and weight-decomposed LoRA, which can help at some ranks.
  • modules_to_save. Layers fully trained and saved with the adapter, such as the new classification head in a sequence-classification task.

The learning rate for LoRA is usually higher than for full fine-tuning (for example 1e-4 to 3e-4), because only a few parameters move.

The peft code

Python
from peft import LoraConfig, get_peft_model, PeftModel, TaskType
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          Trainer, TrainingArguments)

base = AutoModelForCausalLM.from_pretrained("my-org/small-llm")
tok = AutoTokenizer.from_pretrained("my-org/small-llm")

cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules="all-linear",
)
model = get_peft_model(base, cfg)
model.print_trainable_parameters()

args = TrainingArguments("out/lora", learning_rate=2e-4, num_train_epochs=2,
                         per_device_train_batch_size=4,
                         gradient_accumulation_steps=8, bf16=True,
                         logging_steps=20, save_strategy="epoch")
trainer = Trainer(model=model, args=args, train_dataset=train_ds,
                  data_collator=collator)
trainer.train()
model.save_pretrained("out/lora/adapter")        # a few megabytes

# Later: serve it
base = AutoModelForCausalLM.from_pretrained("my-org/small-llm")
model = PeftModel.from_pretrained(base, "out/lora/adapter")
merged = model.merge_and_unload()                 # plain model, no adapter overhead
merged.save_pretrained("out/merged")

Merge only into an unquantized base (for example a 16-bit model). Merging into a 4-bit base is lossy and needs care, which is covered in the QLoRA lesson.

Common mistakes

  • Wrong target_modules names. The library raises an error if no module matches; print model to see actual names.
  • Forgetting modules_to_save for a new head. Without it the randomly initialized classifier stays frozen at random values.
  • Comparing to full fine-tuning with the same learning rate. LoRA wants a larger one.
  • Merging before evaluating. Evaluate the adapter first; the merged model should match it within rounding error.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. In LoRA, which matrices are trained?

    W stays frozen. The update B·A is learned and added to the layer's output.
  2. 2. Why is B initialized to zero?

    With B = 0 the update B·A is zero, so training starts from the pretrained behaviour.
  3. 3. With r=8 on a 4096 x 4096 matrix, how many trainable numbers does the adapter add?

    r · d_in + d_out · r = 8 · 4096 + 4096 · 8 = 65,536.
  4. 4. What does merging a LoRA adapter achieve?

    After merging, the weight already contains the update, so no separate adapter path is needed.
  5. 5. Compared with full fine-tuning, LoRA typically uses

    Only a few parameters change, so they can take bigger steps.

Summary

  • LoRA learns ΔW = B·A with a small rank r and keeps W frozen.
  • A starts random and B starts at zero, so training begins from the pretrained model.
  • r, lora_alpha (scale alpha/r), target_modules and lora_dropout are the main knobs; use a larger learning rate than for full fine-tuning.
  • Adapters are tiny, swappable files, and can be merged into the base for zero-latency serving.
  • Use modules_to_save for new heads and check module names before training.

Next: Quantization fundamentals