Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 53 ยท Customizing Models

LoRA, Quantization and Efficient Models

See how LoRA adds small low-rank matrices instead of retraining everything, how 8-bit and 4-bit quantization shrink models, and do the memory math.

Full fine-tuning updates every weight in a model. For a 7-billion-parameter model that means storing the weights, the gradients and the optimizer state, which together can need several times the memory of the model itself. Most people do not have that hardware. Two ideas make customizing and running large models practical: LoRA, which trains only a tiny add-on, and quantization, which stores numbers with fewer bits.

You will build both ideas in NumPy on small matrices, then use the same arithmetic to estimate real memory needs. If you want the basics of why training is expensive, see How LLMs Are Trained and Fine-Tuning vs Prompting.

Two ways to go small

LoRA Train a small add-on, freeze the rest.
Quantization Store each weight in fewer bits.
QLoRA Both together on one GPU.

LoRA saves training memory. Quantization saves memory when running.

1 / 2

Parameter-efficient fine-tuning

Parameter-efficient fine-tuning (PEFT) methods freeze the original model and train a small number of new parameters. The base model stays shared and unchanged, and each task gets a small file (often tens of megabytes) that you load on top. This also limits forgetting, because the original weights never move.

LoRA (low-rank adaptation) is the most popular PEFT method. It rests on an observation: when you fine-tune a large weight matrix W, the change ฮ”W can usually be well approximated by a low-rank matrix, a product of two thin matrices.

Low-rank matrices

A matrix of rank r can be written as B @ A where B is d ร— r and A is r ร— k. A full d ร— k matrix stores dยทk numbers. The two thin factors store only rยท(d + k). If d = k = 4096 and r = 8, that is 16.8 million numbers versus 65,536.

Python A low-rank matrix needs far fewer numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error collapses at rank 4 because the data really has that structure, and the singular values after the fourth are tiny. LoRA bets that fine-tuning updates behave the same way.

How LoRA works

For a frozen weight matrix W (shape d ร— k), LoRA adds a trainable update:

TEXT
h = W x + (alpha / r) * B (A x)
  • A is r ร— k, initialized with small random values.
  • B is d ร— r, initialized to zero, so at the start the model behaves exactly like the base model.
  • r (the rank, such as 4, 8 or 16) controls capacity; alpha scales the update.
  • Only A and B receive gradients. After training you can merge them: W' = W + (alpha / r) * B @ A, so inference has no extra cost.

LoRA is usually applied to the attention projections (and often the feed-forward layers) of every transformer block.

Python Train a LoRA update on a toy layer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only 64 of 256 numbers were trained, yet the layer reproduces the target behaviour. Real layers are far larger, so the saving is much bigger: with r = 8 on a 4096 ร— 4096 matrix, LoRA trains about 0.4% of its parameters.

Quantization

Models store weights as numbers. Quantization replaces high-precision numbers (16-bit floats) with low-precision ones, usually integers, plus a scale that maps them back.

Format Bits per weight Bytes per 1B weights
float32 32 4 GB
float16 / bfloat16 16 2 GB
int8 8 1 GB
int4 4 0.5 GB

A simple scheme, absmax quantization, scales the weights so that the largest absolute value maps to the largest integer (127 for int8, 7 for signed int4):

TEXT
scale = max(|w|) / qmax
q = round(w / scale)        # stored as a small integer
w_hat = q * scale           # used at inference
Python Quantize weights to 8 and 4 bits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error grows quickly as bits shrink, and one outlier forces a coarse scale on all the other weights. Real methods fix this by quantizing in small blocks (for example 32 or 64 weights share a scale), keeping outlier channels in higher precision, or calibrating on sample data. Names you will meet are GPTQ, AWQ, GGUF "Q4" and "Q8" formats, and NF4 (used by QLoRA). QLoRA keeps the frozen base model in 4-bit and trains LoRA matrices in higher precision, which allows tuning large models on a single consumer GPU.

Memory math

For inference, the weights dominate:

TEXT
weight memory = parameters ร— bytes per weight

Add the KV cache and some overhead (often 10 to 20 percent).

Python Estimate memory for model sizes and precisions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 16 bytes-per-parameter figure is a common estimate for full tuning with the Adam optimizer, not an exact law; the real number depends on the framework and settings.

Size versus quality

Bigger models are generally more capable, but the relationship is not simple. In practice: 8-bit usually costs very little quality; 4-bit often costs a small amount that matters more for reasoning, code and non-English text; below 4 bits quality falls fast. A larger model quantized to 4-bit often beats a smaller model at 16-bit with the same memory, but you must test on your own task using an eval set. Smaller models are also faster and cheaper to serve, which links to Running Models Locally.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is LoRA's matrix B initialized to zero?

    With B = 0 the added update B(Ax) is zero, so at step 0 outputs equal the frozen model's outputs.
  2. 2. A 4096 x 4096 matrix gets a LoRA update of rank 8. About how many trainable numbers is that?

    A is 8 x 4096 and B is 4096 x 8, so 8 x (4096 + 4096) = 65,536 numbers, versus 16.8 million in the full matrix.
  3. 3. Roughly how much memory do the weights of a 7B model need in 4-bit?

    Seven billion weights at 0.5 bytes each is about 3.5 GB, before the KV cache and overhead.
  4. 4. Why can a single outlier weight hurt absmax quantization?

    The scale comes from the largest absolute value. An outlier stretches it, which leaves little resolution for ordinary weights.

Summary

  • PEFT methods freeze the base model and train a small add-on; LoRA adds B @ A with a low rank r to selected weight matrices.
  • LoRA can be merged into the weights after training, so inference costs nothing extra.
  • Quantization stores weights as low-bit integers plus scales; block-wise scales and outlier handling keep quality acceptable.
  • Weight memory is parameters times bytes per weight; 4-bit is about a quarter of fp16.
  • Always test the quality trade-off on your own task.

Next: Alignment: RLHF, DPO and Preference Data.