Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 37 ยท Inference and Optimization

Weight Quantization

Learn how quantization shrinks LLM weights from 16 bits to 8 or 4: scales, zero points, per-channel and group-wise schemes, outliers, GPTQ and AWQ, with NumPy.

A 70B-parameter model in 16-bit floats occupies 140 GB. That does not fit on one GPU, and even if it did, generating each token means reading all those bytes from memory. Quantization stores weights (and sometimes activations or the KV cache) with fewer bits, such as 8 or 4. The model gets smaller, loads faster, and, because decoding is limited by memory bandwidth (KV cache), runs faster too, with a small loss in quality when done well.

The basic scheme: scale and round

Symmetric integer quantization maps floats to integers with one scale factor:

Text
scale = max(|w|) / qmax          (qmax = 127 for int8, 7 for int4)
q     = round(w / scale)         (clipped to the integer range)
w_hat = q * scale                (dequantized approximation)

Asymmetric quantization adds a zero point so a range that is not centered on 0 uses all the integer levels. Weights are roughly symmetric, so symmetric schemes dominate; activations after some nonlinearities are not, so zero points show up there.

Python INT8 and INT4 quantization error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each bit removed roughly doubles the error. 8-bit is almost free, 4-bit is usually acceptable with care, and 2-bit needs sophisticated methods.

The outlier problem and granularity

One scale for a whole matrix is crude. If a single weight is large, the scale grows and every normal weight is rounded onto only a few levels. LLMs have such outliers (in weights and, more severely, in activations). The fix is finer granularity: use one scale per output channel (row), or one scale per group of, say, 64 or 128 weights.

Python Per-tensor vs per-row vs group-wise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Smaller groups confine the damage from an outlier to its own group. The price is extra storage for the scales: with group size 128 and fp16 scales, you add 16 bits per 128 weights, about 0.125 extra bits per weight, so "4-bit" is really about 4.125 bits.

Does the model output change?

What matters is not weight error but the error in the layer's output y = x @ W. We can measure it with random inputs.

Python Error in layer outputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Errors in individual weights partly average out across the dot product. Across dozens of layers they still accumulate, which is why final quality is checked on benchmarks and perplexity, not on weight error alone.

Post-training methods

Method Idea
Round-to-nearest (RTN) what we did above; simple baseline
GPTQ quantize column by column, adjusting the remaining weights to compensate using second-order (Hessian) information from a small calibration set
AWQ protect the weights multiplied by large activations by scaling them before quantizing
SmoothQuant move activation outliers into the weights so both can be 8-bit
GGUF k-quants block-wise formats for CPU and edge inference (for example 4-bit and 5-bit mixes)
QAT quantization-aware training: simulate quantization during training for the best quality

Calibration uses a few hundred sample texts to estimate which weights and activations matter. Using calibration text unlike your real traffic can hurt quality.

What to quantize

  • Weights only (W4A16): the usual choice for decoding; 4-bit weights, 16-bit activations.
  • Weights and activations (W8A8): enables fast int8 matrix multiplies on supporting GPUs; activation outliers make it harder.
  • KV cache: 8-bit or 4-bit gives large memory savings at long context.
  • Not worth quantizing: often the embedding and output layers, and sensitive layers, which are kept at higher precision.

Rules of thumb

  1. 8-bit is nearly lossless; try it first.
  2. 4-bit with group size 64 to 128 is the usual sweet spot for 7B and larger models. Larger models tolerate it better than small ones.
  3. Always measure on your task. Reasoning, code and non-English text degrade first.
  4. Quantized weights need a kernel that unpacks them fast; otherwise the saved bandwidth is lost.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does one outlier weight hurt per-tensor quantization?

    The scale is max(|w|)/qmax; a big outlier stretches it and coarsens the resolution for everything else.
  2. 2. What does group-wise quantization do?

    Every group (for example 128 weights) gets its own scale, at the cost of extra scale storage.
  3. 3. Why does quantizing weights speed up decoding?

    Each step reads all weights from memory; 4-bit weights move a quarter of the bytes of fp16.
  4. 4. What does QLoRA quantize?

    The base model is stored in 4 bits and the small LoRA adapter is trained at higher precision.

Summary

  • Quantization maps weights to few-bit integers with a scale (and sometimes a zero point).
  • Each bit removed about doubles the error; 8-bit is nearly free and 4-bit is the common target.
  • Finer granularity (per-channel, group-wise) handles outliers; GPTQ and AWQ use calibration data to do better than plain rounding.
  • Judge quantized models by task quality, not weight error.

Next: Serving, Batching and Speculative Decoding