Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 38 ยท Efficient Fine-tuning and Quantization

Quantization Fundamentals

Learn how quantization shrinks models: bits and number formats, absmax and zero-point schemes, per-tensor versus per-channel scales, outliers and error, with runnable numpy demos.

A 7-billion-parameter model stored in 16-bit numbers takes about 14 GB. Stored in 8-bit numbers it takes 7 GB, and in 4-bit about 3.5 GB. Quantization is the technique of storing (and sometimes computing) weights with fewer bits, trading a small loss of precision for a large saving in memory and bandwidth. Because large-model inference is mostly limited by how fast weights can be read from memory, smaller weights usually mean faster generation too.

In this lesson you will build the basic quantization schemes yourself, so that the library options in later lessons (load_in_8bit, load_in_4bit, GPTQ, AWQ, GGUF) stop being magic.

Real quantized kernels need a GPU and libraries that are not available on this site. The TryIt stand-ins implement the mathematics with numpy, which is what those libraries do (with fast low-level kernels).

Number formats in a minute

Format Bits What it can represent
float32 32 About 7 decimal digits, huge range
float16 16 About 3 digits, max value 65,504
bfloat16 16 Same range as float32, about 2 digits
int8 8 256 whole numbers, -128 to 127
int4 4 16 whole numbers, -8 to 7

Integers have no exponent, so to store real weights in them you need a scale: store small integers plus one real number that says how big each integer step is. Reconstructing a value is x โ‰ˆ scale * q. This is called dequantization.

Absmax (symmetric) quantization

The simplest scheme maps the largest absolute value in the tensor to the largest integer. For int8, scale = max(|x|) / 127, q = round(x / scale) clipped to [-127, 127]. Zero stays exactly zero, which is convenient.

Python Stand-in: absmax int8 quantization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each time you remove two bits you lose a factor of four in resolution, and the error roughly quadruples. The error is bounded by half a step (scale / 2), so what matters is how large scale is, and scale depends on the biggest value in the tensor.

Asymmetric (zero-point) quantization

Activations such as outputs of ReLU or GELU are not centred on zero: they might range from -0.2 to 6. Symmetric absmax would waste half of the integer range on negative values that never occur. Affine quantization maps the actual minimum and maximum to the full integer range using a scale and a zero-point: q = round(x / scale) + zero_point.

Python Stand-in: symmetric versus zero-point

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For one-sided data the zero-point scheme uses all 256 levels and is about twice as accurate. For weights, which are roughly symmetric around zero, absmax is simple and nearly as good, and avoids extra arithmetic in the matrix multiplication.

One scale or many?

A single scale per whole tensor ("per-tensor") is fragile: one large outlier forces a large scale and crushes all the small weights into a few integer values. Better options are:

  • Per-channel (per-row) scales: each output row has its own scale. Standard for int8 weights.
  • Group-wise / block-wise scales: every 32, 64 or 128 consecutive weights share a scale. This is what 4-bit formats use. The small extra storage (16 bits per block of 64 is 0.25 bits per weight) buys a lot of accuracy.
Python Stand-in: outliers and block-wise scales

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one scale, the outlier's value 4.0 sets the step size so large that almost every ordinary weight rounds to zero and the error is about as big as the weights themselves: the tensor is destroyed. Smaller blocks isolate the damage to one block. This is also why large language models, which have a few huge outlier activation channels, need special handling: LLM.int8() keeps the outlier columns in 16-bit and quantizes the rest.

Post-training versus quantization-aware training

  • Post-training quantization (PTQ) converts a finished model, usually with a small calibration set to choose scales. Methods like GPTQ and AWQ choose the rounding of each weight smartly, using second-order information or activation statistics, to minimize the output error rather than the weight error.
  • Quantization-aware training (QAT) simulates quantization during training so the model learns to tolerate it. It is more accurate but needs a training run.
  • Dynamic quantization quantizes weights ahead of time and activations on the fly; static quantization also fixes activation scales from calibration data.
Python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 8-bit weights, loaded on the fly (real code, GPU required)
model = AutoModelForCausalLM.from_pretrained(
    "my-org/small-llm",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map="auto",
)
print(model.get_memory_footprint() / 1e9, "GB")

Other routes: pre-quantized checkpoints on the Hub (GPTQ, AWQ) that you load with a normal from_pretrained, torchao for PyTorch-native quantization, and GGUF files for CPU runtimes such as llama.cpp.

What changes when you quantize

Smaller 8-bit halves, 4-bit quarters fp16 size.
Often faster Less data moves from memory per token.
Some loss Quality drops slightly; measure it.

Always compare the quantized model to the original on your own task.

1 / 2

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In absmax int8 quantization, what is the scale?

    The largest absolute value is mapped to 127, so one integer step equals max(|x|)/127.
  2. 2. Why can one large outlier ruin per-tensor quantization?

    The step size is set by the largest value, so small weights lose almost all resolution.
  3. 3. What is the purpose of the zero-point in asymmetric quantization?

    It shifts the integer grid so the real minimum and maximum align with the ends of the integer range.
  4. 4. How much smaller is a 4-bit model than the same model in fp16 (ignoring scale overhead)?

    16 bits down to 4 bits is a factor of four.
  5. 5. What distinguishes GPTQ and AWQ from naive rounding?

    They are smart post-training methods that care about output error, not just weight error.

Summary

  • Quantization stores weights with fewer bits plus a scale (and sometimes a zero-point); dequantization is scale * q.
  • Absmax is symmetric; zero-point (affine) suits one-sided data such as activations.
  • Per-channel and block-wise scales protect against outliers; 4-bit formats rely on small blocks.
  • PTQ converts a trained model with calibration data; QAT trains with simulated quantization.
  • Always measure quality after quantizing.

Next: QLoRA and bitsandbytes