Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 38 ยท Efficient Fine-tuning and Quantization

QLoRA and bitsandbytes: 4-bit Fine-tuning

Fine-tune large language models on one GPU with QLoRA: 4-bit NF4 quantization, double quantization, paged optimizers and the BitsAndBytesConfig code, with numpy demos of the ideas.

LoRA made fine-tuning cheaper by training few parameters, but the frozen base model still has to sit in memory. A 7B model in 16-bit is 14 GB; a 70B model is 140 GB. QLoRA attacks that remaining cost: it stores the frozen base model in 4-bit precision and trains LoRA adapters in higher precision on top of it. This made it possible to fine-tune a 65B-parameter model on a single 48 GB GPU, and a 7B model on a consumer card.

QLoRA is not one trick but a bundle of three: the NF4 data type, double quantization and paged optimizers, all provided by the bitsandbytes library and integrated into Transformers and PEFT.

bitsandbytes needs a supported GPU and is not available on this site. The configuration code is real and read-only. The TryIt blocks reproduce the ideas (quantile levels, block-wise absmax, double quantization arithmetic) with numpy; they are not the library's optimized kernels.

How QLoRA trains

During the forward and backward pass, each 4-bit weight block is dequantized to 16-bit on the fly, used in the matrix multiplication, and discarded. The 4-bit weights themselves are never updated. Gradients flow through them into the LoRA matrices A and B, which live in 16-bit (bfloat16) and are the only trainable parameters.

flowchart LR
  Q["frozen base weights<br/>4-bit NF4"] --> D["dequantize<br/>to bf16"]
  D --> M["matmul"]
  X["activations (bf16)"] --> M
  L["LoRA A, B (bf16, trainable)"] --> M
  M --> O["output"]

Storage is 4-bit; arithmetic is 16-bit.

Idea 1: the NF4 data type

Uniform 4-bit codes spread their 16 levels evenly between -max and +max. But trained weights are approximately normally distributed: most are close to zero and few are large. Uniform spacing wastes levels in the sparse tails. NormalFloat 4 (NF4) places the 16 levels at the quantiles of a normal distribution, so each level is used by about the same number of weights, which minimizes the expected error for bell-shaped data.

Python Stand-in: uniform 4-bit versus normal-quantile 4-bit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The real NF4 table is built a little differently (it includes an exact zero and uses asymmetric quantile ranges), but the principle is what you just ran: spend the 16 codes where the weights actually are.

Idea 2: block-wise absmax scales

NF4 is applied per block of 64 weights: divide by the block's largest absolute value, then pick the nearest of the 16 levels. Each block therefore stores 64 four-bit codes plus one scale, a 32-bit float by default. That is 32/64 = 0.5 extra bits per weight, which turns the "4-bit" model into about 4.5 bits per weight.

Idea 3: double quantization

Those scale constants themselves add up: for a 7B model there are about 110 million of them. Double quantization quantizes the scales too, in blocks of 256 scales stored as 8-bit numbers with one 32-bit constant per block. The overhead drops from 0.5 to about 0.127 bits per weight.

Python Stand-in: overhead of the scale constants

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Saving about 0.37 bits per weight is roughly 0.3 GB on a 7B model; for a 65B model it is closer to 3 GB, enough to matter on a single card.

Idea 4: paged optimizers

Gradient checkpointing and long sequences create occasional memory spikes. Paged optimizers (optim="paged_adamw_8bit" or "paged_adamw_32bit") keep optimizer state in unified memory that the driver can page between GPU and CPU RAM automatically, like an operating system swapping pages. The run survives a spike instead of crashing with an out-of-memory error.

The QLoRA recipe in code

Python
import torch
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          BitsAndBytesConfig, TrainingArguments)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

name = "my-org/7b-base-model"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(
    name, quantization_config=bnb, device_map="auto")

model = prepare_model_for_kbit_training(model)     # casts norms to fp32, enables checkpointing
model = get_peft_model(model, LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules="all-linear", task_type="CAUSAL_LM"))
model.print_trainable_parameters()

args = TrainingArguments(
    "out/qlora", per_device_train_batch_size=2, gradient_accumulation_steps=16,
    learning_rate=2e-4, num_train_epochs=1, bf16=True,
    optim="paged_adamw_8bit", gradient_checkpointing=True,
    lr_scheduler_type="cosine", warmup_ratio=0.03, logging_steps=10,
)
trainer = SFTTrainer(model=model, args=args, train_dataset=train_ds,
                     processing_class=tok)
trainer.train()
model.save_pretrained("out/qlora/adapter")

Points to remember:

  • bnb_4bit_compute_dtype is the precision of the arithmetic. Use bfloat16 on GPUs that support it (Ampere and newer), float16 otherwise.
  • prepare_model_for_kbit_training is needed for stable training of quantized models.
  • Parameter names differ between library versions (for example tokenizer= versus processing_class= in trl); check the installed version.
  • The adapter you save is tiny and sits on top of the original base model.

Merging and deploying

You cannot simply merge an adapter into 4-bit weights without loss. The clean approach: reload the base model in 16-bit, apply the adapter, merge_and_unload(), save, and then quantize the merged model again for serving (for example with GPTQ, AWQ or GGUF). Alternatively, serve the 4-bit base with the adapter attached; it is slower than a merged model but needs no extra step.

How much quality do you lose?

On common benchmarks QLoRA fine-tuned models land very close to 16-bit LoRA models, and often within noise. Quality can still fall on tasks that depend on subtle numeric or code behaviour, so measure on your evaluation set before and after. Training is also slower per step than 16-bit LoRA because of the dequantization work, so you trade time for memory.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is stored in 4-bit in QLoRA?

    The base weights are quantized to 4-bit and frozen; the adapters stay in 16-bit and train.
  2. 2. Why is NF4 better than uniform 4-bit codes for neural network weights?

    Weights cluster near zero, so placing more levels there lowers the expected error.
  3. 3. What does double quantization quantize?

    The first-level scales are themselves stored in 8 bits, cutting memory overhead.
  4. 4. What problem do paged optimizers address?

    Optimizer state can be paged between GPU and CPU memory, so spikes do not crash training.
  5. 5. How should you produce a deployable merged model after QLoRA?

    Merging into already-rounded 4-bit weights loses precision; merge in 16-bit first.

Summary

  • QLoRA = frozen 4-bit base model + trainable 16-bit LoRA adapters.
  • NF4 places levels at normal quantiles; block-wise scales and double quantization keep overhead near 4.1 bits per weight.
  • Paged optimizers protect against memory spikes; compute happens in bf16/fp16 after on-the-fly dequantization.
  • Merge adapters into a 16-bit copy of the base, not into 4-bit weights.
  • Always measure quality on your own task after quantization.

Next: Memory math and mixed precision