Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 38 ยท Efficient Fine-tuning and Quantization

Memory Math and Mixed Precision Training

Estimate GPU memory for training and inference, understand fp16 versus bf16, loss scaling, gradient checkpointing and batch size trade-offs, with runnable calculators.

"CUDA out of memory" is the most common error in fine-tuning. It becomes much less mysterious once you know where memory goes. Four things fill a GPU during training: the weights, the gradients, the optimizer state and the activations stored for the backward pass. The first three can be calculated from the parameter count; the fourth depends on batch size and sequence length.

This lesson gives you the formulas, demonstrates why fp16 needs loss scaling and bf16 does not, and lists the standard levers for fitting a run into memory.

Real GPUs are not available here. The calculators are plain Python and numpy; the numbers they print are estimates that real frameworks will exceed by some overhead (CUDA context, fragmentation, temporary buffers).

The memory budget

For a model with N parameters:

  • Weights: N x bytes_per_param (4 for fp32, 2 for fp16/bf16, about 0.5 for 4-bit).
  • Gradients: the same size as the trainable weights, in the training precision.
  • Optimizer state: Adam keeps two moments per trainable weight, plus often an fp32 master copy of the weights when training in mixed precision. A common total is 12 bytes per trainable parameter (4 master + 4 + 4 moments).
  • Activations: intermediate results kept for backward. Grows with batch size, sequence length and layers.

A handy rule: mixed-precision full fine-tuning with Adam costs about 16 bytes per parameter (2 weights + 2 gradients + 12 optimizer), so a 7B model needs about 112 GB before activations. Inference in 16-bit needs just 2 bytes per parameter.

Python Calculator: memory by method

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table makes the strategy obvious: for a 70B model only QLoRA (about 40 GB) is within reach of a single large GPU; full fine-tuning needs a cluster.

Activations and batch size

Activation memory is the part people forget. It scales roughly linearly with batch size and sequence length, and with the number of layers. Doubling the sequence length from 512 to 1024 doubles it (and standard attention's score matrices grow with the square of length unless a memory-efficient attention kernel such as FlashAttention is used). That is why the first fix for an out-of-memory error is usually a smaller per-device batch plus gradient accumulation, which keeps the effective batch size.

fp16, bf16 and why they differ

Both are 16-bit, but they spend the bits differently:

Format Sign Exponent Mantissa Largest value Precision
float32 1 8 23 about 3.4e38 about 7 digits
float16 1 5 10 65,504 about 3 digits
bfloat16 1 8 7 about 3.4e38 about 2 digits

bf16 keeps float32's exponent, so it has the same range and training rarely overflows or underflows. fp16 has more precision but a tiny range: values above 65,504 overflow to infinity, and values below about 6e-8 underflow to zero. numpy can show this directly.

Python Stand-in: fp16 range versus bf16 range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bf16 emulation here simply truncates the lower 16 bits of the float32 (real hardware rounds to nearest), which is accurate enough to show the pattern: bf16 survives 70,000 and 1e-8 where fp16 does not, but represents 3.14159265 more coarsely.

Loss scaling

Gradients in deep networks are often tiny, many below fp16's smallest representable number. If they underflow to zero, the model stops learning. Loss scaling multiplies the loss by a large factor (such as 1024) before the backward pass, so gradients are scaled up into the representable range, then divides the gradients back before the optimizer step. Frameworks adjust the factor dynamically: if an overflow (infinity) appears, skip the step and halve the scale; after many good steps, raise it. bf16 needs none of this.

Python Stand-in: loss scaling rescues small gradients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In Transformers you simply set fp16=True or bf16=True in TrainingArguments; the scaler is handled for you. Prefer bf16=True whenever the hardware supports it (NVIDIA Ampere or newer, recent AMD and TPU).

The levers, in the order to try them

  1. Smaller per-device batch + gradient accumulation: keeps the effective batch size.
  2. Mixed precision (bf16=True): halves activation and weight memory.
  3. Gradient checkpointing (gradient_checkpointing=True): stores only some activations and recomputes the rest in the backward pass. Costs about 20-30% more time for a large memory saving.
  4. Shorter sequences (truncate, or group_by_length).
  5. 8-bit optimizers (optim="adamw_bnb_8bit"): optimizer state shrinks 4x.
  6. PEFT / QLoRA: no gradients or optimizer state for the frozen base.
  7. DeepSpeed ZeRO or FSDP: shard weights, gradients and optimizer state across several GPUs.
  8. FlashAttention (attn_implementation="flash_attention_2" or "sdpa"): attention memory grows linearly instead of quadratically.
Python
args = TrainingArguments(
    "out/fit", per_device_train_batch_size=1, gradient_accumulation_steps=32,
    bf16=True, gradient_checkpointing=True, optim="adamw_bnb_8bit",
    max_steps=1000,
)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Roughly how many bytes per parameter does mixed-precision full fine-tuning with Adam need (weights, gradients, optimizer state)?

    About 2 for weights, 2 for gradients and 12 for the optimizer state and master copy.
  2. 2. Why does bf16 usually not need loss scaling?

    bf16 keeps 8 exponent bits, so its range matches fp32, at the cost of precision.
  3. 3. What does gradient checkpointing trade?

    It recomputes some activations during the backward pass instead of storing them.
  4. 4. Your run hits out-of-memory. Which change keeps the effective batch size the same?

    per_device_batch x accumulation steps stays constant while peak memory falls.

Summary

  • Training memory = weights + gradients + optimizer state + activations; Adam with mixed precision costs about 16 bytes per trainable parameter.
  • Freezing parameters (PEFT) removes their gradients and optimizer state; quantizing the base shrinks the weights.
  • bf16 has fp32's range and needs no loss scaling; fp16 needs dynamic loss scaling.
  • Fix out-of-memory in this order: smaller batch with accumulation, bf16, gradient checkpointing, shorter sequences, 8-bit optimizer, PEFT/QLoRA, sharding.

Next: The evaluate library and metric basics