Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 38 ยท Pipelines and Inference

Batching, Devices and Inference Speed

Speed up inference with batching, sorting by length, half precision, GPU placement, KV caching and memory arithmetic, with runnable estimators.

Running one example at a time works for a demo, but real workloads process thousands of items. A few habits make inference several times faster and cheaper: batch the inputs, avoid wasted padding, use lower precision, keep the model on the right device and stop recomputing what you already know.

Real library code is read-only on this site; the estimators below are stand-ins that make the arithmetic visible.

Batching

A GPU is most efficient when it multiplies big matrices. One sentence uses a tiny fraction of it, while a batch of 32 uses nearly the same time for 32 times more work. Pass a list, and set batch_size in the pipeline:

Python
from transformers import pipeline

clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english", device=0)
results = clf(list_of_10000_texts, batch_size=32)

Bigger batches help up to a point, then you run out of memory or hit diminishing returns. Tune on your hardware by doubling the batch size until memory fails, then step back. On CPU, batching helps less and can even hurt with very uneven lengths.

For huge datasets, pass a generator or a datasets.Dataset and iterate lazily instead of building a giant list:

Python
for out in clf(KeyDataset(dataset, "text"), batch_size=32):
    ...

Padding waste

Every sequence in a batch is padded to the longest one. If one text has 500 tokens and the rest have 20, you pay for 500 tokens x batch size. Two remedies:

  1. Sort by length before batching so neighbours have similar lengths (and restore the original order afterwards).
  2. Cap max_length at what your task really needs.
Python Stand-in: padding waste with and without sorting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sorting alone removes a large share of the waste. This is the idea behind "dynamic padding" and length-grouped batching; the data collator lesson applies it to training.

Half precision

Weights are 32-bit floats by default. Loading them in 16-bit halves the memory and, on modern GPUs, speeds up matrix multiplication with little loss of quality.

Python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "gpt2", torch_dtype=torch.bfloat16, device_map="auto"
)
  • float16 has more precision but a small range and can overflow.
  • bfloat16 has the same range as float32 with less precision; it is the safe default on newer GPUs (Ampere and later).
  • CPUs without 16-bit support may be slower in half precision; test first.
Python Stand-in: precision and rounding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

float16 rounds 1000.7 and overflows to infinity at 70000, which is why bfloat16 (or float32 for sensitive layers) exists.

Moving to the right device

The model and the inputs must live on the same device. In a manual loop:

Python
model.to("cuda").eval()
inputs = tok(texts, padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
    logits = model(**inputs).logits

torch.inference_mode() (or no_grad) saves memory by not storing activations for backpropagation. Moving data between CPU and GPU is slow, so move it once per batch, not per item.

The KV cache in generation

When generating, each new token needs the keys and values of all earlier tokens. Instead of recomputing them, the model stores them in a KV cache. Generation is then fast per token, but the cache grows with sequence length and batch size and can use more memory than the weights.

Python Stand-in: KV cache size estimator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Doubling the context or the batch doubles the cache. That is why serving long chats to many users needs careful memory planning.

More speed-ups

  • torch.compile(model) can speed up repeated calls after a warm-up.
  • Flash attention / SDPA: attn_implementation="sdpa" (or "flash_attention_2" on supported GPUs) reduces memory and time for long sequences.
  • Smaller or distilled models (DistilBERT instead of BERT) are often the biggest win.
  • ONNX export and quantization reduce latency further for CPU serving.
  • Dedicated serving engines (such as Text Generation Inference or vLLM) batch requests continuously for chat workloads.
flowchart LR
  A["Slow inference"] --> B["Batch inputs"]
  B --> C["Sort by length"]
  C --> D["16-bit weights"]
  D --> E["Faster attention"]
  E --> F["Smaller model or serving engine"]

Apply the cheap fixes first and measure after each one.

Measuring properly

Always warm up (run once before timing), call torch.cuda.synchronize() before reading the clock on GPU, and report both latency (time for one request) and throughput (items per second). Improving one can hurt the other: big batches raise throughput but make each request wait longer.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does batching speed up inference on a GPU?

    A single input underuses the GPU; a batch does much more useful work per second.
  2. 2. How does sorting texts by length help?

    Similar-length neighbours mean the longest item in each batch is not much longer than the rest.
  3. 3. Which 16-bit format keeps float32's numeric range?

    bfloat16 has the same exponent size as float32, so it rarely overflows.
  4. 4. What grows with sequence length and batch size during generation?

    Cached keys and values are stored for every token of every sequence.

Summary

  • Batch inputs, sort by length and cap length to avoid padding waste.
  • Use bfloat16 or float16 where supported, and keep model and tensors on one device.
  • Use inference_mode, a KV cache and efficient attention for generation.
  • Measure latency and throughput after warm-up, and change one thing at a time.

Next: Loading Datasets with the datasets Library