Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Deployment and Spaces

Optimized Inference: ONNX, Optimum and Compiled Models

Make Transformers inference cheaper: export to ONNX, use Optimum and ONNX Runtime, fuse operators, quantize weights, compile with torch.compile, and benchmark honestly.

A model that was trained in PyTorch does not have to be served in PyTorch. Training needs flexibility (autograd, dynamic Python code); inference needs the opposite: a fixed computation that can be analysed, simplified and run by a lean engine. This lesson surveys the main routes to faster, cheaper inference for encoder models such as BERT and for smaller generative models, and shows what the key optimizations do.

ONNX Runtime, Optimum and a GPU are not available on this site, so the export commands are read-only and no speed-up numbers from real hardware are claimed. The runnable examples are small numpy stand-ins that demonstrate the ideas: graph simplification, int8 quantization and a trustworthy benchmark loop.

The options at a glance

Route What it does Typical gain Cost
Use torch.inference_mode() and fp16/bf16 No autograd bookkeeping, half the memory traffic Large, free Check accuracy
torch.compile Traces the model and generates fused kernels 1.2x to 2x on GPU Compile time, some model limits
Better attention (SDPA / FlashAttention) Memory-efficient attention kernels Big for long inputs Often one flag: attn_implementation="sdpa"
ONNX + ONNX Runtime Portable graph, graph optimizations, many hardware back ends Good on CPU Export step, new runtime
Quantization (int8, 4-bit) Smaller weights, faster memory-bound ops 2x to 4x smaller Possible accuracy loss
Dedicated servers (TGI, vLLM, TensorRT-LLM) Continuous batching, paged KV cache Largest for LLMs Operational complexity

Start at the top of the table: the cheap changes first, and measure after each one.

Exporting with Optimum

Optimum is the Hugging Face library that connects Transformers to hardware-specific runtimes. For ONNX:

Terminal
pip install "optimum[onnxruntime]"
optimum-cli export onnx --model your-name/support-ticket-classifier onnx_model/

Then load the exported model with an API that mirrors the original:

Python
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

model = ORTModelForSequenceClassification.from_pretrained("onnx_model")
tokenizer = AutoTokenizer.from_pretrained("onnx_model")
clf = pipeline("text-classification", model=model, tokenizer=tokenizer)
print(clf("Where is my order?"))

The pipeline code does not change; only the model class does. Optimum can also apply graph optimization (ORTOptimizer) and dynamic quantization (ORTQuantizer) to the exported file. Always check that the ONNX model's outputs match the PyTorch model's on a sample of inputs (a maximum absolute difference around 1e-4 to 1e-3 is normal) before you rely on it.

What graph optimization does

An exported model is a graph of operators. An optimizer rewrites it into an equivalent, cheaper graph. Two classic passes are constant folding (computing anything that only depends on constants once, at export time) and operator fusion (replacing a chain such as MatMul then Add then GELU with one fused kernel so intermediate results never travel to memory and back).

Python Stand-in: constant folding and fusion on a toy graph

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fewer nodes means fewer kernel launches and fewer round trips through memory, with identical results. Real optimizers apply dozens of such passes (attention fusion, LayerNorm fusion, redundant-cast removal).

Quantization for speed and size

Dynamic int8 quantization stores each weight matrix as 8-bit integers plus a scale, and computes with integer arithmetic where the hardware supports it. The weights shrink fourfold relative to float32. The question is always: how much accuracy do you lose?

Python int8 quantization error of a weight matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The outlier case explains why practical schemes use per-channel scales or group-wise scales: one large value should not coarsen the resolution for every other weight. See quantization fundamentals for the full picture.

Benchmark honestly

Wrong benchmarks lead to wrong decisions. A trustworthy loop warms up first (the first runs include lazy initialisation and compilation), runs many iterations, reports the median and p95, uses realistic input lengths and batch sizes, and compares the same task output (accuracy on a validation sample) alongside speed.

Python
import time, statistics

def bench(fn, warmup=5, runs=50):
    for _ in range(warmup):
        fn()
    times = []
    for _ in range(runs):
        t0 = time.perf_counter()
        fn()
        times.append((time.perf_counter() - t0) * 1000)
    times.sort()
    return statistics.median(times), times[int(0.95 * len(times)) - 1]

On GPUs, remember to synchronize (torch.cuda.synchronize()) before reading the clock, because kernels run asynchronously.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does constant folding do?

    It replaces constant-only subgraphs with their results at export or load time.
  2. 2. Why does operator fusion speed things up?

    Fused kernels avoid writing and re-reading intermediates and reduce launch overhead.
  3. 3. After exporting to ONNX, what should you verify before deploying?

    Export can change numerics or drop dynamic behaviour; compare outputs and accuracy.
  4. 4. Why do benchmarks include warm-up runs?

    One-time costs would otherwise distort the measured latency.

Summary

  • Try the cheap wins first: inference mode, bf16/fp16, SDPA attention, then torch.compile.
  • Optimum exports to ONNX and plugs ONNX Runtime into the same pipeline API; verify output parity after export.
  • Graph optimizers fold constants and fuse operators; quantization shrinks weights at some accuracy risk (watch outliers).
  • Benchmark with warm-up, realistic inputs, median and p95, and measure task quality too.

Next: Project: a support-ticket classifier from data to demo