Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Inference and Optimization

Serving, Batching and Speculative Decoding

Learn how LLM servers balance latency and throughput: continuous batching, prefill vs decode, memory-bound math, speculative decoding speedups and key serving metrics.

Training a model is a one-time cost. Serving it is a cost you pay on every request, forever. A production inference server has to answer thousands of users at once, keep latency low, and keep GPUs busy. This lesson covers the metrics, why batching is so effective, how servers batch requests that finish at different times, and how speculative decoding generates several tokens per model pass.

Metrics

  • Time to first token (TTFT): how long until the first token appears. Dominated by queueing and prefill.
  • Time per output token (TPOT) or inter-token latency: how fast the text streams after that.
  • Throughput: tokens per second across all users, which sets cost per token.
  • Goodput: throughput of requests that met their latency targets.

Latency and throughput pull against each other. Bigger batches raise throughput but make each user wait longer per token.

Why batching works: the roofline

During decoding the GPU must stream all weights from memory to produce one token for one request. The arithmetic is tiny compared with the data movement, so the chip sits mostly idle. If you decode 32 requests together, the weights are read once but used 32 times: almost free extra throughput.

Python Decode throughput vs batch size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Throughput grows almost linearly with batch size until the step becomes compute-bound. (The model ignores the KV cache reads, which grow with batch size and context, and other overheads, so real numbers are lower.) The numbers are illustrative hardware figures, not a benchmark of any specific GPU.

Static vs continuous batching

With static batching the server collects a batch, runs until the longest request is finished, then starts the next batch. Short requests wait for long ones, wasting slots.

With continuous batching (also called in-flight batching), the scheduler works at the level of single decode steps. When a request finishes, a waiting request takes its slot immediately. Combined with paged KV memory (KV cache), this is the core of modern servers such as vLLM, TGI and TensorRT-LLM.

Python Static vs continuous batching

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Continuous batching finishes the same work in fewer steps because no slot waits for a straggler.

Prefill and decode interference

A long prompt's prefill is compute-heavy and can stall decode steps of other users. Servers therefore use chunked prefill (process long prompts in slices mixed with decode steps), or disaggregated serving (separate GPU pools for prefill and decode).

Speculative decoding

Decoding is memory-bound, so verifying several tokens in one pass costs about the same as generating one. Speculative decoding exploits this:

  1. A small, fast draft model proposes k tokens.
  2. The big target model scores all k in a single forward pass.
  3. Tokens are accepted left to right while they match what the target would have sampled; at the first rejection the target supplies a corrected token.

With the right acceptance rule the output distribution is exactly the target model's, so quality is unchanged. If each draft token is accepted with probability a, the expected number of tokens produced per target pass is (1 - a^(k+1)) / (1 - a).

Python Expected speedup of speculative decoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the draft model costs a fraction c of a target pass, the wall-clock speedup is roughly expected_tokens / (1 + k*c). A good draft is both cheap and well-aligned with the target. Variants: self-speculation (extra heads on the model itself, such as Medusa or EAGLE) and prompt lookup (copy n-grams from the prompt, handy for editing and summarizing).

Other levers

  • Prefix caching: reuse prefill for repeated system prompts.
  • Tensor and pipeline parallelism: split a large model across GPUs.
  • Quantization: smaller weights, more bandwidth headroom (Weight Quantization).
  • Fused kernels (FlashAttention): avoid writing the big attention matrix to memory.
  • Routing to a smaller model: send easy requests to a cheaper model.
  • Caching answers: exact or semantic caches for repeated questions.
  • Structured output decoding: constrain tokens to a grammar or JSON schema.

Why outputs vary even at temperature 0

Floating-point addition is not associative. Different batch sizes or kernels sum in different orders, giving tiny differences that can flip a close argmax. So "deterministic" decoding (Decoding) is deterministic only within one exact setup.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does batching increase decode throughput so much?

    Decoding is memory-bound; extra requests in the batch add little extra data movement.
  2. 2. What problem does continuous batching solve?

    Slots are refilled at every decode step, so finished requests do not leave capacity idle.
  3. 3. In speculative decoding, who decides the final tokens?

    Draft tokens are accepted only when consistent with the target's distribution; rejection yields a target-sampled token.
  4. 4. Which metric measures how long the user waits before text starts to appear?

    TTFT covers queueing plus prefill.

Summary

  • Serving is judged by TTFT, per-token latency and throughput.
  • Decode is memory-bound, so batching gives near-free throughput until compute becomes the limit.
  • Continuous batching and paged KV memory keep GPUs full; chunked prefill protects latency.
  • Speculative decoding yields several tokens per target pass with unchanged output quality.

Next: Perplexity and Language Metrics