Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 53 ยท Production

Cost, Latency and Caching

Learn what drives LLM latency and cost: time to first token, streaming, batching, prompt caching, response caching and model routing, with simulators.

A demo that answers in eight seconds and costs a cent per call is fine for a blog post. The same demo with 50,000 users is slow, expensive and a little embarrassing. Production LLM apps live or die by two numbers: how long the user waits and how much each request costs. Luckily the same few ideas improve both.

You already know how tokens are priced and why long contexts are costly. This lesson turns that knowledge into a toolbox: streaming, batching, caching at two levels, routing and smaller models. Everything here runs as simulations with invented numbers, so you can experiment freely. Measure your own system before you copy any figure.

Where the time goes

An LLM response has two phases with different speeds:

  1. Prefill. The model reads your whole prompt in parallel. This sets the time to first token (TTFT). Longer prompts mean a longer wait.
  2. Decode. The model writes one token at a time. The speed here is tokens per second, and total time grows with the answer length.

So a rough model of total latency is TTFT + output_tokens / tokens_per_second, plus network time and anything your own code adds (retrieval, tool calls).

Two phases of latency

Prefill Reads the prompt in parallel. Sets time to first token.
Decode Writes one token at a time. Sets the speed of the text.

Cut the prompt to speed up the first phase, cap the answer to speed up the second.

1 / 2

Streaming: faster feel, same total

With streaming, the server sends tokens as they are generated instead of waiting for the full answer. Total time is the same, but the user sees the first words after the TTFT. For a 400-token answer that is the difference between staring at a spinner for ten seconds and reading along after one.

Python Perceived vs total latency

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Streaming does not make anything cheaper. It also has costs of its own: you must handle partial output (half a JSON object is not valid JSON), and moderation checks work on text you have already shown.

Prompt caching

Many requests start with the same long prefix: a system prompt, tool definitions, a few examples, a big document. Prompt caching (also called context caching) lets the provider remember the processed prefix. A later request that starts with exactly the same tokens skips most of the prefill work, so it is faster and the cached input tokens are billed at a discount.

Rules of thumb that hold for most providers (check the details of yours):

  • Only an identical prefix can be reused. One changed character early in the prompt breaks the match for everything after it.
  • Put stable content first (instructions, documents) and volatile content last (the user's question, the current time).
  • Caches expire after minutes, so they help most for busy traffic or repeated calls on the same document.

Never put a timestamp or request ID at the top of a prompt. It makes every prefix unique and the cache never hits.

Response caching

A response cache sits in front of the model: if you have answered this exact question before, return the stored answer with no model call at all. It is the cheapest and fastest request there is. It only fits when the answer does not depend on hidden state and a slightly stale answer is acceptable (FAQ bots, classification of repeated inputs, deterministic extraction).

Two cautions. First, normalize the key (trim, lowercase, collapse spaces) or near-duplicates miss. Second, never share cached answers between users if the prompt contains private data. A semantic cache matches by embedding similarity instead of exact text; it hits more often but can return a wrong answer for a question that only looks similar, so set the threshold high.

The simulator below replays a stream of questions through a response cache and compares the bill.

Python Response cache simulator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only five distinct questions here, so almost everything hits. The saving depends entirely on how repetitive your traffic is. A support bot may see a 60% hit rate; a creative writing tool may see 0%.

Model routing and smaller models

Not every request needs your biggest model. A router sends easy requests to a small, fast, cheap model and hard ones to a large one. The router itself can be a few rules (length, keywords, the feature being used) or a tiny classifier. A common variant is a cascade: try the small model first and escalate only if a check fails or its confidence is low.

Smaller models are also easier to speed up: they have lower TTFT and higher tokens per second. If a small model passes your evaluation set for a task, using it is almost always the right call. Other levers worth knowing:

  • Batch APIs. For work nobody waits for (nightly summaries, bulk labeling), many providers offer asynchronous batch endpoints at a discount in exchange for results within hours.
  • Shorter prompts. Trim examples and boilerplate. Every input token costs money and prefill time.
  • Output limits. Set max_tokens and ask for concise formats, because output tokens are the expensive and slow ones.
Python Routing and caching combined

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the three rows together: routing keeps most of the quality at a fraction of the large-model bill, and its errors (a hard task sent to the small model) are where the quality is lost. That is why you measure quality per task type, not just overall.

A checklist before launch

  • Measure TTFT and total latency at the 50th and 95th percentile, not only the average.
  • Stream every user-facing answer.
  • Order prompts with stable content first so provider caching can hit.
  • Cache what is safe to cache; log hit rates.
  • Pick the smallest model that passes your tests, and route the rest.
  • Set budgets and alerts on spend (see Deploying and Monitoring LLM Apps).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What mainly determines time to first token?

    Before the first output token, the model must read the whole prompt. Longer prompts mean a longer wait. Answer length affects total time instead.
  2. 2. Why should stable instructions go at the start of a prompt when you use prompt caching?

    Prefix caching matches tokens from the start. Any difference near the top invalidates everything after it, so volatile parts belong at the end.
  3. 3. Which is the main risk of a semantic cache?

    Semantic matching is approximate. Questions that look alike can have different correct answers, so use a high similarity threshold and test it.
  4. 4. What does streaming improve?

    Streaming shows output as it is produced. The total generation time and the bill stay the same.

Summary

  • Latency has two parts: prefill (time to first token) and decode (tokens per second times answer length).
  • Streaming improves how fast the app feels, not the total time or cost.
  • Prompt caching reuses an identical prefix; put stable text first and volatile text last.
  • Response caches skip the model entirely for repeated questions; normalize keys and never leak private data.
  • Routing and cascades send easy work to small models; batch endpoints discount work nobody waits for.
  • Measure percentiles, hit rates and quality per task type before and after each change.

Next: keep the app reliable and visible in Deploying and Monitoring LLM Apps.