Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท Prompt Patterns in Production

Cost, Latency and Caching for LLM Apps

Estimate and cut what a prompt costs: count tokens, shorten prompts, cache repeated work, route easy tasks to small models, stream output and batch background jobs.

A prompt that works is not yet a prompt you can afford. Providers charge per token, with different prices for input and output, and users give up on slow answers. At a thousand requests a day a wasteful prompt is a rounding error; at a million it is a budget line. This lesson shows how to estimate cost, then lists the levers: shorter prompts, caching, model routing, streaming and batching.

Prices and limits change often, so the numbers below are made-up example prices to teach the arithmetic. Always read your provider's current price list.

Estimating cost

Cost of one call = input tokens x input price + output tokens x output price. Output tokens usually cost several times more per token than input tokens, and they are also what makes responses slow, because the model produces them one at a time.

Python Cost per call and per month

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even though the output is five times shorter than the input here, it accounts for a large share of the bill. Both sides matter.

Lever 1: shorter prompts and shorter answers

Lever 2: caching

Most traffic repeats. Three caches help, from simplest to cleverest:

  1. Exact-match cache. Key = hash of (prompt version, model, settings, full input). Identical request, stored answer, zero cost and almost zero latency. Only safe when the answer does not need to be fresh and the output does not depend on the user (see privacy: include the user in the key when the prompt holds private data). Works best with temperature 0.
  2. Provider prompt caching. Many providers discount, and speed up, the prefix of a prompt that they have seen recently (a long system prompt or document followed by a changing question). To benefit, put the stable text first and the changing text last, and keep the prefix byte-identical.
  3. Semantic cache. Match similar questions by embedding similarity. It hits more often but can return a wrong answer for a question that only looks similar, so use a strict threshold and only for low-risk uses.
Python Exact-match cache with hit rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the prompt version is part of the key, a prompt release automatically invalidates old answers.

Lever 3: route by difficulty

Not every request needs your biggest model. A router sends easy requests to a small, cheap, fast model and hard ones to a large model. Start simple: route by task type (classification to small, long reasoning to large), then by a cheap difficulty signal (input length, a keyword, a confidence score). Measure quality per route with your test set so that savings do not silently cost accuracy. A common pattern is a cascade: try the small model, and escalate to the large one only if a guardrail or confidence check fails.

Python Cascade routing and its savings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the price of escalation: a request that falls through pays for both models. A cascade only wins when the small model handles most traffic.

Latency

Time to a finished answer is roughly: network + time to first token (grows with input length) + output tokens x time per token. To cut it:

  • Stream the response so the user reads while it is generated. Total time is unchanged, but perceived speed improves a lot.
  • Shorten the output. Fewer output tokens is the most reliable speed-up.
  • Parallelize independent calls (for example classify and extract at the same time) instead of chaining them.
  • Avoid needless chains. Each extra chained step adds a full round trip.
  • Use smaller models where quality allows, and set sensible timeouts with a fallback.

Batching background work

Work that nobody is waiting for (nightly summaries, bulk classification, evaluations) can use a provider's batch interface, which typically trades a delay of hours for a lower price. Also remember rate limits: queue, retry with exponential backoff and jitter on rate-limit errors, and cap concurrency.

Watch the budget

Log tokens in and out, cost and latency for every request, tagged with prompt name and version (monitoring). Set per-user and per-day limits, and alert on sudden jumps: a runaway tool loop or an attacker pasting huge inputs shows up first on the bill.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can output tokens dominate the cost and latency of a call?

    Output tokens are priced higher and generated sequentially, so long answers are slow and expensive.
  2. 2. To benefit from provider prompt caching, how should you order the prompt?

    Caching works on a shared prefix, so the unchanging part must come first and stay identical.
  3. 3. Why include the prompt version in a cache key?

    A new version changes the key, automatically invalidating stale answers.
  4. 4. What is the risk of a semantic cache?

    Similarity is not equality; a strict threshold and low-risk use are advised.
  5. 5. Which change improves perceived latency without changing total time?

    Streaming lets users start reading immediately.

Summary

  • Cost = input tokens x input price + output tokens x output price; measure it per request.
  • Levers: shorter prompts and outputs, exact and prefix caching, cascades and routers, streaming, parallel calls and batch jobs.
  • Every optimization must be checked against your test set so quality does not drop.
  • Log tokens, cost and latency per prompt version and alert on spikes.

Next: Monitoring and Feedback Loops