Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 32 ยท Evaluation and Deployment

Caching, Cost and Latency

Cut LLM cost and latency: response caching, prompt caching, smaller prompts, model routing, batching and parallel calls. Runnable Python stand-ins with a cost calculator.

Two numbers decide whether an LLM feature is viable: what each request costs and how long the user waits. Both are driven by the same things: how many model calls you make, how big their prompts and outputs are, and which model handles them. Most apps can cut cost and latency dramatically with a handful of techniques that need no change to the product. This lesson goes through them in order of effort.

LangChain and real model providers are not available on this site, so real code is read-only. The runnable examples are plain Python with made-up prices and a fake model, which is enough to see the effect of each technique.

Where the cost comes from

Providers charge per token, usually with separate prices for input (prompt) and output (completion), and output tokens cost more. A request's cost is roughly:

cost = input_tokens x input_price + output_tokens x output_price

An agent multiplies that: each loop iteration re-sends the whole history, so a ten-step run pays for the early messages ten times. Long tool outputs and long system prompts are the usual culprits.

Latency has three parts: network, time to first token (grows with prompt length) and generation time (grows with output length). Output tokens are the slowest, so shorter answers are faster.

Python Stand-in: estimate cost of an agent run

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Input tokens grow faster than the number of steps, because each step carries all earlier ones. Trimming and summarizing history (earlier lesson) attacks exactly this.

Technique 1: cache identical requests

If the same prompt is sent twice, the answer can be reused. LangChain has a global LLM cache:

Python
# Real LangChain (read-only)
from langchain_core.globals import set_llm_cache
from langchain_core.caches import InMemoryCache

set_llm_cache(InMemoryCache())      # for production use a persistent cache (SQLite, Redis, ...)

model.invoke("What is the capital of France?")   # calls the provider
model.invoke("What is the capital of France?")   # served from the cache, free and instant

The cache key is the full prompt plus the model settings, so only exact repeats hit. That fits FAQ-style traffic, batch jobs that rerun and test suites, but not free-form chat. Do not cache answers that depend on fresh data, per-user data or randomness, and remember that a cache is a copy of an answer that may become stale.

Python Stand-in: a response cache with hit rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A semantic cache goes further: it embeds the question and reuses an answer for a similar earlier question. That raises the hit rate, but similar questions can need different answers, so set a strict similarity threshold and monitor mistakes.

Technique 2: provider prompt caching

Several providers can cache the prefix of a prompt on their side. If every request starts with the same long system prompt or the same big document, the repeated part is billed at a much lower rate and starts faster. To benefit, put stable content first (instructions, reference text, tool definitions) and variable content last (the user's question). Check your provider's documentation for how it is enabled and for the cheaper price and cache lifetime.

Technique 3: shorter prompts and outputs

  • Cut redundant instructions; test whether the shorter prompt scores the same (evaluation).
  • Retrieve fewer, better chunks instead of many.
  • Truncate or summarize large tool outputs before they go into the history.
  • Ask for brief output, set max_tokens, and use structured output with short fields.
  • Do not send fields the model does not need.

Technique 4: right-size the model

Not every step needs the strongest model. Route easy work to a small, cheap, fast model and escalate hard cases:

  • classification, extraction, short rewrites and routing decisions often work well on a small model;
  • planning, difficult reasoning and final writing may need a larger one;
  • summaries inside a memory middleware can use a small model.
Python Stand-in: route by difficulty

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Evaluate before routing: measure that the small model's quality on the easy tasks is good enough, instead of assuming.

Technique 5: do less, in parallel

  • Parallel calls: independent steps (several documents to summarize, several tool calls) can run together. In LangChain, RunnableParallel, .batch() with max_concurrency and abatch do this. Latency becomes the slowest call, not the sum.
  • Batch APIs: for offline work (nightly summaries, evaluations) many providers offer asynchronous batch endpoints at a discount.
  • Skip the model: handle greetings, exact matches and simple rules in code.
  • Stream so users see output early; it does not reduce total time but greatly improves perceived speed (previous lesson).
Python
# Real LangChain (read-only)
summaries = summarize_chain.batch(
    [{"text": d} for d in documents],
    config={"max_concurrency": 5},      # at most 5 provider calls at once
)

Measure first

Do not guess. Use traces (tracing lesson) to find the biggest line items: which step uses the most tokens, which call is slowest, which prompts repeat. Fix the top item, re-measure, repeat. Also set budgets and alerts: per-user token limits, a daily spend cap and an alert when cost per request jumps after a deploy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do agent runs get expensive quickly?

    Input tokens accumulate because every model call includes all earlier messages.
  2. 2. When does an exact-match LLM cache hit?

    The key is the full prompt plus model parameters, so only exact repeats are served from cache.
  3. 3. How should you arrange a prompt to benefit from provider prompt caching?

    Caching works on the repeated prefix, so keep the unchanging part at the start.
  4. 4. What is model routing?

    Routing matches model size to task difficulty to save cost and time.

Summary

  • Cost scales with tokens in and out; agents re-send history every step.
  • Cache exact repeats, use provider prompt caching with stable prefixes, and consider semantic caching carefully.
  • Shorten prompts, outputs and tool results; route easy tasks to small models.
  • Run independent work in parallel, use batch APIs offline, and stream for perceived speed.
  • Measure with traces, evaluate every change, and set budgets and alerts.

Next: Project: A Research Assistant Agent