Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 53 ยท Production

Deploying and Monitoring LLM Apps

Learn how to run LLM apps reliably: gateways, rate limits, retries with backoff, traces, model versioning, feedback loops and fallbacks, with simulations.

An LLM call is a network request to a slow, sometimes overloaded, occasionally wrong service that you do not control. The code that wraps it decides whether your users see a polished product or a stream of error pages. Deploying an LLM app is mostly ordinary backend engineering, plus a few habits that are specific to models: you log prompts, you pin model versions, and you expect failures.

This lesson covers the moving parts of a production setup: where calls go, how to survive rate limits, what to log, how to change models safely and how to learn from your users. The code is a set of simulations, because we cannot call a real API from the browser. The patterns are the same ones you would write around a real client (see Calling an LLM API).

The shape of a deployment

Most teams do not call the provider from every service. They put a thin layer in between, often called an LLM gateway.

flowchart LR
    A["Your app"] --> B["Gateway<br/>(keys, limits, logs)"]
    B --> C["Primary model"]
    B --> D["Fallback model"]
    B --> E[("Trace store")]

The gateway holds the API keys, enforces limits, retries, falls back and records every call.

A gateway gives you one place to:

  • keep API keys out of client code and rotate them;
  • apply rate limits and spending caps per user or per feature;
  • add retries, timeouts and fallbacks once instead of in every service;
  • log every request and response in a uniform format;
  • swap or add models without touching application code.

For a small app, the "gateway" can be a single module in your backend. What matters is that every model call goes through one function.

Rate limits and retries

Providers limit requests per minute and tokens per minute. When you exceed a limit, or the service is briefly overloaded, you get an error such as HTTP 429 (too many requests) or 503 (unavailable). These errors are transient: the same request succeeds a moment later. Other errors, such as 400 (bad request) or 401 (bad key), are permanent and retrying only wastes time.

The standard remedy is exponential backoff with jitter:

  1. Wait a short time, then retry.
  2. Double the wait after each failure (1 s, 2 s, 4 s ...), up to a maximum.
  3. Add random jitter so thousands of clients do not all retry at the same instant.
  4. Give up after a few attempts and fail clearly, or use a fallback.

Also honor a Retry-After header when the provider sends one, and always set a timeout on each call so a hung request does not hold a worker forever.

Python Retry with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The waits are only printed, not slept, so the simulation finishes instantly. In a real client you would call time.sleep(delay) (or await asyncio.sleep(delay)).

Retries multiply load. If every layer of your system retries three times, one failure becomes 27 calls. Retry in one place (the gateway) and keep the attempt count small. Also make sure a retried call is safe: if a tool call sends an email, retrying it may send two.

Fallbacks

When retries are not enough, a fallback keeps the feature alive. Options, from best to worst experience:

  • a second model from the same or another provider (your prompts may need small adjustments, so test it);
  • a smaller model with a shorter prompt;
  • a cached or canned answer;
  • a clear message ("The assistant is busy, please try again") rather than a stack trace.

A circuit breaker adds one more safety: after several consecutive failures, stop calling the primary for a minute and go straight to the fallback. This saves time and gives the provider room to recover.

Python Primary, fallback and a circuit breaker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

While the circuit is open the primary is not called at all. When the cool-down ends the next request tests it again: if it fails the breaker reopens, and once it succeeds the app is back to normal.

What to log

You cannot debug what you cannot see. For every model call, record a trace: a structured record that ties together everything about one request.

  • a request ID, user or session ID (hashed if needed) and timestamp;
  • the feature or prompt name and its version;
  • the model name and exact version, plus parameters such as temperature;
  • input and output token counts, latency (including time to first token) and cost;
  • the status: success, retried, fell back, timed out, blocked;
  • the prompt and response themselves, if your privacy rules allow it (see Bias, Privacy, Copyright and Responsible Use).

For multi-step systems such as agents, a trace contains several spans (one per model call or tool call) under a single request. Log as JSON lines so any tool can query them.

Python Structured traces and a daily report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dashboards built from traces answer the questions you will actually be asked: Why is it slow today? Which feature costs the most? Did the new prompt change the error rate? Watch percentiles (p95, p99), not only averages, because a few very slow calls hurt users the most.

Versioning models and prompts

Behavior changes when the model changes, and it can change even when you did not touch your code. Protect yourself:

  • Pin exact model versions (a dated snapshot name) in production instead of a moving alias such as "latest".
  • Treat prompts like code: keep them in files, give them versions and store the version in each trace (Prompt Templates and Versioning).
  • Before switching models or prompts, run your evaluation suite and compare the scores.
  • Roll out gradually: send 5% of traffic to the new version, compare metrics, then increase. Keep a one-step rollback.
  • Watch provider deprecation notices. Old model versions are retired on a schedule.

Feedback loops

Your users produce the best test data you will ever get. Capture signals such as thumbs up or down, edits to a draft, a retry click, or a handoff to a human. Store each signal with the trace ID so you can read the exact prompt and answer behind a thumbs down.

Then close the loop: review a sample of bad traces every week, add the interesting ones to your evaluation set as regression cases, and fix the prompt, the retrieval or the model choice. Remove personal data before such examples leave the production system.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which error should you NOT retry?

    An invalid key is permanent: the same request will fail every time. Rate limits, overload and timeouts are transient and worth retrying with backoff.
  2. 2. Why add random jitter to the backoff delay?

    Without jitter, all clients that failed together wait the same time and return together. Randomizing the wait spreads the load.
  3. 3. What does a circuit breaker do?

    After repeated failures the breaker opens, so requests skip the broken primary and users get a fast fallback while the provider recovers.
  4. 4. Why pin an exact model version in production?

    When an alias points to a new model, your outputs, costs and failure modes can change silently. Pinning lets you upgrade deliberately after testing.

Summary

  • Route every model call through one gateway function or service that owns keys, limits, retries, fallbacks and logging.
  • Retry only transient errors, with exponential backoff, jitter, a cap, a timeout and a small attempt count.
  • Fallbacks (another model, a smaller model, a cached answer, a clear message) and circuit breakers keep features alive during outages.
  • Log structured traces with prompt version, model version, tokens, latency, cost and status; read percentiles, not averages.
  • Pin model versions, version prompts, test with an evaluation suite and roll out gradually.
  • Feed user feedback back into your evaluation set.

Next: take the model off the network entirely in Running Models Locally.