Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 39 ยท Frameworks and Deployment

Deploying Agents to Production

Take an AI agent from notebook to production: architecture, queues for long runs, timeouts, retries, rate limits, cost control, secrets, versioning and a launch checklist.

A demo agent runs once, on your laptop, while you watch. A production agent runs thousands of times, unattended, for people who will blame you for every failure. The model is the easy part. The hard part is everything around it: runs that take minutes, providers that time out, users who retry, bills that spike, and prompts that change behaviour overnight.

This lesson is a practical tour of what changes when an agent goes live: architecture, handling long runs, reliability (timeouts, retries, idempotency), cost controls, secrets, versioning and release practice. It finishes with a launch checklist. Two runnable examples show retries with backoff and a budget guard.

Hosting, queues and model APIs are not available on this site, so the code focuses on the logic you write yourself, using stand-in functions for flaky calls.

A typical architecture

flowchart LR
  U["User or app"] --> API["API endpoint"]
  API --> Q["Job queue"]
  Q --> W["Agent worker"]
  W --> M["Model provider"]
  W --> T["Tools and databases"]
  W --> S["State store"]
  W --> L["Traces and metrics"]
  API --> S

The API returns quickly; workers do the slow agent run and save progress in the state store.

Why a queue? An agent run can take 30 seconds or 30 minutes, longer than a normal web request should wait. The usual pattern: the endpoint creates a run, puts a job on a queue and returns a run id. The client polls for status or receives streamed updates. Workers pick up jobs, which also lets you scale them separately and survive restarts.

Streaming and progress

Users hate staring at a spinner. Stream partial output (tokens, step names such as "searching the docs") to the client. Progress messages make a 40-second run feel responsive and let users cancel. Provide a cancel operation that stops the worker between steps.

Reliability: timeouts, retries and backoff

External calls fail. Set a timeout on every model and tool call. Retry transient errors (timeouts, rate limits, 5xx) with exponential backoff and a little randomness (jitter), and do not retry permanent errors (invalid request, authentication).

Python Retry with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sleep is stubbed out so the demo finishes instantly; in a real worker you would pass time.sleep.

Idempotency: safe to repeat

Retries and user double-clicks mean the same request may run twice. If your agent sends an email or charges a card, running twice is a real bug. Give each request an idempotency key; store the result under that key and return it for repeats. Tools that change the world should accept the key too.

Python Idempotent run submission

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Controlling cost

Agent bills are driven by tokens, and runs can loop. Put limits in code.

Python A per-run budget guard

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More levers: cache repeated model calls and tool results, use a cheaper model for easy steps (routing, classification) and a stronger one for hard steps, shorten prompts (see context windows), and set per-user and per-day quotas with rate limiting.

Secrets and security

  • Keep API keys in a secrets manager or environment variables, never in prompts, code or traces.
  • Give the agent scoped credentials, one per tool, with the least access needed.
  • Run code-execution tools in a sandbox with no network and strict time and memory limits.
  • Apply the layers from guardrails and approvals from human-in-the-loop.
  • Authenticate every user and enforce per-user data access outside the model.

Versioning and release

Prompts, tool descriptions, model names and code together define the agent's behaviour, so version them together and record the version in every trace. Release carefully:

  1. Run the eval set (see evaluating agents) in CI; block on regressions.
  2. Deploy to a small percentage of traffic (canary) or to shadow mode, where the new version runs but its output is not shown.
  3. Compare success, cost and latency against the old version.
  4. Keep a one-step rollback.

Pin the model version where the provider allows it. Model updates can silently change behaviour, so re-run evals whenever the provider changes the model.

Operating the agent

Monitor with the metrics from observability and tracing: success rate, latency percentiles, cost per run, tool error rate. Set alerts, keep an on-call runbook ("what to do if the provider is down") and plan for provider outages with a fallback model or a graceful "try again later" message.

Launch checklist

  • Timeouts, retries with backoff, and idempotency keys in place.
  • Budgets: steps, tokens, cost and per-user rate limits.
  • Long runs on a queue with status, streaming and cancel.
  • Secrets managed, tools least-privileged, risky actions approved.
  • Evals in CI, canary release, rollback ready, versions recorded.
  • Tracing, dashboards, alerts and redaction of personal data.
  • A clear message for users when the agent cannot help or fails.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why put agent runs on a job queue?

    The endpoint returns a run id quickly while workers do the long job.
  2. 2. Which errors should usually be retried?

    Retry transient failures with backoff; permanent errors will fail again.
  3. 3. What is an idempotency key for?

    The same key returns the stored result instead of running the action again.
  4. 4. Why version prompts together with code?

    Recording the whole configuration lets you compare releases and revert.
  5. 5. How can you release a new agent version safely?

    Staged rollout with metrics limits the damage of regressions.

Summary

  • Production agents run behind a queue, stream progress and support cancel.
  • Use timeouts, retries with backoff and idempotency keys to survive failures and repeats.
  • Enforce budgets, quotas and caching to keep costs predictable.
  • Protect secrets, scope credentials, sandbox code and approve risky actions.
  • Version prompts and code together, gate releases on evals, canary and keep rollback.

Next: Project: Research Assistant Agent