Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 32 ยท Evaluation and Deployment

Serving with FastAPI and Streaming

Serve a LangChain agent as an API with FastAPI: async endpoints, token streaming with server-sent events, thread ids, auth and rate limits. Runnable asyncio stand-ins.

A notebook is not a product. To let a website or mobile app use your chain or agent, you put it behind an HTTP API. Two features matter far more for LLM apps than for typical APIs. First, streaming: a model can take many seconds to finish, but users are happy if the first words appear in half a second. Second, state: conversations need a thread id so the checkpointer can find the history. This lesson shows a FastAPI service that handles both, and the surrounding production concerns.

FastAPI, LangChain and a model provider cannot run on this site, so real code is read-only. The runnable examples use asyncio and generators to show how streaming works, without any server.

A minimal API

Python
# Real FastAPI + LangChain (read-only)
from fastapi import FastAPI
from pydantic import BaseModel
from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver

app = FastAPI()
agent = create_agent(
    model="anthropic:claude-sonnet-4-5",
    tools=[],
    system_prompt="You are a concise assistant.",
    checkpointer=InMemorySaver(),   # use a database-backed saver in production
)

class ChatRequest(BaseModel):
    message: str
    thread_id: str

class ChatResponse(BaseModel):
    reply: str

@app.post("/chat", response_model=ChatResponse)
async def chat(req: ChatRequest) -> ChatResponse:
    config = {"configurable": {"thread_id": req.thread_id}}
    result = await agent.ainvoke(
        {"messages": [{"role": "user", "content": req.message}]}, config
    )
    return ChatResponse(reply=result["messages"][-1].content)

Points worth noting:

  • Create the agent once at startup, not per request; building it is wasteful.
  • Use async (ainvoke) so one slow model call does not block the whole server. Every LangChain runnable has ainvoke, astream and abatch.
  • Request and response models (pydantic) validate input and document the API automatically.
  • The thread id comes from the client; never let one user pass another user's id (see authentication below).

Why streaming matters

sequenceDiagram
    participant B as Browser
    participant S as API
    participant M as Model
    B->>S: POST /chat/stream
    S->>M: start generation
    M-->>S: token "Tides"
    S-->>B: data: Tides
    M-->>S: token " rise"
    S-->>B: data: rise
    M-->>S: done
    S-->>B: data: [DONE]

Tokens flow to the browser as they are produced.

LangChain streams with astream (and stream). With stream_mode="messages" a graph yields message chunks token by token; with "updates" it yields one update per step, handy for showing "searching documents..." in a UI.

Python
# Real FastAPI + LangChain (read-only)
import json
from fastapi.responses import StreamingResponse

@app.post("/chat/stream")
async def chat_stream(req: ChatRequest):
    config = {"configurable": {"thread_id": req.thread_id}}

    async def events():
        async for chunk, meta in agent.astream(
            {"messages": [{"role": "user", "content": req.message}]},
            config,
            stream_mode="messages",
        ):
            if chunk.content and meta.get("langgraph_node") == "model":
                yield f"data: {json.dumps({'token': chunk.content})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(events(), media_type="text/event-stream")

The response uses server-sent events (SSE): a long-lived HTTP response of lines that start with data: and end with a blank line. Browsers consume it with EventSource or with fetch and a stream reader. The exact shape of the chunks and metadata can differ between versions, so print one chunk in your own environment before depending on its fields.

An async streaming stand-in

The generator below plays the role of the model and astream; the loop formats each token as an SSE line, exactly as the endpoint does.

Python Stand-in: async token stream formatted as SSE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Concurrency: why async helps

While one request waits for the model, the server can handle others. The next stand-in handles three requests at once; the total time is about that of the slowest one, not the sum.

Python Stand-in: three slow requests handled concurrently

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Production checklist

Concern What to do
Persistence A Postgres or SQLite checkpointer, not InMemorySaver; see persistence
Authentication Identify the user from a token; build the thread key from the authenticated user id plus chat id, never trust a raw thread id from the client
Rate limiting Per user and per IP, so one client cannot drain your model budget
Timeouts Limit total run time and tool time; set recursion limits
Input limits Maximum message size; reject empty or abusive input
Secrets API keys in environment variables or a secrets manager
Retries Retry transient provider errors with backoff; return clear errors to the client
Observability Tracing on, request ids in logs, metrics (tracing)
Scaling Several stateless workers behind a load balancer; state lives in the database
Health checks A cheap /health endpoint that does not call the model

Other deployment options

You do not have to write the API yourself. LangChain's company offers an agent server and hosting for LangGraph apps (with built-in persistence, streaming endpoints and a studio for debugging), and older tutorials mention LangServe, which is now in maintenance mode. For a custom product, FastAPI with your own auth and logging remains a solid, simple choice. Package the service in a container (for example with the Docker tutorial) and run it on any platform that runs containers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why stream tokens to the client?

    Streaming improves perceived latency since generation can take many seconds.
  2. 2. What does server-sent events (SSE) use on the wire?

    SSE is plain text over HTTP with the text/event-stream media type.
  3. 3. Where should the thread key come from in a multi-user service?

    Deriving it from the authenticated user prevents one user from reading another's history.
  4. 4. When should you create the agent object?

    Building an agent is wasteful to repeat; create it once and reuse it.

Summary

  • Wrap the agent in an async API; build it once, call ainvoke or astream per request.
  • Stream with SSE for responsiveness; stream_mode chooses tokens or step updates.
  • Take thread ids from authenticated identity, use a persistent checkpointer, add rate limits and timeouts.
  • Package the service as a container and keep state in the database so workers stay stateless.

Next: Caching, Cost and Latency