Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 38 ยท Deployment and Spaces

Serving a Model with FastAPI and Batching

Wrap a Hugging Face pipeline in a FastAPI service, load it once, validate input, and use dynamic micro-batching to raise throughput. Includes runnable latency simulations.

A notebook proves a model works. A service lets other programs use it: a web app sends text over HTTP and gets a prediction back. The two hard parts are not the web framework but the model: it is slow to load, uses a lot of memory, and runs much more efficiently on several inputs at once than on one.

This lesson shows the standard shape of a Transformers service, then explains and simulates dynamic batching, the main trick for turning a model into a high-throughput endpoint.

FastAPI, uvicorn and Transformers do not run on this site, so the service code is shown read-only. The runnable examples are plain Python simulations of the ideas (virtual clock, fake model cost), clearly labelled as stand-ins.

A minimal service

The rule of thumb: load the model once at startup, never per request. Loading can take seconds and gigabytes; a request should only run the forward pass.

Python
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from transformers import pipeline

models = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    models["clf"] = pipeline(
        "text-classification",
        model="your-name/support-ticket-classifier",
        device_map="auto",
    )
    yield
    models.clear()

app = FastAPI(lifespan=lifespan)

class Ticket(BaseModel):
    text: str = Field(min_length=1, max_length=4000)

class Prediction(BaseModel):
    label: str
    score: float

@app.post("/classify", response_model=Prediction)
def classify(ticket: Ticket) -> Prediction:
    result = models["clf"](ticket.text, truncation=True, max_length=256)[0]
    return Prediction(label=result["label"], score=round(result["score"], 4))

@app.get("/health")
def health() -> dict:
    return {"status": "ok", "model_loaded": "clf" in models}

You would start it with uvicorn app:app --host 0.0.0.0 --port 8000. Notes on the choices:

  • def (not async def) endpoints run in a thread pool, so a slow forward pass does not block the event loop. An async def endpoint that calls the model directly would freeze every other request while it runs.
  • Validate input (length limits, types) before the model sees it; set truncation=True and max_length so one huge request cannot exhaust memory.
  • Add a health endpoint for load balancers and a readiness check that is true only after the model has loaded.
  • Return plain, stable JSON (label and a rounded score), not raw tensors; clients should not depend on library internals.

Why batching matters

A GPU does the same work for 1 input as for 8 inputs in nearly the same time, because most of the cost is fixed (launching kernels, reading the weights from memory). Serving requests one by one wastes that capacity. Dynamic batching collects requests that arrive close together, runs them as one batch, and splits the results back to the callers.

A batcher has two knobs:

  • max_batch_size: stop collecting when this many requests are waiting.
  • max_wait_ms: never make the first waiting request wait longer than this for company.

The trade-off is explicit: a little added latency for each request, much higher throughput overall.

Python Simulation: one by one versus micro-batching

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Run it and compare the rows: the single-request setting cannot keep up with one request every 6 ms (each takes 22 ms alone), so a queue builds and latency explodes, while the batched settings serve the same traffic with a short, steady delay. Your exact numbers depend on the cost model; the shape of the result is the point.

Padding waste inside a batch

Inside a batch, every sequence is padded to the longest one (see data collators and dynamic padding). If one request has 400 tokens and seven have 20, the batch computes 8 x 400 tokens. Servers reduce this by sorting waiting requests by length into buckets, or by using continuous batching (used by servers such as Text Generation Inference and vLLM for LLMs), where finished sequences leave the batch and new ones join at every generation step.

Python Padding waste in one batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other production concerns

  • Concurrency and workers. One process holds one copy of the model. Several uvicorn workers each load their own copy, so memory multiplies. On a GPU, prefer one process with batching.
  • Timeouts and back-pressure. Bound the queue; when it is full, return HTTP 429 or 503 quickly instead of letting latency grow forever.
  • Warm-up. Run one dummy request at startup so the first real user does not pay for lazy initialisation.
  • Observability. Log request size, queue time, model time and status; export p50/p95/p99 latency. See troubleshooting and practice.
  • Containers. Package with a Dockerfile, pin library versions, and bake in or mount the model files. Never put tokens in the image.
flowchart LR
  C1["Client A"] --> Q["Request queue"]
  C2["Client B"] --> Q
  C3["Client C"] --> Q
  Q --> B["Batcher: max size / max wait"]
  B --> M["Model forward pass"]
  M --> S["Split results"]
  S --> C1
  S --> C2
  S --> C3

The batcher trades a few milliseconds of waiting for much better hardware use.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When should the model be loaded in a web service?

    Loading is slow and memory-hungry; requests should only run the forward pass.
  2. 2. What does dynamic batching trade?

    Requests wait briefly (max_wait) so they can share one efficient forward pass.
  3. 3. Why are plain def endpoints often safer than async def for a blocking model call?

    A blocking call inside async def freezes the event loop and every other request.
  4. 4. Eight requests of lengths 20, 20, 20, 20, 20, 20, 20 and 400 share one padded batch. What is the main problem?

    Every sequence is padded to 400, so most compute is wasted; bucketing by length helps.

Summary

  • Load the model once at startup, validate and truncate input, expose a health endpoint, and return stable JSON.
  • GPUs run small batches almost as fast as single items, so group requests with max_batch_size and max_wait.
  • Bucket by length (or use continuous batching for LLMs) to cut padding waste.
  • Bound queues, warm up at startup, and measure p95 latency, not only the average.

Next: Gradio demos and Hugging Face Spaces