Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 41 ยท Serving and Monitoring

Building a Prediction Service

Design a model prediction API: typed request validation with pydantic, health and readiness checks, versioned responses, error handling, and a FastAPI sketch.

An online model is usually wrapped in a small web service: a client sends JSON, the service returns JSON. The model is the easy part. The service around it needs to validate input, report its health, say which model produced an answer, and fail in a way callers can understand.

This lesson shows the pieces. A real HTTP server cannot run in the browser, so the complete web layer (FastAPI) appears as read-only code, while the logic that matters (validation, prediction, error handling, health) is a set of plain Python functions you can run here. This is a good design anyway: keep the web framework thin and the logic testable.

The contract

Decide the request and response shape first, because other teams will depend on it.

JSON
POST /v1/predict
{ "age": 41, "income": 52000, "tenure_months": 18 }

200 OK
{ "score": 0.27, "label": "stay", "model_version": "churn-v7", "request_id": "b1f4..." }

Rules that save pain later:

  • Version the URL (/v1/predict). Breaking changes go to /v2.
  • Return the model version in every response. You will need it when someone asks why a prediction was odd.
  • Return a request id so a single call can be found in the logs.
  • Use clear status codes: 200 success, 422 invalid input, 503 model not ready, 500 unexpected error.

Validate input with pydantic

pydantic declares types and limits once and rejects bad payloads with readable messages. It works in this runner, so you can see real validation errors.

Python Typed request validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Validation at the edge protects the model. A model given an age of -5 or a string will not raise a helpful error; it may return a plausible number computed from garbage.

The handler as a plain function

Keep the logic in a function that takes a dict and returns (status_code, body). A framework then only has to connect HTTP to this function, and your tests can call it directly.

Python Prediction handler with errors and health

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two endpoints with different jobs appear above:

  • Liveness (/health) answers "is the process running?". Orchestrators restart containers that fail it.
  • Readiness (/ready) answers "can it take traffic?". It returns 503 until the model has loaded, so a load balancer does not send requests to a half-started instance.

The web layer with FastAPI

The framework code is short because the logic already exists. This is read-only here (no web server can run in the browser):

Python
# app.py  (run with: uvicorn app:app --port 8000)
from fastapi import FastAPI, HTTPException
import joblib
from service import PredictRequest, Service

app = FastAPI(title="Churn model")
svc = Service()

@app.on_event("startup")
def startup():
    svc.load(version="churn-v7")        # load once, not per request

@app.get("/health")
def health():
    return svc.health()[1]

@app.get("/ready")
def ready():
    code, body = svc.ready()
    if code != 200:
        raise HTTPException(status_code=code, detail=body)
    return body

@app.post("/v1/predict")
def predict(req: PredictRequest):       # FastAPI validates with pydantic for you
    code, body = svc.predict(req.model_dump())
    if code != 200:
        raise HTTPException(status_code=code, detail=body)
    return body

Run the service in a container with the model file baked in or mounted, and a command such as uvicorn app:app --host 0.0.0.0 --port 8000. Packaging is covered in Model Packaging and Registry.

Many ML-specific servers exist (BentoML, KServe, TorchServe, TensorFlow Serving, NVIDIA Triton). They give you batching, versioning and metrics out of the box. The design points in this lesson apply to all of them. See the tooling landscape.

Operational details that matter

  • Timeouts: set a time limit on feature lookups and the model call, and return a fast failure instead of hanging.
  • Concurrency: a CPU-heavy model blocks a single worker. Run several worker processes, or move prediction to a thread pool.
  • Input size limits: reject enormous payloads before parsing.
  • Logging: log request id, model version, latency and (where privacy allows) inputs and outputs. These logs feed monitoring and drift detection.
  • Secrets: never put credentials in the image; read them from the environment.

Common mistakes

  • No validation: garbage in, confident garbage out.
  • Loading the model inside the request handler.
  • Returning 200 with an error message in the body. Use proper status codes so monitoring and clients can react.
  • One endpoint for liveness and readiness. A slow model load will then cause restart loops.
  • Unversioned API. Changing a field name breaks every caller.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which status code best fits a request with an invalid age value?

    422 Unprocessable Entity (or 400) signals that the input is invalid, and the caller should fix it.
  2. 2. What does a readiness endpoint report?

    Readiness gates traffic; liveness only tells that the process is alive.
  3. 3. Why include model_version in every response?

    Traceability: you can match behavior to a specific registered model.
  4. 4. Why keep prediction logic in a plain function and not inside the route function?

    A thin web layer over testable logic is easier to test and to move between frameworks.

Summary

  • Define a versioned API contract that returns the score, the model version and a request id.
  • Validate every input at the edge (pydantic) and use meaningful status codes.
  • Provide separate liveness and readiness endpoints; load the model once at startup.
  • Keep logic in plain functions and the web layer thin.
  • Add timeouts, size limits, concurrency and logging for production.

Next: Latency, Throughput and Load