Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 41 ยท Serving and Monitoring

Latency, Throughput and Load

Measure model serving speed the right way: p50/p95/p99 latency, throughput, Little's law for capacity planning, and a queue simulation that shows why spikes hurt.

"Our model is fast" is not a measurement. Users feel the slowest requests, capacity bills grow with traffic, and a service that is fine at 10 requests per second can collapse at 100. This lesson gives you the vocabulary and the arithmetic to talk about speed honestly and to size a prediction service before it falls over.

The two numbers

  • Latency is how long one request takes, from arrival to answer (milliseconds).
  • Throughput is how many requests are completed per unit of time (requests per second, RPS).

They are related but different. Batching can raise throughput while making each individual request wait longer. A good service has a latency target (for example "95 percent of answers in under 100 ms") and a throughput target.

Speed vocabulary

Latency Time for one request.
Throughput Requests finished per second.
Concurrency Requests in flight at once.
Utilization How busy the workers are.

Latency explodes as utilization nears 100 percent.

1 / 2

Averages lie: use percentiles

Imagine 100 requests: 95 take 20 ms and 5 take 2000 ms (because of cold caches or garbage collection). The average is 119 ms and looks acceptable, but one user in twenty waits two seconds. Percentiles tell the truth:

  • p50 (median): half the requests are faster.
  • p95: 95 percent are faster; the slow tail starts here.
  • p99: the worst one in a hundred.
Python Measuring percentiles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the warm-up call that is not timed: the first call pays one-off setup costs that would distort the numbers. Also notice that the p99 is typically a multiple of the p50, even for a simple local model. Set targets on percentiles, never on the mean.

Little's law: how many workers do I need?

A simple and powerful relation connects the three quantities:

concurrency = arrival rate x average time in system

If 200 requests arrive per second and each takes 50 ms (0.05 s), then on average 200 x 0.05 = 10 requests are being processed at any moment. One single-threaded worker handles one at a time, so you need at least 10 workers just to keep up, and more to keep utilization comfortably below 100 percent.

Python Capacity planning with Little's law

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 60 percent utilization target is deliberate. The next section shows why.

Why utilization near 100 percent is dangerous

Requests do not arrive evenly. When a burst arrives faster than workers can finish, requests queue, and queue waiting time is added to latency. The closer the system runs to full capacity, the longer the queues, and the growth is not linear: it explodes.

The simulation below uses one worker, random arrivals and random service times, and reports the average wait for different loads.

Python Queue simulation: waiting explodes near capacity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table: moving from 50 to 70 percent busy costs a little, but moving from 85 to 95 percent costs a lot. This is why capacity plans leave headroom and why autoscaling adds workers when utilization or queue length crosses a threshold (for example 60 to 70 percent), rather than waiting until the service is saturated.

Ways to make a service faster

Pick the cheapest fix first.

  1. Do less work per request. Precompute features, cache repeated answers, drop unused features.
  2. Make the model cheaper. Smaller trees, distillation, quantization (lower-precision numbers), pruning, or a simpler model that is "good enough". The quality gate checks that accuracy stays acceptable.
  3. Use a faster runtime. Export to an optimized format and run it with an inference engine (ONNX Runtime, TensorRT, OpenVINO); these tools usually beat running the training framework as is.
  4. Batch dynamically. Collect requests for a few milliseconds and predict them together (micro-batching).
  5. Scale out. Add workers or replicas behind a load balancer.
  6. Use hardware suited to the model. GPUs help large neural networks, but for small tabular models a CPU is usually faster and cheaper.

Load testing

A load test sends realistic traffic to a service in a test environment and records latency percentiles and errors while ramping up. Tools such as Locust, k6, wrk and Vegeta exist for this. A read-only Locust sketch (it needs a live server, so it cannot run in the browser):

Python
# locustfile.py
from locust import HttpUser, task, between

class Caller(HttpUser):
    wait_time = between(0.1, 0.5)

    @task
    def predict(self):
        self.client.post("/v1/predict",
                         json={"age": 41, "income": 52000, "tenure_months": 18})

Good load tests:

  • use realistic inputs, because some inputs make some models slower;
  • ramp up gradually to find the point where p95 breaks the target;
  • run on hardware like production;
  • include a spike and a soak (hours at steady load, to catch memory leaks).

Common mistakes

  • Reporting the mean. Track p95 and p99.
  • Timing the first call. Warm up first.
  • Benchmarking the model alone. Real latency includes feature lookup, validation, serialization and the network.
  • Running at 90+ percent utilization. Queues make tail latency unpredictable.
  • Testing with one repeated input. Caches make that unrealistically fast.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are percentiles better than the mean for latency targets?

    A few very slow requests barely move the mean but are visible at p95 and p99.
  2. 2. 200 requests arrive per second and each takes 50 ms. How many requests are in flight on average (Little's law)?

    200 x 0.05 s = 10. You need at least that many workers, plus headroom.
  3. 3. What happens to waiting time as utilization approaches 100 percent?

    Queues build up when bursts arrive faster than workers finish, and the growth is non-linear.
  4. 4. Why exclude the first prediction call from timing?

    Warm-up cost would distort the steady-state numbers.

Summary

  • Track latency (per request) and throughput (requests per second) separately and set targets on percentiles.
  • Little's law (concurrency = rate x time) gives a first estimate of the number of workers.
  • Leave headroom: queueing delay explodes near full utilization.
  • Speed up by doing less, shrinking the model, using an optimized runtime, batching and scaling out.
  • Load test with realistic inputs, ramp-ups, spikes and long soaks.

Next: Safe Rollouts: Canary, Shadow and A/B