Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 65 ยท Thinking in Systems

Latency, Throughput and Bottlenecks

Understand latency, throughput, bandwidth and queueing, learn Little's law, and find the bottleneck of a request pipeline with a Python simulation.

Two numbers describe how a system performs. Latency is how long one request takes from start to finish. Throughput is how many requests (or bytes) the system completes per unit of time. They are related but not the same, and optimizing one can hurt the other.

An analogy: a highway. Latency is how long your drive takes. Throughput is how many cars pass a point each minute. Adding lanes raises throughput without making any single car faster. Raising the speed limit lowers latency. Every system design conversation about "fast" should say which one it means.

Latency

Latency is measured per request, in milliseconds, and it is the sum of everything that request waits for:

  • Network time: the round trip between client and server, which depends on distance and the number of hops.
  • Queueing time: waiting for a free worker, connection or lock.
  • Processing time: CPU work, serializing data, running code.
  • Storage time: reading a cache, a database or a disk.
  • Downstream calls: other services your service depends on.

Because latency varies from request to request, we describe it with percentiles (p50, p95, p99), as in the requirements lesson. The tail matters: when a page needs 20 backend calls, the chance that at least one hits a p99 delay is about 18%.

Throughput and bandwidth

Throughput is completed work per second: requests per second (RPS or QPS), transactions per second, messages per second, or megabytes per second. Bandwidth is the maximum capacity of a link, such as 10 Gbit/s. Throughput can never exceed the bandwidth or capacity of the narrowest part of the path.

Latency and throughput trade off through batching: sending 100 records in one request is more efficient (higher throughput) but each record waits for the batch to fill (higher latency). Streaming systems, databases and message queues all expose this knob.

Little's law: connecting the two

For any stable system, a simple formula ties these quantities together:

L = ฮป x W

where L is the average number of requests inside the system, ฮป (lambda) is the arrival rate in requests per second, and W is the average time each request spends inside (its latency). If requests arrive at 200 per second and each takes 0.05 seconds, then on average 200 x 0.05 = 10 requests are in flight at once. That tells you how many worker threads or connections you need: at least 10, and more for headroom.

Python Little's law in practice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The slow case (500 ms) needs ten times more concurrent workers than the fast one at the same traffic. That is why a slow dependency can exhaust a thread pool: when latency grows, concurrency grows with it, until you run out and requests queue.

Queues explain why "almost full" is dangerous

When a server is lightly loaded, requests rarely wait. As utilization (the fraction of capacity in use) approaches 100%, waiting time does not grow slowly; it shoots up. For a simple single-server queue, the average time in system is about the service time divided by (1 minus utilization). At 50% utilization the wait is about twice the service time; at 90% it is ten times; at 99% a hundred times.

Python Latency explodes near full utilization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the strongest argument for headroom: run servers at 50 to 70% utilization at peak, not 95%. The last few percent of capacity are not usable capacity; they are where latency collapses.

Finding the bottleneck

The bottleneck is the part of the pipeline with the lowest capacity. The whole system's throughput equals the throughput of its bottleneck, and improving any other stage changes nothing.

flowchart LR
  A["Load balancer<br/>50,000 rps"] --> B["Web tier<br/>8,000 rps"]
  B --> C["Cache<br/>100,000 rps"]
  B --> D[("Database<br/>1,200 rps")]

Figure: the database limits cache misses to 1,200 rps; the web tier limits everything to 8,000 rps.

Common bottlenecks, with typical cures:

Bottleneck Symptom Typical cure
CPU High CPU, slow computation More servers, faster algorithm, caching results
Memory Swapping, out-of-memory kills Bigger instances, smaller data structures, evict caches
Disk I/O High wait times, slow queries SSDs, indexes, caching, sequential access
Network Saturated links, high transfer times Compression, CDN, regional deployment
Database Slow queries, lock waits, connection limits Indexes, replicas, caching, sharding
Locks Threads waiting on one resource Smaller critical sections, partitioning
Slow dependency Latency tracks another service Timeouts, caching, circuit breakers, async work

Method: measure, do not guess

  1. Instrument every stage (timing, queue depth, error rate).
  2. Find the stage with the highest utilization or the longest wait.
  3. Fix that stage, then measure again: the bottleneck moves to the next weakest part.
  4. Repeat until the requirement is met. Stop when it is met; do not optimize further.

Here is a pipeline simulation that reports each stage's utilization at a given offered load and which stage saturates first. Run it, then raise capacity in the weakest stage and see the bottleneck move.

Python Which stage is the bottleneck?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At 5,000 rps the web tier is the busiest stage at 62%. At 12,000 rps it is over 100%, and the database sits exactly at its limit even though the 90% cache hit rate shields it from nine of every ten requests. Try setting CACHE_HIT_RATE to 0.5 and the database becomes the bottleneck at a much lower load. That is exactly the argument for caching.

Optimization techniques at a glance

  • Cache results so work is done once.
  • Add indexes so the database reads less.
  • Compress payloads and use efficient formats.
  • Batch small operations; pipeline requests.
  • Parallelize independent work.
  • Move work off the request path to background queues.
  • Bring data closer to users with a CDN.
  • Reduce round trips: one request that returns what the page needs, not ten.

Always weigh each against its cost: a cache adds an invalidation problem, a queue adds delay and complexity.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between latency and throughput?

    One describes how long a single request takes, the other how much the system finishes per second.
  2. 2. Requests arrive at 400 per second and each takes 0.25 seconds. How many are in flight on average (Little's law)?

    L = 400 x 0.25 = 100 concurrent requests.
  3. 3. Why is running a server at 95% utilization risky?

    Waiting time scales roughly with 1/(1 - utilization), so the last few percent are very expensive.
  4. 4. After you fix the current bottleneck, what usually happens?

    Throughput is set by the slowest stage, so the next slowest becomes the limit.

Summary

  • Latency is time per request; throughput is work completed per second; batching trades one for the other.
  • Little's law: requests in flight = arrival rate x time in system.
  • Latency rises sharply as utilization nears 100%, so keep headroom.
  • The bottleneck is the stage with the lowest capacity; measure, fix, remeasure.
  • Typical cures: cache, index, compress, batch, parallelize, go async, move data closer to users.

Next: Availability, Reliability and Maintainability