Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Capacity Planning and Bottleneck Hunting

Use stepped load test results to find the saturation point, estimate how many servers you need, apply the utilization law, and write a capacity recommendation.

After a load test you know how the system behaves at a few load levels. Capacity planning turns that into decisions: how many users can this setup serve, which resource runs out first, how many servers do we need for next year's traffic, and what do we fix first? This project teaches the method with a stepped test and a dataset you analyse yourself. JMeter cannot run on this site; the data below is invented for practice, and the Python code is a stand-in for the spreadsheet or script you would use on real results.

Run a stepped (staircase) test

Use a Stepping Thread Group or Concurrency Thread Group: start at 20 users, add 20 users every 5 minutes, up to well beyond the expected peak. For every step, record after a one-minute settle time:

  • throughput (requests per second),
  • p95 response time and error rate,
  • CPU, memory, disk and network on every tier (web, app, database),
  • queue indicators: database connection pool use, thread pool queue length, GC time.

Steps give you a table you can reason about, which a smooth ramp does not.

The Utilization Law

For any resource, utilization = throughput x service demand. If one request uses 8 ms of CPU on the application server, and the server has 4 cores (4,000 ms of CPU per second), the CPU saturates at 4000 / 8 = 500 requests per second. Measure the service demand per request:

TEXT
demand (ms of CPU per request) = CPU% x cores x 1000 / throughput (req/s)

Compute it for each resource and tier. The resource with the highest utilization at a given throughput is the bottleneck; it sets the ceiling for the whole system until you fix it. After removing it, the next resource becomes the bottleneck.

Python Stand-in: find the bottleneck from stepped results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table the way an analyst does. Service demand per request stays roughly constant (about 22 ms of application CPU), so CPU is not the problem. The database pool, however, climbs steadily and hits 100 percent exactly where throughput stops at about 120 requests per second and response times explode. The fix is not "buy more CPU": it is to raise the pool size (if the database can take it), shorten transactions, add an index, or cache a hot query. The next test will reveal the next bottleneck, probably the application CPU near 85 to 90 percent.

Choose a safe operating point

Do not run production at the knee. Common guidance:

  • Keep CPU below about 70 percent and every pool below about 80 percent at the planned peak.
  • Plan for N+1 redundancy: when one node fails, the others must still carry the peak.
  • Leave a headroom factor (20 to 30 percent) for traffic spikes and for measurement error.

In the example above, the knee is around 110 users (about 112 requests per second). A safe peak with 25 percent headroom is about 85 requests per second.

Forecast the servers you need

Python Stand-in: how many nodes for next year

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scaling efficiency line is the scalability test idea: repeat the test with more nodes. If four nodes give only 3.1 times the throughput of one, a shared resource (database, lock, cache, load balancer) limits you, and adding web servers alone will not help.

Bottleneck hunting checklist

Symptom in results Likely bottleneck Next check
Throughput flat, CPU low, response time rising pool, lock, downstream call, rate limit pool metrics, thread dumps, traces
CPU of the app at 90 to 100 percent compute-bound code profiler, hot methods, serialization
DB CPU high, slow queries missing indexes, inefficient queries slow-query log, EXPLAIN
Latency high, bandwidth at the limit network or payload size compression, caching, CDN
Memory climbing over time leak or unbounded cache heap dump, soak test
Periodic spikes in response time GC pauses, cron jobs, checkpoints GC logs, scheduler
Errors only above a given load limits: max connections, file descriptors, queue size server logs and config

Fix one bottleneck at a time and rerun the same stepped test; keep the old report for comparison. Document each change and its effect, so the tuning history is clear.

Write the recommendation

A good capacity note is a one-page decision document: current capacity (with units and the conditions), bottleneck, the evidence, recommended fix, expected gain, the safe operating point, and the plan for retesting after the change.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Application CPU is 40 percent at 100 req/s on 4 cores. What is the CPU demand per request?

    0.40 x 4 cores x 1000 ms / 100 req/s = 16 ms of CPU per request.
  2. 2. Throughput stops growing, but CPU is low and a connection pool is at 100 percent. What is the bottleneck?

    A saturated pool queues requests while CPUs idle.
  3. 3. Why plan N+1?

    With N+1 the remaining nodes can still carry the peak when one fails.
  4. 4. Four nodes give 3.1 times the throughput of one. What does that suggest?

    Efficiency below 100 percent suggests a shared bottleneck such as the database.

Summary

  • Use stepped tests and record both application metrics and resource utilization per step.
  • Utilization = throughput x service demand; the most utilized resource is the bottleneck.
  • Find the knee, then operate well below it with headroom and N+1.
  • Forecast nodes from capacity per node, growth and scaling efficiency.
  • Fix one bottleneck at a time and re-run the same test.

Next: Practice: Troubleshooting Test Plans