Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 119 ยท Testing

Load Testing Java Services

Learn load testing for Java services - throughput, latency percentiles, open and closed workloads, JMeter and Gatling basics, and how to find the real bottleneck.

Unit tests tell you that a service gives the right answer once. They say nothing about what happens when 2,000 people use it at the same moment. Under load, code that is perfectly correct can become slow, run out of database connections, exhaust its threads or fall over completely. Load testing (also called performance testing) answers questions such as: how many requests per second can this service handle, how does response time change as traffic grows, and which resource gives out first?

This lesson explains the vocabulary and the shape of a good test, shows a miniature load generator you can run, uses a queue simulation to explain why latency explodes near capacity, introduces Apache JMeter and Gatling, and ends with a method for finding bottlenecks. It builds on Integration Testing and on the HTTP client from HTTP Client in Depth.

What to measure

Four numbers describe most performance results:

Metric Meaning
Throughput Requests completed per second (RPS or TPS)
Latency / response time How long one request takes, measured by the caller
Error rate Share of requests that fail (5xx, timeouts, refused connections)
Saturation How full the busiest resource is: CPU, threads, connection pool, memory

Averages hide problems. If 99 requests take 50 ms and one takes 5 seconds, the average is about 100 ms and looks fine, yet one user in a hundred waits five seconds, and a page that makes ten calls hits that slow one almost every time. Report percentiles: p50 (the median), p95, p99. "p99 = 800 ms" means that 99% of requests were faster than 800 ms. Service-level objectives are usually written this way: "p95 under 300 ms and fewer than 0.1% errors".

A load generator in forty lines

Every load tool does the same thing: run many virtual users in parallel, each sending requests and recording how long they take. The program below starts a small HTTP server on the loopback interface (four worker threads, 20 ms of work per request) and hits it with 8 virtual users that send 25 requests each. The exact timings change from run to run, so it prints only facts that always hold.

Java A tiny load generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice the shape: the server has a fixed number of threads (a resource), the clients are concurrent (the load), and each measurement is taken on the client side around the whole call. A real tool adds ramp-up, think time, reports and distributed generators, but the core is this loop.

Open and closed workloads

There are two ways to model users, and they behave very differently when the system slows down.

  • Closed workload: a fixed number of users, each sending the next request only after the previous reply (like the program above). When the server slows, users slow down too, so load falls automatically and the test can hide a collapse. JMeter's thread groups work this way by default.
  • Open workload: new users arrive at a given rate (say 50 per second) no matter how the system is doing, which is how real internet traffic behaves. If the system cannot keep up, queues build, latency climbs and errors appear. Gatling's injectOpen models this.

For a public website or API, prefer open arrival rates; use closed models for internal batch clients and connection-limited callers.

Why latency explodes near capacity

Suppose your service has 4 worker threads and each request needs about 40 ms. It can finish 4 / 0.040 = 100 requests per second. At 50 per second it is half busy and requests rarely wait. At 95 per second it is 95% busy, and because arrivals are random, bursts regularly find all workers occupied, so requests queue. Latency does not grow gently with load: it grows slowly, then suddenly. The simulation below uses a fixed random seed so the results are reproducible. It models 4 workers, random service times averaging 40 ms and random arrivals.

Java The latency hockey stick

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Even at 40% busy, p95 is already several times the average work time because of random bursts. From 90% to 97% busy the median doubles while p95 and p99 triple. That is why load tests look at tail percentiles, and why production services are usually run at 50 to 70% of their measured capacity. Real services are messier (garbage collection pauses, lock contention, cache misses), but the curve has the same shape.

Apache JMeter

JMeter is a Java desktop and command-line tool configured through a GUI and saved as a .jmx file. A test plan contains:

  • a Thread Group: number of users, ramp-up period and loop count;
  • Samplers: for example an HTTP Request to GET /api/products;
  • Config elements: HTTP defaults, CSV data (different users and ids for each virtual user), header managers;
  • Assertions: fail a sample if the status is not 200 or the body lacks an expected text;
  • Listeners: results tables and graphs.

Build and debug the plan in the GUI with a handful of users, but run the real test from the command line, because the GUI itself consumes CPU and distorts results:

Terminal
jmeter -n -t shop-plan.jmx -l results.jtl -e -o report/

-n is non-GUI mode, -t the plan, -l the raw results, and -e -o generate an HTML report with throughput, percentile tables and error breakdowns.

Gatling

Gatling describes scenarios as code, which fits version control and code review well. Its Java DSL (also available for Kotlin and Scala) reads like the user journey it simulates:

Java
import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;

import io.gatling.javaapi.core.*;
import io.gatling.javaapi.http.*;

public class ShopSimulation extends Simulation {

    HttpProtocolBuilder protocol = http
            .baseUrl("http://localhost:8080")
            .acceptHeader("application/json");

    ScenarioBuilder browse = scenario("Browse the catalog")
            .exec(http("list products").get("/api/products").check(status().is(200)))
            .pause(1, 3)
            .exec(http("one product").get("/api/products/42").check(status().is(200)));

    {
        setUp(browse.injectOpen(rampUsers(200).during(60)))
                .protocols(protocol)
                .assertions(
                        global().responseTime().percentile(95).lt(500),
                        global().failedRequests().percent().lt(1.0));
    }
}

This ramps up 200 new users over a minute (an open workload), makes each one browse the catalog with a random pause of one to three seconds, and fails the build if p95 exceeds 500 ms or more than 1% of requests fail. Because the assertions are part of the simulation, it can run in CI as a performance gate. Simpler tools exist too (wrk, k6, hey), and are handy for a quick check of a single endpoint.

Running a trustworthy test

Most bad performance conclusions come from bad test setups, not bad code. Follow these rules:

  • Test a production-like environment. A laptop with an in-memory database proves little. Match the instance size, the database engine and the data volume (a table with 100 rows behaves differently from one with 50 million).
  • Do not run the generator on the same machine as the service, or they compete for CPU. Make sure the generator itself is not the bottleneck.
  • Warm up. The JVM compiles hot code with the JIT compiler and fills caches, so the first minute is slower. Discard it from the results.
  • Use realistic data and paths: varied ids, a realistic read/write mix, and think time. Hitting one cached record repeatedly measures the cache.
  • Change one thing at a time, repeat runs, and record the numbers. A single run is an anecdote.
  • Never point a load test at a system you don't own or a shared production environment without agreement.

Finding the bottleneck

A load test tells you that something is slow; observability tells you why. Run the test while watching the application's metrics from Actuator and Observability and the database.

flowchart TD
    A["Run load test"] --> B{"Meets target?"}
    B -->|yes| Z["Record baseline, done"]
    B -->|no| C["Find the saturated resource"]
    C --> D["CPU, threads, DB pool, DB queries, GC, network"]
    D --> E["Fix the biggest one"]
    E --> A

Performance work is a loop: measure, fix the largest bottleneck, measure again.

Typical findings in Spring applications:

  • Connection pool exhausted: threads wait for a database connection, so latency rises while CPU stays low. Check the pool metrics (hikaricp.connections.pending) and see Connection Pooling.
  • N+1 queries: one request fires hundreds of SQL statements. The fix is a fetch join or batch fetching, as in Fetching and Performance.
  • Slow or missing index: one query dominates the database time; examine its execution plan.
  • Thread pool saturation: all Tomcat worker threads are blocked on slow downstream calls. Add timeouts and circuit breakers (Circuit Breakers and Resilience), or consider virtual threads.
  • Garbage collection pressure: frequent long pauses from excessive allocation or a heap that is too small. Use JDK Flight Recorder and the tools from Debugging and Profiling Tools.
  • Missing caching of data that rarely changes (Scheduling, Caching and Async).

Always fix the largest bottleneck first; after you remove it, the next one becomes visible, so rerun the test.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A service has an average response time of 100 ms, but users complain about slowness. Which measurement is most likely to reveal the problem?

    Averages hide slow outliers. Tail percentiles such as p95 and p99 show what the unluckiest users experience.
  2. 2. Which workload model keeps sending new requests at a fixed arrival rate even when the system is slowing down?

    In an open workload, arrivals are independent of the responses, so queues and errors appear as soon as capacity is exceeded. Closed workloads slow down with the server and can hide the collapse.
  3. 3. Why should you run a JMeter test from the command line instead of the GUI?

    Use the GUI to build and debug the plan with few users. Real runs use non-GUI mode (-n) so the tool does not compete for resources.
  4. 4. During a test, latency is high but CPU on the application server is low and the database connection pool shows many pending requests. What is the most likely bottleneck?

    Threads are waiting for connections, not computing. Look at pool size, slow queries that hold connections too long, and transactions that stay open too long.
  5. 5. Why does latency rise sharply when a service gets close to 100% utilization?

    Even below nominal capacity, random arrivals form queues. The closer to 100%, the more often and the longer they wait.

Summary

  • Load tests measure throughput, latency percentiles, error rate and resource saturation; judge results by p95/p99, not averages.
  • Every tool runs many virtual users in parallel and records latency on the client side. Closed workloads fix the number of users; open workloads fix the arrival rate and reveal overload honestly.
  • Latency rises slowly and then steeply as utilization nears 100%, so run production services with headroom.
  • JMeter uses thread groups, samplers, assertions and listeners (run non-GUI); Gatling expresses scenarios as code with assertions that can gate a CI build.
  • Test a production-like environment, warm up the JVM, use realistic data, separate the generator from the system, and change one variable at a time.
  • Find bottlenecks with metrics, profiling and database plans (pool exhaustion, N+1 queries, missing indexes, saturated threads, GC), fix the biggest, and measure again.

Next: Logging with SLF4J and Logback.