Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Analysis and Reporting

Percentiles, Averages and Why Outliers Matter

Learn why averages mislead in performance testing, how to compute p50, p90, p95 and p99, how coordinated omission skews results, and how to set response-time goals.

Imagine a page with an average response time of 400 ms. Sounds fine. But if 95 percent of visits take 100 ms and the other 5 percent take 6 seconds, one visitor in twenty has a bad experience, and the average hides it completely. Response-time data is almost never symmetric, so the mean is a poor summary. Percentiles describe the distribution much better. This lesson explains them with runnable Python stand-ins, because JMeter itself cannot run on this site.

What a percentile is

The p-th percentile is the value below which p percent of observations fall. The median (p50) splits the data in half. p90 means "90 percent of requests were at least this fast". p99 describes the slowest 1 percent, the "tail".

Measure Tells you Weakness
Average total time divided by count; good for capacity maths hidden outliers; dominated by a few slow requests
Median (p50) the typical request ignores the tail
p90 / p95 what most users see on a bad day needs enough samples
p99 / p99.9 worst-case for almost everyone noisy with few samples
Max the single worst sample one-off, rarely actionable
Python Stand-in: average vs percentiles on skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average (around 400 ms) describes nobody: the typical request is about 100 ms and one in twenty takes 6 seconds. The median and p90 describe the main population, and p99 exposes the tail.

Why you cannot average percentiles

If agent A reports p95 = 300 ms and agent B reports p95 = 500 ms, the combined p95 is not 400 ms. Percentiles must be computed from the raw samples (or from mergeable histograms). This matters in distributed tests, where you should merge .jtl files and recompute, rather than average the per-agent numbers. The HTML dashboard computes percentiles from the merged file.

Likewise, do not average the averages of different labels, and do not add p90 values of sequential steps to get a p90 for the whole flow. Use a Transaction Controller to measure the whole flow as one sample.

How many samples do you need?

p99 from 100 samples is just the second-slowest request: meaningless. As a rule of thumb you want at least 100 samples above the percentile you care about: about 1,000 for p90, 10,000 for p99. For short tests, report p90 or p95 and mention the sample count.

Reading the shape of the distribution

A response-time histogram or percentile chart shows the shape:

  • One tight peak: healthy, consistent.
  • Two peaks (a fast group and a slow group): two code paths, a cache hit and miss, or a periodic stall such as garbage collection.
  • A long, thin tail: queueing, retries, locks, timeouts.
  • A hard wall at a round number (1,000 ms, 30,000 ms): a timeout configured somewhere.
Python Stand-in: a text histogram of response times

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two humps mean two populations. Fixing "the average" would be a mistake: you should investigate why 30 percent of requests miss the cache.

Coordinated omission

In a closed-loop test, a virtual user sends a request, waits for the response, thinks, then sends the next. If the server freezes for 10 seconds, that thread sends no requests during the freeze, so only one slow sample is recorded even though dozens of real users would have been affected. This under-reporting is called coordinated omission. Ways to reduce it:

  • Use an open model (arrival rate): the Concurrency Thread Group with a throughput timer, or the Precise Throughput Timer, keeps requests arriving at a fixed rate.
  • Analyse throughput drops and gaps in the time series, not only the distribution of the samples that exist.
  • Add timeouts to requests (connect and response timeouts) so stalls become errors.

Setting a goal

A good performance requirement uses a percentile and a load, never "fast":

Search returns in under 800 ms at p95 and under 2 s at p99, at 300 requests per second, with an error rate below 0.5 percent.

Check each requirement per label, over the steady state, with enough samples. The Apdex score is a second way to convert response times to a single satisfaction number.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. 95 percent of requests take 100 ms and 5 percent take 6,000 ms. Which statement is true?

    0.95 x 100 + 0.05 x 6000 = 395 ms, while the typical request is 100 ms. p99 would be about 6,000 ms.
  2. 2. Agent A has p95 = 300 ms and agent B p95 = 500 ms. What is the combined p95?

    Percentiles cannot be averaged; merge the data and recompute.
  3. 3. A response-time histogram with two separate humps suggests...

    Two modes usually mean two code paths or conditions behind the same request.
  4. 4. What causes coordinated omission?

    While a thread is blocked, the load it would have produced is simply not generated, which hides how bad a stall was.

Summary

  • Averages hide outliers; report median, p90, p95 and p99 with the sample count.
  • Percentiles cannot be averaged: merge raw data first.
  • Histograms reveal shapes such as two populations or timeout walls.
  • Closed-loop tests under-report stalls (coordinated omission); consider open-model timers.
  • State goals as a percentile at a defined load and error budget.

Next: Apdex and Error Analysis