Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 36 ยท Reliability and Operations

Observability and Tail Latency

Understand logs, metrics and traces, why averages hide pain, and how fan-out multiplies tail latency, with NumPy experiments you can run.

On one machine you can attach a debugger. In a distributed system, a single user request may touch twenty services on a dozen hosts, and by the time you look, the evidence is gone. Observability is the ability to understand what a system is doing from the data it emits. Without it, you cannot find out why requests are slow, which node is misbehaving, or whether your fix worked.

This lesson covers the three classic signals (logs, metrics and traces), then a topic that surprises almost everyone: tail latency. A service that is fast "on average" can still feel slow to many users, and the more services one request calls, the worse it gets. You will see the effect with NumPy experiments. The patterns that fight it are in Resilience Patterns.

Three signals

Logs Discrete events with details.
Metrics Cheap numbers over time.
Traces One request across many services.

Use metrics to notice, traces to locate, logs to explain.

1 / 2

Logs, metrics and traces

Logs are timestamped records of events: "payment failed for order 1234 because the bank timed out". They are rich but expensive at scale, and hard to search if they are free-form text. Emit structured logs (JSON with fields such as service, level, request ID and duration) so they can be filtered and counted.

Metrics are numbers sampled over time: requests per second, error rate, queue depth, CPU. They are small and cheap, perfect for dashboards and alerts. A useful starting set is the "golden signals": latency, traffic, errors and saturation (how full the service is). Beware of high-cardinality labels such as a user ID on every metric; each unique value creates a new time series.

Traces follow one request through the system. Each unit of work is a span with a start, an end and a parent. All spans share a trace ID that is passed along in request headers, so the pieces from different services can be joined. A trace answers "this checkout took 180 ms. Where did the time go?"

The sample trace below is plain data. The script computes each span's self time, the part not spent in child spans, and draws a tiny timeline.

Python Reading a trace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trace makes the answer obvious: almost all the time is in the external bank-api call, not in your own code. A metric would only have told you that checkout was slow. The structured log line at the end shows the same fact in a form a log system can index.

Why averages lie

Latency is not a bell curve. Most requests are quick, a few are very slow because of garbage collection pauses, cache misses, queuing, a retry or a noisy neighbor. The mean mixes both groups and describes nobody. Percentiles describe what people experience: the 50th percentile (p50, the median) is typical, p99 is the latency that 1 request in 100 exceeds.

Python Mean versus percentiles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mean of about 28 ms looks harmless, yet one request in a hundred takes over 400 ms. The last two lines show a classic trap: you cannot average percentiles. The mean of the per-window p99 values is far from the true p99. To combine data from many servers or minutes, merge histograms (counts per latency bucket) and compute the percentile from the merged histogram. Pick the thresholds in your alerts from percentiles, not means, and always say which percentile you mean.

Tail latency at scale

Now consider a request that must call many backends in parallel and wait for all of them, like a search that queries many shards. If one backend is slow on 1% of calls, the chance that at least one of N calls is slow is 1 - 0.99^N. The user sees the slowest reply, so the rare slow case becomes the common case. This is fan-out latency amplification, and it is why large services obsess over the tail.

Python Fan-out amplifies the tail

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the p50 column. With a single backend the median is about 10 ms, but with a fan-out of 100 the median request is already over 200 ms, because 63% of requests include at least one slow call. The simulated "slow" share matches the formula closely. A problem that looked like "1% of calls" is now "most users".

This is why Google engineers wrote about "tail at scale": at large fan-out you cannot fix tail latency by making the average faster. You reduce the tail itself.

Fighting the tail

Common techniques, several of which you have seen already:

  • Hedged requests: send a duplicate to another replica after a short delay and take the first reply.
  • Tied requests: send to two replicas at once and let them cancel each other when one starts.
  • Timeouts and partial results: return what you have after a deadline, marking missing shards.
  • Load shedding and bounded queues: queues are the main source of tail latency, so keep them short (Resilience Patterns).
  • Isolate and fix the causes: garbage collection pauses, background compaction, noisy neighbors and cold caches (Distributed Caching).
  • Smaller work units: many small shards balance better than a few large ones.

Alerting and service level objectives

Dashboards help you look; alerts make the system tell you. A service level indicator (SLI) is a measurement, such as "share of requests finished under 300 ms". A service level objective (SLO) is the target, for example "99% of requests under 300 ms over 30 days". The gap between the target and 100% is your error budget, which you can spend on releases and experiments. Alerting on symptoms that users feel (error rate, latency percentiles) is better than alerting on every CPU spike, and tracing makes it easy to go from an alert to a cause.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which signal best shows where time was spent inside one slow request across several services?

    Traces join spans from many services by a trace ID, showing the timing and parent-child structure of one request.
  2. 2. Why is averaging per-minute p99 values a bad way to get the overall p99?

    Percentiles do not compose by averaging. Merge the underlying histograms or raw data, then compute the percentile.
  3. 3. Each backend is slow on 1% of calls. A request waits for all 100 backends in parallel. About what share of requests see at least one slow call?

    1 - 0.99^100 is about 0.63.
  4. 4. What is an error budget?

    If the SLO is 99.9%, the remaining 0.1% is the budget for failures that you can spend on change.

Summary

  • Observability rests on logs (events), metrics (numbers over time) and traces (a request across services).
  • Use structured logs, golden signals (latency, traffic, errors, saturation) and trace IDs propagated through every call.
  • Averages hide the tail: report percentiles, never average them, and merge histograms instead.
  • Fan-out turns rare slow calls into common ones: the chance of a slow request is 1 - (1 - p)^N.
  • Reduce the tail with hedging, short queues, timeouts and by removing sources of delay; define SLOs and alert on user-visible symptoms.

Next: Testing Distributed Systems.