Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 27 ยท Monitoring Fundamentals

RED, USE and the Golden Signals

Choose what to measure: the RED method, the USE method, the four golden signals, plus SLIs, SLOs and error budgets, with runnable examples.

A real service can emit thousands of different numbers. Staring at all of them is not monitoring. The skill is choosing a small set that tells you whether users are happy and where to look when they are not. Three well-known checklists do this for you: the RED method, the USE method and Google's four golden signals.

After the checklists, this lesson covers how to turn a signal into a promise you can measure: SLIs, SLOs and error budgets.

RED for services

Rate Requests per second.
Errors Requests that fail.
Duration How long requests take.

Measured from the caller's point of view.

1 / 3

The RED method

RED is for anything that handles requests: a web service, an API, a queue consumer. For each service, measure:

  • Rate: how many requests per second arrive.
  • Errors: how many of those requests fail, as a count or a percentage.
  • Duration: how long requests take, as a distribution, not a single average.

RED works because it is the caller's view. A caller does not care about your CPU. It cares whether its request was answered, correctly, and fast. Put the same three panels on every service and anyone on call can read any dashboard.

Always report duration as percentiles: p50 (the median), p95 and p99. A mean hides the slow tail. If 5% of checkout requests take over a second, the mean can still look healthy while those customers leave.

The USE method

USE is for resources, the things your services consume: CPUs, memory, disks, network links, connection pools, thread pools, queues. For each resource, check:

  • Utilization: the fraction of time or capacity in use (a disk 70% full, a CPU 85% busy).
  • Saturation: how much extra work is waiting because the resource is already busy (run-queue length, pending database connections).
  • Errors: faults the resource reports (disk I/O errors, network packet drops).

Saturation is the one people forget, and it often explains slowness: a connection pool can sit at 100% utilization with an empty queue and be fine, but once requests start queuing for a connection, latency climbs for everyone.

RED tells you a service is hurting, and USE helps you find which resource is responsible. They are partners.

The four golden signals

The Google SRE book popularized four signals for user-facing systems:

  • Latency: the time to serve requests. Track successful and failed requests separately, because a fast error can make latency look better than it is.
  • Traffic: the demand on the system, such as requests per second.
  • Errors: the rate of failed requests, including requests that return success codes but wrong content.
  • Saturation: how full the most constrained resource is.

Compare the lists: traffic is RED's rate, errors are the same, latency is RED's duration, and saturation is the USE idea added in. The golden signals are essentially RED plus saturation, so pick whichever vocabulary your team likes. What matters is that every service has all of these covered.

Computing RED from request records

Prometheus would store counters and histograms and answer these questions with PromQL later in the tutorial. As a stand-in, the code below takes 300 request records from a 60-second window and calculates rate, errors and duration per route. The random generator has a fixed seed so you get the same numbers each time.

JS RED numbers from request records

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Both routes have a similar median (about 130 ms), so a single average would make them look alike. The RED breakdown shows the difference: /checkout has a 6.1% error rate against 1.0% for /products, and its p95 is 897 ms against 219 ms. Splitting by route (a label in Prometheus terms, see Metrics, Labels and Time Series) is what makes the problem findable.

USE in practice

Typical USE checks for a Linux server and an application:

Resource Utilization Saturation Errors
CPU busy percentage run-queue length rarely any
Memory used fraction swapping, OOM kills allocation failures
Disk space used, I/O busy time I/O queue length I/O errors
Connection pool connections in use / size requests waiting for a connection connection failures

You do not need all cells on day one. Start with the ones that have caused outages before.

SLIs, SLOs and error budgets

Metrics become much more useful once you decide what "good enough" means.

  • A service level indicator (SLI) is a measurement of user-visible quality, usually a ratio: good requests / all requests. Examples: the fraction of requests that succeed, or the fraction that finish within 300 ms.
  • A service level objective (SLO) is a target for an SLI over a time window: "99.5% of requests succeed over 30 days".
  • The error budget is what the SLO allows you to get wrong: 100% minus the target. With 2,000,000 requests in 30 days and a 99.5% SLO, 10,000 requests may fail.

An error budget turns "is the service reliable enough?" into an arithmetic question. While budget remains, you can ship features and take risks. When it is nearly gone, you slow down and fix reliability. An SLO of 100% is the wrong goal: it is unreachable and leaves no room to change anything. The SLI queries come in Quantiles, Histograms and SLI Queries, and the alerts that protect the budget in Project: Alerts with Burn Rates and Routing.

Watching a budget burn

The stand-in tracks failed requests across a month against the budget from the example above.

JS Spending an error budget

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The incident on day 15 used 41% of the whole month's budget at once, and by day 25 the team is over budget. A team following this policy would freeze risky releases at that point and spend its time on reliability.

Choosing what to measure

  • Start from users: what would make them say "it is broken"? That is your SLI.
  • Add RED to every service and USE to every shared resource.
  • Prefer a few good signals over hundreds of metrics.
  • Label by dimensions you will actually filter on, such as route, status code and service, and keep the number of distinct values small.
  • Measure at the edge too: a load balancer or a synthetic probe sees what users see, even when the service itself thinks it is fine.

What is real and what is the stand-in

RED, USE, golden signals, SLIs and SLOs are real practices used by most large engineering teams. In real life the numbers come from Prometheus counters and histograms queried with PromQL, not from an in-memory array of records. The TryIts reproduce the arithmetic on a small dataset so you can see what the queries will compute.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the E in RED stand for?

    RED is Rate, Errors, Duration, the three things to measure for every request-handling service.
  2. 2. Which USE metric shows that work is waiting because a resource is already busy?

    Saturation measures queued or waiting work, such as a run queue or requests waiting for a database connection.
  3. 3. A service has a 99.9% SLO over 30 days. What is its error budget?

    The budget is 100% minus the target, so 0.1% of requests may fail during the window.
  4. 4. Which signal does the set of golden signals add compared with RED?

    Traffic, errors and latency correspond to RED's rate, errors and duration. Saturation is the extra signal.
  5. 5. Why report latency as p95 or p99 rather than only the mean?

    A few very slow requests barely move the mean but show clearly in high percentiles.

Summary

  • RED (rate, errors, duration) describes services, and USE (utilization, saturation, errors) describes resources.
  • The four golden signals are latency, traffic, errors and saturation: roughly RED plus saturation.
  • Report latency as percentiles, and split signals by a few useful labels such as route.
  • An SLI measures quality, an SLO sets the target, and the error budget is what the target lets you get wrong.
  • Choose a small set of signals that reflect what users feel.

Next: How Prometheus Works.