Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท Operating Microservices

Observability: Structured Logs and Metrics

See inside a running system: correlation IDs, structured JSON logs, counters and histograms, and the RED and USE methods, with a runnable metrics collector.

In a monolith, when something breaks you open one log file. In a system of thirty services, one user click can touch six of them, each writing to its own log on its own machine. Without preparation, finding out what happened is detective work. Observability is the preparation: building services so that their internal state can be understood from the outside, using the data they emit.

The classic data types are logs (what happened, in detail), metrics (numbers over time, cheap and aggregated) and traces (the path of one request, covered in Distributed Tracing). This lesson covers logs and metrics, and shows how each is built.

The three pillars

Logs Detailed events. Great for "why".
Metrics Numbers over time. Great for "is it healthy".
Traces One request's path. Great for "where".

You usually start from a metric alert and dig down.

1 / 2

Correlation IDs

The single most useful habit is the correlation ID (also called request ID). The first service that receives a request, usually the gateway (API Gateway and Backend for Frontend), creates a unique ID. Every service then:

  1. reads the ID from an incoming header (commonly X-Correlation-ID or X-Request-ID),
  2. includes it in every log line it writes,
  3. passes it on in the headers of every outgoing call or message.

Afterward, searching your log system for one ID returns the whole story, in order, across every service. If a service receives a request with no ID, it generates one and starts a new story. Messages on a broker (Asynchronous Messaging and Events) should carry the ID in their metadata too.

Structured logs

A log line like Order 17 failed for user 8 after 3 tries is readable by humans and nearly useless to machines: to count failures per user you must write fragile regular expressions. Structured logging writes each event as a record of named fields, usually one JSON object per line.

A good log record has a timestamp, a level, the service name, a short constant message, the correlation ID, and any context as separate fields. Keep the message constant ("order rejected") and put variable data in fields ("sku": "SKU-9"): then you can group and count by message.

Choose levels with intent:

  • error: something failed and a human or retry logic needs to care.
  • warn: unexpected but handled, such as a fallback used.
  • info: normal business events, sparingly. Not every function call.
  • debug: detail for investigations, off in production by default.

Write logs to standard output and let the platform collect them; a service should not manage log files or shipping itself. Avoid logging secrets, tokens, card numbers and personal data (Configuration, Feature Flags and Secrets).

Logs across two services

The stand-in below puts an orders service and a stock service in one file. Calls are plain function calls and the clock is faked so the output is repeatable, but the logging pattern is exactly what real services do: the correlation ID arrives in a header, is added to every record, and is forwarded.

Node.js Correlation IDs and structured JSON logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first request arrived with the ID req-a1, so all four lines (two from orders, two from stock) share it. The second request had no header, so orders generated req-1004 and passed it to stock. Filtering for req-1004 shows the failure from start to finish: stock ran out of SKU-9, which orders turned into a 409. Because the records are JSON, a log tool can also answer questions like "how many out of stock errors per SKU today?" without parsing prose.

Centralized log collection

Each container writes its own logs and then disappears, so reading logs on individual machines does not work in microservices. Distributed logging means every service writes logs to standard output, a log shipper (an agent on each node, or a sidecar) collects them, and a pipeline delivers them to one searchable store. The usual stages are:

  1. Collect: agents such as Fluent Bit, Vector or Filebeat tail container output and add metadata (service, version, host, pod).
  2. Transport and buffer: a queue or broker absorbs bursts so a log spike does not drop events or slow the services.
  3. Process: parse JSON, drop noise, mask secrets and personal data, and add fields.
  4. Store and index: Elasticsearch/OpenSearch, Loki or a cloud logging service, with retention rules (for example 14 days hot, 90 days cheap archive).
  5. Search, dashboards and alerts: query by correlation ID, service and level; alert on error-rate patterns, in near real time.

Practices that keep this useful and affordable:

  • Standardize the format across all services (same field names for service, level, correlation_id, trace_id), ideally via a shared logging library, so one query works everywhere.
  • Never log secrets or personal data, and mask them before storage (see Securing the Edge and Data).
  • Scale the pipeline horizontally and sample or drop debug logs in production; logging volume is a major cost.
  • Never let logging break the request. Writes to the pipeline are asynchronous and failures are swallowed or counted.
  • Document and train. Teams should know the format, the dashboards and how to follow a request using the correlation ID, or the data stays unused.

Logs answer "what happened in this service", metrics answer "how often and how bad", and traces answer "where did the time go across services". Use all three together.

Metrics

Logs are detailed but expensive to store and slow to aggregate. Metrics are numbers sampled over time, cheap to keep for months and ideal for dashboards and alerts. The three basic types:

  • Counter. A value that only goes up (requests served, errors). You rarely care about the raw number; you care about its rate of change, such as requests per second.
  • Gauge. A value that goes up and down (queue length, memory in use, open connections).
  • Histogram. Counts of observations in ranges ("buckets"), typically used for durations. From the buckets you can estimate percentiles.

Why percentiles, not averages

Suppose 95 requests take 100 ms and five take 5 seconds. The average is about 345 ms, which describes nobody's experience. The 95th percentile (p95) says "95% of requests were at most this fast," and the p99 reveals the slow tail. In a system where one page calls many services, the tail dominates what users feel, so alert on p95 or p99 latency, not on the mean.

Beware of labels (also called tags or dimensions): route and status are fine, but a label such as userId creates one time series per user and can overwhelm a metrics system. Keep label values to small, bounded sets.

The RED and USE methods

Instead of inventing dashboards from scratch, use two checklists.

RED is for every request-handling service, from the caller's point of view:

  • Rate: requests per second.
  • Errors: how many of them fail.
  • Duration: how long they take (percentiles).

USE is for resources such as CPUs, disks, connection pools and queues:

  • Utilization: how busy is it?
  • Saturation: is work waiting because it is too busy (queue length)?
  • Errors: is it reporting faults?

Google's SRE book calls a related set the "four golden signals": latency, traffic, errors and saturation. Putting RED on every service and USE on every shared resource gives a dashboard that looks the same everywhere, which is invaluable at 3 a.m.

A metrics collector you can run

The stand-in below builds a counter and a histogram from scratch, feeds them 200 simulated requests (a seeded random generator keeps the output repeatable), and reports RED numbers. Real systems use a library such as Prometheus clients or OpenTelemetry for this, and a server scrapes or receives the data; the counting idea is the same.

Node.js Counters, histograms and RED numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read it as a RED panel. Rate is 200 requests (3.33 per second over the assumed minute). Errors are 6 requests, 3.0%. Duration is in the buckets: most requests fall in the 100 to 250 ms bucket, but a handful took 500 to 1000 ms or more. The mean (153 ms) looks fine, while p99 (at most 1000 ms) exposes the slow tail. Notice that the percentile is only as precise as the buckets: p50 and p95 both say "at most 250 ms" because the buckets are wide. Choose bucket bounds around the latency you care about, such as your timeout and your target.

From data to action

Collecting data is half the job.

  • Alert on symptoms, not causes. Page a human for "error rate above 2% for five minutes" or "p99 above 1 second," things users feel. CPU at 80% is a graph, not a page.
  • Define SLOs. A service level objective such as "99.9% of checkout requests succeed over 30 days" turns vague hopes into a budget you can spend on risky releases.
  • Link signals. A metric alert should lead to logs and traces for the same service and time window, using the correlation ID.
  • Watch the cost. Debug logs at high volume and high-cardinality metrics are the usual causes of surprise bills. Sample or drop what you never read.
  • Health endpoints. Expose /health (is the process alive?) and /ready (can it take traffic?), which orchestrators use (Containers, Docker Compose and Kubernetes).

What is real and what is the stand-in

JSON logs, correlation headers and the RED and USE methods are exactly what production teams use. In production the logs flow to a collector and a search system, and the metrics go to a time-series database with dashboards and alert rules; none of that can run on this site. The TryIts implement the core mechanics in memory: a logger that adds fields, a counter, and a bucketed histogram.

If you adopt only one thing, adopt correlation IDs plus JSON logs on day one. Retrofitting them into thirty services is far harder than starting with them.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a correlation ID for?

    Every service logs and forwards the same ID, so one search shows the request's whole journey.
  2. 2. Why prefer structured (JSON) logs to free-text lines?

    Named fields let you query, group and count events directly, and a constant message makes grouping easy.
  3. 3. Why alert on p99 latency rather than the average?

    A few very slow requests barely move the mean, but the high percentiles show them.
  4. 4. What do the letters in RED stand for?

    RED covers the request rate, the errors and the duration of requests for each service.
  5. 5. Which label is a bad idea on a metric?

    Each distinct label value creates a new time series, so unbounded values such as user ids can overwhelm the metrics system.

Summary

  • Observability lets you understand a running system from the data it emits: logs, metrics and traces.
  • Create a correlation ID at the edge, log it everywhere and forward it on every call and message.
  • Write structured JSON logs to standard output with a constant message and context in fields; never log secrets.
  • Counters, gauges and histograms are the metric types; use percentiles for latency and keep label values bounded.
  • Use RED for services and USE for resources, alert on symptoms, and define SLOs.

Next: Distributed Tracing.