Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 65 ยท Reliability and Operations

Monitoring, Logging and Tracing

Learn observability for system design: metrics, structured logs, distributed tracing, SLIs, SLOs and error budgets, percentiles, alerting and the golden signals, with Python.

You cannot fix what you cannot see. In a system of dozens of services across hundreds of machines, "is it working?" and "why is it slow?" are real engineering problems, not things you answer by logging into a server. Observability is the ability to understand what is happening inside a system from the data it emits. This lesson covers the three signal types (metrics, logs, traces), how to turn them into targets (SLIs and SLOs) and how to alert without burning out your team.

The three pillars

Three kinds of signals

Metrics Numbers over time. Cheap and good for alerts.
Logs Events with details. Good for finding why.
Traces One request across services. Good for where.

Metrics say something is wrong, traces say where, logs say why.

1 / 2

Metrics are numeric measurements aggregated over time: requests per second, error rate, CPU, queue depth, latency percentiles. They are small and cheap to store, so you can keep them for months, graph them and alert on them. Tools: Prometheus, Datadog, CloudWatch.

Logs are timestamped records of discrete events: "payment failed for order 42: card declined". They carry the detail that metrics lack but are bulky. Make them structured (JSON with fields) rather than free text, so they can be searched and aggregated. In a distributed system, ship them to a central store (Elasticsearch/OpenSearch, Loki, Splunk) because the machine that logged an error may be gone. Include a correlation ID in every line.

Traces follow a single request through every service it touches. Each unit of work is a span with a start, duration and parent; spans form a tree. A trace shows that the 1.2-second checkout spent 900 ms waiting for the inventory service.

Metrics worth having: the golden signals

Google's SRE book names four signals that cover most services:

Signal Question Example
Latency How long do requests take? p50, p95, p99 response time
Traffic How much demand? Requests per second
Errors How many fail? 5xx rate, failed jobs
Saturation How full is the system? CPU, memory, queue depth, connection pool use

For resources use the USE method (Utilization, Saturation, Errors); for request-driven services use RED (Rate, Errors, Duration).

Percentiles, not averages

Averages hide pain. If 99 requests take 100 ms and one takes 10 seconds, the average is about 200 ms, which looks fine while one user in a hundred waits ten seconds. Use percentiles: p50 is the median, p99 means 99 percent of requests were faster than this value. Because a page often makes many backend calls, the tail latency of the slowest call dominates what users feel.

Python Why the average lies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is the tail at scale effect: with 20 parallel calls, roughly one page in five experiences a "1 percent" slow call.

SLIs, SLOs and error budgets

Three terms keep reliability discussions concrete:

  • SLI (service level indicator): a measured quantity, such as the fraction of requests that succeed in under 300 ms.
  • SLO (service level objective): the target for the SLI, such as 99.9% over 30 days.
  • SLA (service level agreement): a contractual promise with penalties, usually looser than the internal SLO.

The gap between 100% and the SLO is the error budget: a 99.9% SLO allows 0.1% failures, about 43 minutes a month. While budget remains, teams ship features; when it is spent, they prioritize reliability work. That turns "how reliable should we be?" into a measurable trade-off (see High Availability).

Python An error budget calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tracing across services

To trace a request you propagate a trace ID (and span IDs) in headers such as the W3C traceparent. Each service creates spans and reports them to a collector; the tooling (OpenTelemetry for instrumentation, Jaeger, Tempo or Zipkin for storage) stitches them into a tree. Because tracing everything is expensive, systems sample: keep all error traces and a small percentage of normal ones.

Python Reading a trace as a span tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The picture makes the answer obvious: the call to the bank API inside payments dominates the request.

Alerting

Good alerts are rare and actionable. Principles:

  • Alert on symptoms users feel (error rate, latency SLO burn), not every cause (CPU at 80%). Use causes as dashboards and debugging aids.
  • Every alert needs an owner and a runbook. If nobody knows what to do, it is noise.
  • Page for urgent, ticket for slow. Wake someone only if the issue needs action now.
  • Use burn-rate alerts on the error budget (fast burn and slow burn) to catch both sudden outages and slow degradations.
  • Fight alert fatigue: review noisy alerts weekly and delete or fix them.
  • Add health checks at several levels: liveness (is the process running?), readiness (can it serve traffic?) and synthetic probes that exercise real user journeys.

Dashboards and incident response

Keep a few focused dashboards: a service overview with the golden signals, a dependency view and a business view (orders per minute). During incidents, stay organized: declare severity, assign an incident commander, communicate status, mitigate first (roll back, shift traffic) and investigate later. Afterward write a blameless postmortem that records the timeline, the contributing factors and concrete follow-ups.

Log what you will need at 3 a.m.: request ID, user or tenant ID (never secrets, tokens or full card numbers), operation, duration and outcome. Keep log levels honest: ERROR means someone should look at it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which signal best finds WHERE time is spent across several services for one request?

    A trace follows one request across services and shows each span's duration.
  2. 2. Why prefer p99 latency over the average?

    A small fraction of very slow requests barely moves the mean but is felt by many users, especially with fan-out.
  3. 3. A 99.9% availability SLO for 30 days allows roughly how much downtime?

    0.1% of 30 days is about 43 minutes. That allowance is the error budget.
  4. 4. Which is the best alert?

    Symptom-based alerts on SLO burn are actionable and user-focused. Cause-level thresholds create noise.

Summary

  • Observability rests on metrics (what), traces (where) and structured logs (why), tied together by correlation IDs.
  • Watch the golden signals: latency, traffic, errors, saturation; use percentiles, not averages.
  • SLIs measure, SLOs set targets and error budgets balance new features against reliability.
  • Propagate trace IDs, sample sensibly and read traces as span trees.
  • Alert on user-visible symptoms with runbooks; run blameless postmortems.

Next: Deployments and CI/CD