Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 41 ยท Serving and Monitoring

Monitoring Service Health

Monitor a deployed model like any production service: request rate, errors, latency histograms, Prometheus-style metrics, structured logs, SLOs and sensible alerts.

Once a model is live you need to know three things: is it up, is it fast enough, and is it still producing sensible answers? This lesson covers the first two, service health. The third, whether predictions are still good, is the subject of drift detection and performance monitoring. All three layers matter, and the first is the easiest to automate.

Metrics, logs and traces

Observability rests on three kinds of data:

  • Metrics: numbers over time (requests per second, error rate, p95 latency). Cheap to store, good for dashboards and alerts.
  • Logs: a record of individual events ("request b1f4 scored 0.27 with model churn-v7"). Good for debugging one case.
  • Traces: the path of one request through several services with timings. Good for finding which step is slow.

The RED method

Rate Requests per second.
Errors Share of failed requests.
Duration Latency percentiles.

Three numbers describe most service problems.

1 / 2

The RED method (Rate, Errors, Duration) is a good default for any request-driven service. For ML services add a few model-specific signals: the model version, the distribution of output scores, and the share of missing or out-of-range inputs.

Counters and histograms

Two metric types do most of the work:

  • A counter only goes up (total requests, total errors). You compute rates from it.
  • A histogram counts observations into buckets (how many requests took under 10 ms, under 50 ms, under 100 ms...). From buckets you can estimate percentiles without storing every request.
Python A tiny metrics registry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

That text is the format a Prometheus server reads from a /metrics endpoint every few seconds. Real services use the official client library (prometheus_client) instead of hand-written counters, which cannot be installed here, but the exposition format above is exactly what it produces.

Why cumulative buckets? Because a query such as "what fraction of requests finished under 100 ms" is then one division, and percentiles can be estimated by interpolating between bucket boundaries.

Structured logs

Log lines should be machine-readable. One JSON object per line makes searching and aggregating easy, and lets monitoring pick fields out without fragile text parsing.

Python Structured prediction logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Logs can contain personal data. Decide what you may store, mask or hash identifiers, and set a retention period. Never log passwords, tokens or full payment details.

SLIs, SLOs and error budgets

  • An SLI (service level indicator) is a measured number: "share of requests answered successfully in under 200 ms".
  • An SLO (objective) is the target: "99.5 percent of requests over 30 days".
  • The error budget is what is left: with a 99.5 percent SLO, 0.5 percent of requests may fail.

When the budget is nearly used up, stop shipping risky changes and fix reliability. When plenty is left, you can move faster. It turns a vague argument about "how reliable is reliable enough" into arithmetic.

Python Error budget arithmetic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A burn rate above 1 means you will exhaust the budget before the window ends. Alerting on burn rate catches real trouble early and ignores harmless blips.

Alerts that people trust

Bad alerts train people to ignore alerts. Good ones follow rules:

  • Alert on symptoms, not causes. "Error rate above 2 percent for 5 minutes" matters to users; "CPU at 80 percent" may not.
  • Require a duration (for 5m) so a single blip does not page anyone.
  • Every alert needs an action. If nobody knows what to do, it is a dashboard panel, not an alert.
  • Separate pages from tickets. Urgent and user-visible means a page; slow degradation means a ticket.
  • Link a runbook with the first three things to check and how to roll back.

A Prometheus alert rule looks like this (read-only, since the site cannot run Prometheus):

YAML
groups:
  - name: model-service
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(predict_requests_total{status=~"5.."}[5m]))
          / sum(rate(predict_requests_total[5m])) > 0.02
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "More than 2% of predictions are failing"
          runbook: "https://wiki.example.com/runbooks/model-errors"

What to put on the dashboard

One page, top to bottom: traffic (requests per second), error rate, latency p50/p95/p99, saturation (CPU, memory, queue length), the current model version and deploy markers, and the score distribution next to its usual shape. Deploy markers (vertical lines at release times) answer "did it start after the release?" instantly.

Common mistakes

  • Only monitoring infrastructure. A server can be "healthy" while the model returns nonsense.
  • No model version label. You cannot compare old and new models.
  • Averages instead of percentiles.
  • Alert fatigue from too many noisy rules.
  • High-cardinality labels (user id as a label) that overload the metrics system. Keep labels to a small set of values.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the RED method stand for?

    Requests per second, share of failures and how long requests take cover most service problems.
  2. 2. Why are histogram buckets cumulative?

    Cumulative buckets make threshold fractions and percentile estimates simple.
  3. 3. An SLO of 99.9 percent over 30 days with 10 million requests allows how many bad requests?

    0.1 percent of 10,000,000 is 10,000.
  4. 4. Which alert is better?

    It describes a user-visible symptom, requires persistence and tells responders what to do.

Summary

  • Monitor rate, errors and duration, plus ML extras: model version, score distribution and missing inputs.
  • Use counters and cumulative histograms; expose them in Prometheus format.
  • Write structured, privacy-aware logs.
  • SLOs and error budgets turn reliability into arithmetic; alert on burn rate.
  • Alert on symptoms with durations and runbooks, and keep alerts few and meaningful.

Next: Understanding Drift