Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: Observability for a Web Service

Plan end-to-end observability for a small web service: tags, tracing, logs, a dashboard, monitors and a readiness checker you can run and extend yourself.

In this project you take an imaginary service, shop-api, and give it complete observability in Datadog. You will write the plan, then use a runnable checker that scores how ready the service is. The Datadog product cannot run here, so the project is a design exercise with real configuration snippets and a stand-in tool that validates the plan. You can apply the same steps in your own Datadog account later.

The service

shop-api is a Node.js or Python web API behind a load balancer, with a PostgreSQL database and a call to an external payments provider. It runs as containers on Kubernetes in two environments, staging and prod. The team is team:storefront.

Step 1: Agent and tags

Deploy the Agent as a DaemonSet (Helm chart or the Datadog Operator) with APM, logs and process collection enabled. Set the unified service tags on the pods:

YAML
env:
  - name: DD_SERVICE
    value: shop-api
  - name: DD_ENV
    value: prod
  - name: DD_VERSION
    value: "2024.11.3"
  - name: DD_LOGS_INJECTION
    value: "true"
  - name: DD_PROFILING_ENABLED
    value: "true"

Also add pod labels so Datadog attaches them as tags: tags.datadoghq.com/service, tags.datadoghq.com/env, tags.datadoghq.com/version.

Step 2: Tracing and logs

  • Install the tracer in the image and start the app through it.
  • Log as JSON to standard output (one line per event) so the Agent tails container logs and the pipeline parses fields.
  • Add a log exclusion filter for GET /health.
  • Add custom spans for pricing.calculate and tags such as cart.items.

Step 3: A dashboard

Design it top-down, using the layout from Dashboards and Widgets:

  1. Query values: request rate, error rate, p95 latency, SLO status.
  2. Timeseries: hits, errors and latency by resource_name, with deploy markers.
  3. Dependencies: PostgreSQL query time, payments-provider latency.
  4. Infrastructure: pod CPU and memory, restarts.
  5. A log stream of errors, filtered by $env.
  6. Template variables: $env, $version.

Step 4: Monitors

Monitor Type Condition
Error rate APM metric errors / hits above 2% for 5 minutes
Latency APM metric p95 above 800 ms for 10 minutes
Saturation Metric, multi alert by pod memory above 90% for 10 minutes
Crash loops Metric kubernetes.containers.restarts increasing
Availability Synthetic API test /health fails from 2 of 3 locations
Log errors Log new error pattern after deploy

Each monitor has a priority, team:storefront, a runbook link and routing by state (Alert pages, Warn goes to chat).

Step 5: Readiness checker

This stand-in turns the plan into a data structure and scores it. Extend it with checks that matter to your own team.

JS Stand-in: observability readiness checker

Press Run (or Ctrl + Enter) to see the console output here.

No output.

All checks pass for this plan. Break it on purpose (remove runbook from a pager monitor, or delete logInjection) and see which check fails.

Step 6: Prove it works

Observability you have not tested is a guess. Run these experiments in staging:

  1. Send traffic and confirm traces show the database and payments spans.
  2. Trigger an error (a failing endpoint) and confirm the monitor fires and the message links to the right dashboard.
  3. Kill a pod and see the crash-loop signal and recovery.
  4. Deploy a new version and check the version tag and deploy marker appear.
  5. Break the synthetic test on purpose (for example, point it to a wrong path) and confirm the alert route.

Write down what you saw for each experiment. That list becomes your acceptance test every time you onboard another service.

Extension ideas

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which tags must be set so that traces, logs and metrics connect?

    Unified service tagging uses service, env and version.
  2. 2. Why exclude GET /health from indexed logs?

    Frequent health checks produce many low-value lines.
  3. 3. Why test alerts in staging before relying on them?

    Experiments prove detection and routing work.
  4. 4. Which monitors should page a person?

    Page for urgent, actionable problems; send the rest to chat or tickets.

Summary

  • Plan observability end to end: tags, Agent, tracing, logs, dashboard, monitors, Synthetics and SLO.
  • Standardise owners, priorities and runbooks on every monitor.
  • Use a checklist (or code) to score readiness, and prove alerts work with experiments.

Next: Project: SLOs and an Alerting Plan