Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Incident Drill

Practice a realistic incident: read metrics, traces and logs from a prepared dataset, find the root cause with a runnable investigation script, and write a postmortem.

Tools matter less than the investigation habit: form a hypothesis, test it against data, narrow down. In this drill you play the on-call engineer for shop-api. You receive an alert, then work through the three pillars (metrics, traces, logs) using a prepared dataset and runnable code that behaves like Datadog queries. No Datadog account is needed, and everything is a labelled stand-in; on a real system you would use the Explorer pages and dashboards from the earlier lessons.

The alert

[P2] Checkout error rate high on shop-api (env:prod). Value: 6.4% (limit 2%) over the last 5 minutes. Started 14:07 UTC. A deploy of version 2.15.0 finished at 14:03.

You are the incident commander and technical lead. Declare a SEV-2 and start a timeline (see Incident Management and On-Call).

Step 1: metrics. What changed?

Compare errors by version and by resource_name. In Datadog you would write sum:trace.http.request.errors{service:shop-api,env:prod} by {version}. Here we group a dataset.

JS Stand-in: group errors by version and resource

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Observation: the new version 2.15.0 has a high error rate and much higher latency on POST /orders only. Hypothesis one: the deploy broke checkout. Rolling back is a quick mitigation; do it if the evidence is strong. First look at traces to learn why.

Step 2: traces. Where is the time?

Open an error trace. Here are the spans of one failing request (invented data):

JS Stand-in: find the slow dependency in an error trace

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The payments call times out at 2 seconds. Hypothesis two: something changed in how 2.15.0 calls the provider (a timeout, a URL or a retry setting) or the provider itself is slow. Check the other versions: version 2.14.0 calls it fast, so the provider is probably fine.

Step 3: logs. Why exactly?

Logs from 2.15.0 pods, filtered to service:shop-api status:error and a pattern on the message:

JS Stand-in: cluster error messages

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The log lines say the new version points to payments-sandbox.internal, which is unreachable from production. A configuration value was shipped with the wrong environment. Root cause found: a staging configuration reached production with the deploy.

Step 4: mitigate and verify

  1. Mitigate: roll back to 2.14.0 (or fix the environment variable). Post an update to the incident channel.
  2. Verify: watch the error rate monitor recover, the checkout SLO burn rate stop rising, and a synthetic checkout test pass.
  3. Resolve the incident when metrics are normal for a reasonable time.

Step 5: write the postmortem

Use this skeleton:

  • Summary: from 14:03 to 14:31 UTC, 6% of checkout requests failed. About N customers saw an error.
  • Timeline: 14:03 deploy; 14:07 alert; 14:10 incident declared; 14:19 root cause found; 14:24 rollback; 14:31 recovered.
  • Root cause: staging payment URL in the production config. Contributing factors: no check on environment-specific values in CI; deploy had no canary; the alert fired only after 4 minutes.
  • Went well: tags allowed filtering by version at once; the runbook was linked in the alert.
  • Action items: add a CI check that rejects non-prod hostnames in prod config (owner, date); canary deploy with automatic rollback on error rate; add a synthetic checkout test; shorten the monitor window for fast burns.

Notice that the postmortem blames the process, not a person. "Who edited the config?" is less useful than "why could an unsafe config be shipped?".

Extension ideas

  • Re-run the drill for a different cause: a slow database query, an exhausted connection pool, a memory leak. Write the dataset yourself.
  • Time yourself: how quickly can you move from alert to hypothesis?
  • Present the postmortem to a teammate and ask what was unclear.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why group errors by version during an incident that starts after a deploy?

    Comparing versions quickly shows whether the change is the likely cause.
  2. 2. In the trace, which span contained the main self time?

    The payments call took 2000 ms and failed with a timeout.
  3. 3. What is a sensible first mitigation when a deploy clearly broke production?

    Restore service first, then investigate in depth.
  4. 4. Which is a good postmortem action item?

    Action items should change the system so the failure cannot easily repeat.

Summary

  • Investigate in a loop: observe metrics, narrow with traces, confirm with logs.
  • Compare by version, resource and dependency to isolate a cause.
  • Mitigate first, verify recovery with monitors and synthetics, then write a blameless postmortem with owned actions.

Next: Practice Questions and Next Steps