Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Cost and Operations

Incident Management and On-Call

Run incidents with Datadog: declare incidents from monitors, severity levels, roles, timelines, paging with on-call schedules, and blameless postmortems.

Monitoring finds problems; incident management is how people respond to them. Datadog bundles tools for this: Incident Management (declare, coordinate, record), On-Call (schedules, escalation, paging) and Case Management (smaller tracked work items). Datadog is not available on this site, so workflows are described precisely and the runnable stand-ins model the maths of schedules and metrics.

The incident life cycle

flowchart LR
  D["Detect<br/>(monitor, user report)"] --> T["Declare<br/>severity, commander"]
  T --> M["Mitigate<br/>restore service"]
  M --> R["Resolve<br/>confirm recovery"]
  R --> P["Postmortem<br/>learn, assign actions"]

Mitigate first (stop the pain), find the root cause after.

  1. Detect: a monitor fires, a customer reports, or someone notices a graph.
  2. Declare: from a monitor (Declare incident button), from Slack (/datadog incident), or in Incidents > New Incident. Set a title, severity, and the commander.
  3. Coordinate: one place for the timeline, tasks and communication.
  4. Mitigate and resolve: rollback, failover, scale, disable a feature flag.
  5. Postmortem: a blameless review with action items.

Severity

Define severities with plain, user-based criteria and write them down before you need them:

Severity Example meaning Response
SEV-1 Core service down for many customers Page on-call immediately, incident commander, status page update
SEV-2 Major feature degraded or a large subset affected Page the owning team
SEV-3 Minor impact, workaround exists Fix in working hours
SEV-4 No customer impact Ticket

Datadog lets you customise severity levels and fields (for example customer impact, services, detection method).

Roles

  • Incident commander (IC): owns the response and decisions; does not debug.
  • Operations / technical lead: investigates and fixes.
  • Communications lead: updates stakeholders and the status page.
  • Scribe: records what happened and when.

Small teams can combine roles, but the IC must be explicit. Datadog assigns these as incident responders and creates an incident Slack channel with a pinned summary.

The incident record

In an incident's page you will find a timeline (events you add and automatic ones such as monitor alerts or deploys), attached graphs and notebooks (snapshot the dashboard you used), tasks, responders and fields. Add notes during the incident; reconstructing a timeline later from memory is unreliable.

On-Call schedules

Datadog On-Call lets you define teams, schedules (rotations, for example weekly handoffs on Monday 10:00), escalation policies (page A, wait 5 minutes, page B, then the manager), and routing rules (alerts tagged team:payments page the payments policy). A page goes to the app, SMS, phone or Slack. Monitor messages can page with a handle such as @oncall-payments. You can also integrate PagerDuty or Opsgenie instead.

Stand-in: who is on call?

JS Stand-in: weekly rotation lookup

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second date is Monday at 09:00, before the 10:00 handoff, so the previous person is still on call.

Stand-in: incident metrics

After a quarter, measure the process, not people: MTTD (mean time to detect), MTTA (to acknowledge), MTTR (to restore), incident counts per severity, and how many action items closed.

JS Stand-in: MTTD, MTTA and MTTR

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With few incidents, averages mislead; show the median and the individual values too.

Blameless postmortems

Datadog has postmortem templates inside Incidents (also available as a Notebook). A good one contains:

  1. Summary and impact: who was affected, for how long.
  2. Timeline from the incident record.
  3. Root causes and contributing factors: systems and process, not "Bob forgot".
  4. What went well and what did not (detection, tooling, communication).
  5. Action items with an owner and a date, tracked in tickets (Case Management or Jira).

Blameless means you assume everyone acted reasonably with the information they had, and you ask why the system allowed the mistake.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What should an incident commander mainly do?

    The IC owns coordination and decisions; others investigate and fix.
  2. 2. What is the first priority during an incident?

    Stop the customer impact first; root cause analysis follows.
  3. 3. What does an escalation policy define?

    It lists the order and timing of people or teams that are paged.
  4. 4. Why is a postmortem 'blameless'?

    Blame discourages honest reporting; fixing systems prevents repeats.

Summary

  • Incidents follow detect, declare, mitigate, resolve, postmortem.
  • Define severities and roles before you need them; the IC coordinates.
  • On-Call schedules, escalation policies and routing rules decide who is paged.
  • Record the timeline during the incident; measure MTTD, MTTA and MTTR.
  • Postmortems are blameless and end in owned action items.

Next: Project: Observability for a Web Service