Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Dashboards and Monitors

Downtimes and Alert Hygiene

Schedule Datadog downtimes, mute noisy monitors safely, measure alert quality with simple statistics, and run a monthly clean-up that reduces alert fatigue.

Alert fatigue is what happens when people receive more alerts than they can act on. They start ignoring all of them, including the one that matters. This lesson covers the Datadog tools for keeping monitors quiet when work is planned (downtimes) and a repeatable routine for keeping the whole set healthy.

The product is not available on this site, so steps are described precisely and the analysis is done in labelled stand-ins.

Downtimes

A downtime silences notifications for monitors that match a scope, for a time window. The monitor still evaluates and changes state; only messages are suppressed. Use it for planned maintenance, a risky deploy or a known third-party outage.

In the app: Monitors > Manage Downtimes > Schedule Downtime. You choose:

  1. What to mute: all monitors, monitors by name or tag (team:payments), and an optional scope such as host:db-1 or env:staging.
  2. Schedule: one-time or recurring (for example every Sunday 02:00 to 03:00 UTC).
  3. Notifications: optional message to tell subscribers a downtime began and ended.

Rules of thumb:

  • Always set an end time; an open-ended mute is how real outages are missed.
  • Scope narrowly: mute service:billing env:prod, not "everything".
  • Mute a monitor only if the noise is a symptom of one fixable cause; otherwise fix the monitor.
  • Use downtimes for known work; use better thresholds for routine noise.

What makes an alert good

Ask, for every alert that fired in the past month:

Question If the answer is no
Did a human need to act? Lower to Warn, send to chat, or delete the monitor
Was it a real problem? Fix the threshold, window or scope
Was it actionable (runbook, owner)? Add a message with a runbook link and an owner
Did it fire once for one problem? Group with composite monitors or multi alert tags

Stand-in: rank noisy monitors

Datadog's Monitor Quality page and the Monitor Notifications Overview help you find noisy monitors. This stand-in does the same calculation on a list of past alerts.

JS Stand-in: which monitors are noisy?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A monitor with many alerts and a low action rate is the best candidate for a change.

Reduce noise without losing coverage

  • Widen the window or add a recovery threshold if the monitor flaps.
  • Add a warning level and page only on the critical one.
  • Use composite monitors: alert only if "error rate high AND traffic above 100 rpm", so a quiet night with two failed requests does not page.
  • Use anomaly or forecast monitors instead of fixed numbers for seasonal data.
  • Set No Data carefully: turn it off for jobs that run once a day, or set the window longer than the job's interval.
  • Group by the right tag: one alert per service, not per pod.

Stand-in: composite logic

JS Stand-in: AND of two conditions

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The composite avoids a false page at minute 1 but still catches minute 2.

A monthly clean-up routine

  1. Open Monitors > Manage Monitors, filter by team:<yours> and sort by last triggered.
  2. Find monitors in No Data for weeks: the system they watch is probably gone. Delete or fix.
  3. Find monitors that fire more than a few times a week with no action. Tune or delete.
  4. Check every monitor has an owner tag, a runbook link and a priority.
  5. Review downtimes with no end date, and mute rules that hide real monitors.
  6. Write down what you changed so the next review can compare.

Do not delete a monitor just because it is quiet. Quiet is the goal. Delete it only when the thing it watches no longer exists or when its signal is covered better elsewhere.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a downtime do to a monitor?

    Downtimes mute messages; the monitor still runs and changes state.
  2. 2. Why should every downtime have an end time?

    Forgotten mutes are a common cause of missed incidents.
  3. 3. Which signal marks a monitor as a candidate for tuning?

    High volume with a low action rate means noise.
  4. 4. How does a composite monitor reduce false pages?

    Requiring conditions such as high error rate AND meaningful traffic removes low-volume noise.

Summary

  • Downtimes mute notifications for planned work; keep them narrow and time-boxed.
  • Judge each alert by whether it needed action, was real and was actionable.
  • Fix noise with warning levels, recovery thresholds, composites and better scopes.
  • Run a monthly review of noisy, silent and unowned monitors.

Next: Synthetic Tests