Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 37 ยท Dashboards and Monitors

Monitor Types and Thresholds

Create Datadog monitors that fire when they should: metric, anomaly, forecast, composite, log and APM monitors, thresholds, recovery, no-data and evaluation delay.

A monitor watches data and changes state when a condition holds. Dashboards help you look; monitors tell you when to look. A good monitor is specific (it fires for a real problem), timely and actionable (a person knows what to do).

Datadog is not available on this site, so the real syntax is shown read-only and the runnable code is a stand-in evaluator for the logic.

Monitor states

A monitor is always in one state: OK, Warn, Alert or No Data. It can have two thresholds, a warning and a critical one. Status changes trigger notifications (Notifications and SLOs).

Common monitor types

Type Use it when
Metric A number crosses a limit: CPU above 90%, queue length above 1,000
Anomaly The value is unusual for its normal pattern (daily or weekly seasonality)
Outlier One host or pod behaves differently from its peers
Forecast A value will cross a limit soon, such as disk filling in 7 days
Change Value changed by more than X% compared to a past period
Composite Combine several monitors with AND, OR (for example high latency AND high traffic)
Log Count or measure of logs matching a query
APM Trace metrics for a service or resource: error rate, latency percentiles
Integration / Service check A check reports OK/CRITICAL, such as a process or port check
Synthetic A synthetic test fails (Synthetic Tests)
SLO / Error budget Burn rate or remaining budget (Notifications and SLOs)

A metric monitor definition

The monitor query combines an evaluation window, an aggregation and a comparison:

TEXT
avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 90
  • avg(last_5m) is the time aggregation: average over the last five minutes.
  • The inner avg: is the space aggregation across series.
  • by {host} makes it a multi alert: one alert per host, tracked separately.
  • > 90 is the critical threshold. You can add a warning threshold of 80 and recovery thresholds so the monitor does not flap.

Settings worth knowing:

  • Alert on missing data / No Data after N minutes: decide whether silence is a problem. For a heartbeat metric it is; for a batch job it may not be.
  • Evaluation delay: wait N seconds before evaluating, so late-arriving cloud metrics (often 5+ minutes late) do not cause false alarms.
  • Require a full window: only evaluate when the whole window has data.
  • Renotify: repeat the message every N minutes while unresolved.
  • Priority (P1 to P5) and tags such as team:payments, which you can use for routing and for filtering later.

Stand-in: a monitor evaluator

This code evaluates a series against a warning and critical threshold with recovery thresholds, so you can see why recovery thresholds stop flapping.

JS Stand-in: thresholds and recovery

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With recovery thresholds the monitor holds its state until the value is clearly better, avoiding rapid alert/OK flips around the limit.

Stand-in: a simple anomaly check

Datadog's anomaly algorithms (basic, agile, robust) learn seasonality. The simplified stand-in below flags a point that is far from the recent mean in units of standard deviation.

JS Stand-in: z-score anomaly

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The spike at 240 is flagged. A real anomaly monitor does the same idea with trend and seasonality handled for you, so a normal Monday-morning peak is not an anomaly.

Choosing thresholds

  • Alert on symptoms, not causes. A user-facing error rate above 2% matters; one pod's CPU at 85% usually does not.
  • Start from data. Look at the last 30 days on a graph and pick a limit that would have caught real incidents without firing daily.
  • Use multi alerts for fleets so one bad host does not hide another.
  • Make windows match impact. last_5m reacts fast; last_1h is calm but slow. Pair them with warning/critical.
  • Test the monitor. The editor has an evaluation graph and a Test Notifications button.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a multi alert monitor (by {host}) do?

    Group by makes each group (here each host) its own alert with its own status.
  2. 2. Why use a recovery threshold?

    A recovery threshold keeps the alert on until the metric is clearly back to normal.
  3. 3. Cloud integration metrics arrive about 5 minutes late. Which setting prevents false alarms?

    An evaluation delay waits so late data is present when the monitor evaluates.
  4. 4. Which monitor type predicts that a disk will fill soon?

    A forecast monitor projects a trend forward and alerts before the limit is hit.

Summary

  • Monitors have states OK, Warn, Alert and No Data.
  • Choose the type for the question: metric, anomaly, outlier, forecast, composite, log, APM, synthetic or SLO.
  • A query has a time window, aggregation, scope, group and threshold.
  • Use recovery thresholds, evaluation delay and No Data settings to prevent noise.
  • Alert on symptoms and test with historical data.

Next: Notifications and SLOs