Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Cost and Operations

API, Terraform and Automation

Manage Datadog as code: API and application keys, submitting metrics and events, the Terraform provider for monitors and dashboards, and a safe review workflow.

Clicking in the UI is fine for the first monitor. For the fiftieth it is a risk: no review, no history, no way to copy staging to production. Everything in Datadog is available through an HTTP API, and the official Terraform provider lets you keep monitors, dashboards and SLOs in version control.

You cannot call Datadog from this site, so API calls and Terraform are read-only here. The runnable stand-ins build the requests and check the configuration locally, which is the part you can test without an account.

API keys and application keys

Datadog uses two credentials:

Credential Purpose Header
API key Identifies your organisation; lets you send data DD-API-KEY
Application key Identifies a user or service account with scopes; lets you read and manage things DD-APPLICATION-KEY

Both live in Organization Settings > API Keys / Application Keys. Rules:

  • Store them in a secret manager or CI secrets, never in Git or in frontend code.
  • Create separate keys per use (CI, Terraform, a script) so you can rotate one without breaking others.
  • Give application keys the minimum scopes.
  • The site matters: api.datadoghq.com (US1), api.datadoghq.eu (EU), api.us3.datadoghq.com and others. A key for one site fails on another.

Submitting a metric

Terminal
curl -X POST "https://api.datadoghq.com/api/v2/series" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -d '{
    "series": [{
      "metric": "orders.created",
      "type": 1,
      "points": [{ "timestamp": 1760000000, "value": 7 }],
      "tags": ["env:prod", "service:checkout-api"]
    }]
  }'

type is 1 for count, 2 for rate and 3 for gauge in the v2 endpoint. For a running service you usually do not call HTTP per point; you send metrics through the Agent with DogStatsD (UDP port 8125) using a client library, for example statsd.increment("orders.created", tags=["env:prod"]).

Managing monitors through the API

Terminal
curl -X POST "https://api.datadoghq.com/api/v1/monitor" \
  -H "DD-API-KEY: ${DD_API_KEY}" -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Checkout error rate high",
    "type": "query alert",
    "query": "avg(last_5m):sum:trace.http.request.errors{service:checkout-api,env:prod}.as_rate() / sum:trace.http.request.hits{service:checkout-api,env:prod}.as_rate() > 0.02",
    "message": "{{#is_alert}}Error rate above 2% @slack-payments{{/is_alert}}",
    "tags": ["team:payments", "service:checkout-api"],
    "options": { "thresholds": { "critical": 0.02, "warning": 0.01 } }
  }'

You can also GET, PUT and DELETE monitors, dashboards, SLOs, downtimes and more. Check the current API reference for exact fields: the API evolves and new endpoints appear under v2.

Stand-in: build and validate a monitor payload

This code builds the same JSON in a helper and refuses to produce a monitor missing the basics your team requires (owner, runbook, thresholds).

JS Stand-in: monitor payload builder with checks

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Terraform

The provider DataDog/datadog maps resources one to one: datadog_monitor, datadog_dashboard_json, datadog_service_level_objective, datadog_synthetics_test, datadog_downtime_schedule and many more.

HCL
terraform {
  required_providers {
    datadog = { source = "DataDog/datadog" }
  }
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.datadoghq.com/"
}

resource "datadog_monitor" "checkout_errors" {
  name    = "Checkout error rate high (${var.env})"
  type    = "query alert"
  query   = "avg(last_5m):sum:trace.http.request.errors{service:checkout-api,env:${var.env}}.as_rate() / sum:trace.http.request.hits{service:checkout-api,env:${var.env}}.as_rate() > 0.02"
  message = "{{#is_alert}}Errors above 2%. @slack-payments{{/is_alert}}"
  tags    = ["team:payments", "env:${var.env}"]

  monitor_thresholds {
    critical = 0.02
    warning  = 0.01
  }
}

Workflow: terraform plan shows the change, a pull request gets reviewed, terraform apply runs in CI. Use a module that wraps a monitor with your standard message and tags, then call it per service. Run terraform import to bring existing UI-created monitors under management. Turn on restricted_roles or editing limits so people do not edit managed monitors by hand (the next apply would revert them).

Dashboards are easiest as JSON: build in the UI, export JSON, store it as a file and use datadog_dashboard_json with file().

Other automation

  • datadog-ci: upload source maps, run synthetic tests, send deployment events, SBOMs.
  • Events API: send a deployment event so markers appear on graphs.
  • Workflow Automation and App Builder (in Datadog) run actions after a monitor fires, such as restarting a service or creating a ticket.
  • Webhooks integration posts monitor events to your own endpoint.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which key lets a script submit metrics?

    The API key (DD-API-KEY header) authorises sending data for your organisation.
  2. 2. Why keep monitors in Terraform?

    Infrastructure as code gives pull requests, history and consistent environments.
  3. 3. A key from datadoghq.eu does not work against api.datadoghq.com because:

    Each site is a separate environment with its own keys.
  4. 4. What is DogStatsD used for?

    Applications send metrics to the Agent's DogStatsD server, usually on port 8125.

Summary

  • API keys send data; application keys read and manage resources; protect both.
  • Use the API for scripts and DogStatsD for application metrics.
  • The Terraform provider manages monitors, dashboards, SLOs and Synthetics as code.
  • Wrap standards (owner, runbook, tags) in modules and validate them in CI.

Next: Incident Management and On-Call