Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 37 ยท Agent and Integrations

Datadog Cloud Integrations: AWS, Azure and Google Cloud Setup

Connect Datadog to AWS, Azure and Google Cloud: cross-account IAM roles, CloudWatch polling versus metric streams, log forwarding, tag collection, and keeping costs in check.

The Agent monitors what runs on a machine. But managed services such as Amazon RDS, S3, SQS, Azure SQL or Cloud Storage have no machine you can install anything on. For those, Datadog uses cloud integrations: it reads metrics, tags and logs from the provider's APIs using credentials you grant. This lesson covers the common pattern across the three big clouds, with AWS as the worked example.

Nothing here runs on this site. Policies and commands are real patterns shown read-only, and the stand-ins check the logic.

The idea: grant read access, let Datadog poll

You never give Datadog your cloud password. You create a role or service account with read-only permissions and let Datadog assume it.

flowchart LR
    DD["Datadog"] -->|"assume role + external ID"| R["IAM role<br/>DatadogIntegrationRole"]
    R -->|"read-only"| CW["CloudWatch metrics"]
    R -->|"read-only"| API["EC2, RDS, tags APIs"]
    FW["Forwarder Lambda"] -->|"logs"| DD

Datadog assumes a role in your account, reads metrics and metadata, and receives logs through a forwarder you deploy.

AWS setup in outline

  1. In Datadog, open Integrations > Amazon Web Services > Add AWS Account. Datadog shows its own AWS account ID and generates an external ID.
  2. Create an IAM role in your account. Its trust policy lets Datadog's account assume it, but only when the external ID matches (this prevents the "confused deputy" problem). The easiest way is the CloudFormation quick-start link in the setup page, which creates the role and policy for you.
  3. Attach a policy with read-only actions. A minimal sketch, shortened:
JSON
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "cloudwatch:ListMetrics",
      "cloudwatch:GetMetricData",
      "ec2:DescribeInstances",
      "ec2:DescribeTags",
      "tag:GetResources",
      "rds:DescribeDBInstances",
      "rds:ListTagsForResource"
    ],
    "Resource": "*"
  }]
}

Use the policy Datadog documents, because it lists the exact actions for each service you enable; the sample above only shows the shape.

  1. Back in Datadog, paste the role ARN, choose regions and services to collect, and save. Metrics begin to appear under names like aws.rds.cpuutilization and aws.sqs.number_of_messages_sent.

Azure uses an app registration (service principal) with the Monitoring Reader role at subscription scope. Google Cloud uses a service account with Cloud Monitoring Viewer, Compute Viewer and similar read roles, and Datadog impersonates it.

Polling versus streaming

By default Datadog polls CloudWatch on a schedule. That is simple, but metrics arrive with a delay of a few minutes and CloudWatch API calls are billed by AWS. For lower latency and fewer API calls, use CloudWatch Metric Streams through Kinesis Data Firehose into Datadog. Streams arrive in near real time but need Firehose infrastructure and also cost money. Choose polling for a start and streams for latency-sensitive production accounts.

Logs from cloud services

Cloud services write logs to places like S3 buckets, CloudWatch Logs groups or Pub/Sub topics. Datadog collects them with a forwarder: the Datadog Forwarder (an AWS Lambda from the official CloudFormation template), the Kinesis Firehose destination, or an Azure Event Hub / GCP Dataflow template. You subscribe the log source to the forwarder. See log collection.

Tags, hosts and resources

  • Cloud resources arrive with tags such as region, availability-zone, account and name. Resource tags (for example AWS Team=payments) become Datadog tags, lowercased and cleaned (rules in tags).
  • Use tag filters in the integration settings to include or exclude resources (for example only env:prod), which also lowers cost.
  • EC2 instances collected by the integration can count as hosts for billing. If you also install the Agent, Datadog merges them into one host using the instance ID, but you want the Agent for detailed data.

Costs and safeguards

  1. Enable only the services and regions you need.
  2. Use tag filters to skip noisy or temporary resources.
  3. Remember CloudWatch API and Firehose charges on the AWS side.
  4. Metrics like aws.ec2.cpuutilization come free with the integration; custom metrics you publish to CloudWatch can become Datadog custom metrics.
  5. Review who can edit the integration: it controls access to your cloud account.

Other integrations in the same family

Crawler-based integrations also exist for SaaS tools: GitHub, GitLab, Slack, PagerDuty, Jira, Okta, Cloudflare, Snowflake. They use OAuth or API tokens set up in the integration tile, and are mostly for events, ownership and notifications.

Python Stand-in: check an IAM policy for the required read actions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: polling delay versus streaming

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers in the stand-in are examples chosen to show the shape of the trade-off; real polling intervals and AWS publish lag vary by account, service and settings.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the AWS role trust policy use an external ID?

    The external ID blocks the confused deputy problem: only requests carrying your ID can assume the role.
  2. 2. Which option gives lower-latency AWS metrics than default polling?

    Metric Streams push metrics in near real time instead of waiting for polls.
  3. 3. What does Azure use for Datadog access?

    A service principal (app registration) with read roles lets Datadog read Azure metrics.
  4. 4. How do you reduce cloud integration cost and noise?

    Collect only what you need and filter by tags.

Summary

  • Cloud integrations poll provider APIs with read-only credentials you grant; no Agent is needed for managed services.
  • AWS uses a cross-account IAM role with an external ID; Azure uses a service principal; GCP a service account.
  • Polling is simple but delayed; Metric Streams reduce latency.
  • Logs flow through a forwarder; use tag filters and limit services to control cost.

Next: Metric Types and Aggregation