Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Agent and Integrations

Datadog Agent Architecture: Collector, Forwarder, DogStatsD and Ports

How the Datadog Agent works inside: collector and checks, forwarder, DogStatsD, trace agent, process and system probe, default ports, intervals and what leaves your network.

You can use the Datadog Agent for years without knowing what is inside it. But the day a metric is missing, a port is already used, or the firewall team asks "what does this thing connect to?", the internal layout matters. This lesson explains the Agent's moving parts, the ports they use, and the path data takes from your host to Datadog.

The Agent is open source software written mostly in Go, with an embedded Python runtime for checks. It cannot run on this site, so the code you see is configuration and shell, and the TryIt blocks are stand-ins.

One package, several processes

On a typical Linux host the Agent runs as a handful of cooperating processes under the datadog-agent service:

flowchart LR
    subgraph Host
      C["Collector<br/>(runs checks every 15s)"]
      D["DogStatsD<br/>UDP 8125"]
      T["Trace Agent<br/>TCP 8126"]
      P["Process Agent"]
      S["System Probe<br/>(eBPF, optional)"]
      F["Forwarder<br/>(queue + retry)"]
    end
    APP["Your apps"] -->|"custom metrics"| D
    APP -->|"traces"| T
    C --> F
    D --> F
    T --> F
    P --> F
    S --> P
    F -->|"HTTPS 443"| DD["Datadog intake"]

Everything funnels into the forwarder, which sends over HTTPS.

  • Core Agent contains the collector (runs integration checks and the built-in system checks), the forwarder (buffers payloads and sends them over HTTPS with retries) and the DogStatsD server.
  • DogStatsD is a metrics aggregation server that listens on UDP port 8125 (and optionally a Unix domain socket). Your applications send tiny text packets to it. See DogStatsD and custom metrics.
  • Trace Agent listens on TCP port 8126 for traces from the APM libraries and forwards them after sampling.
  • Process Agent reports running processes (when enabled) and container data.
  • System Probe is an optional privileged component that uses eBPF for network performance monitoring, universal service monitoring and security features.
  • Security Agent collects security signals when those products are on.

You can switch components on or off in datadog.yaml, for example apm_config.enabled, process_config.process_collection.enabled and logs_enabled.

The collector and checks

A check is a small program that collects data from one thing: the operating system, a database, a web server. The collector runs each configured check on a schedule. The default min_collection_interval is 15 seconds, and you can raise it per instance for expensive checks. A check can emit metrics, service checks (an up/down status such as nginx.can_connect), and events.

Checks come in two flavors:

  1. Core checks compiled into the Agent in Go (CPU, memory, disk, network, many containers checks).
  2. Python checks (integrations) that ship in the embedded Python or that you install. Custom checks are Python as well; see custom checks.

What the Agent sends, and how often

Metrics from checks are collected every 15 seconds but flushed to Datadog in batches roughly every 15 seconds as well. DogStatsD aggregates incoming packets into a 10-second flush interval before forwarding. When you graph a long time range, Datadog rolls the points up into larger buckets.

All traffic goes outbound over HTTPS (443) to Datadog endpoints for your site, such as api.datadoghq.com, agent-intake.logs.datadoghq.com and trace.agent.datadoghq.com. Nothing inbound is required from the internet. If a host cannot reach the internet, route through the proxy setting or a forwarder host.

Ports you may need to know

Port Protocol Used by
8125 UDP DogStatsD
8126 TCP Trace Agent (APM)
5000 TCP Agent's local IPC and expvar (localhost)
5001 TCP Agent's CLI/IPC API (localhost)
443 TCP outbound Datadog intake

The local ports bind to localhost, so they are not exposed. Set dogstatsd_non_local_traffic: true only on purpose: it lets other hosts (or containers) send to this Agent's DogStatsD, and must be paired with a firewall rule.

Hostname and host tags

Datadog needs a hostname to identify the host. The Agent finds it by trying, in order: hostname in datadog.yaml, the cloud provider's instance ID or name when it detects one, then the OS hostname. Duplicates cause two machines to overwrite each other, so on cloned VMs set hostname explicitly. Tags set in datadog.yaml (tags:) or through DD_TAGS apply to every metric, log and trace the Agent sends from that host.

Python Stand-in: pipeline latency of the Agent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: which component handles this?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reliability and resource use

The forwarder keeps a retry queue in memory (and can spill to disk on newer versions) if Datadog is unreachable, so short network outages do not lose data. The Agent is designed to be light: a typical host uses on the order of tens of megabytes of memory and a few percent of one CPU core, but this grows with the number of checks, containers and DogStatsD traffic, so measure on your own workload. The datadog-agent status output shows per-check execution time, which helps you find a slow check.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which port does DogStatsD listen on by default?

    DogStatsD listens on UDP 8125; the trace agent uses TCP 8126.
  2. 2. What is the default check collection interval?

    Checks run every 15 seconds unless min_collection_interval says otherwise.
  3. 3. Which Agent component buffers payloads and sends them to Datadog?

    The forwarder queues, retries and sends all payloads over HTTPS.
  4. 4. What direction of network traffic does the Agent need to Datadog?

    The Agent only makes outbound HTTPS connections; nothing inbound from the internet is needed.
  5. 5. What should you do when cloned VMs share a hostname?

    Identical hostnames make two hosts appear as one, so give each a unique hostname.

Summary

  • The Agent is a set of components: collector, forwarder, DogStatsD, trace agent, process agent, system probe.
  • Checks run every 15 seconds by default; DogStatsD aggregates over 10 seconds.
  • Ports: UDP 8125 (DogStatsD), TCP 8126 (traces), outbound 443 to Datadog.
  • The hostname identifies the host; set it explicitly on cloned machines.

Next: Installing the Agent on Linux, Windows, Docker and More