Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท Metrics and Tags

DogStatsD and Custom Metrics: Send Your Own Metrics to Datadog

Send custom metrics with DogStatsD: the datagram format, counters, gauges, histograms, distributions and sets, sampling, client libraries, the 8125 port, and custom metrics billing.

The built-in metrics tell you about hosts and software you already monitor. The numbers that matter most to your business are in your own code: orders placed, jobs queued, cache hits, payment latency. DogStatsD is how your application sends them. It is a small UDP server inside the Agent, a Datadog-flavored version of the StatsD protocol, with tags and a few extra metric types.

Your application sends a short text packet and moves on. The call is fire-and-forget over UDP, so a slow or missing Agent never slows your code. The Agent aggregates the packets for 10 seconds and forwards them. Datadog is not available on this site, so the stand-ins here build and parse the packets and aggregate them the way DogStatsD does in principle.

The datagram format

TEXT
<METRIC_NAME>:<VALUE>|<TYPE>|@<SAMPLE_RATE>|#<TAG_KEY>:<TAG_VALUE>,<TAG2>

Examples:

TEXT
shop.orders.placed:1|c|#env:prod,payment:card
queue.depth:42|g|#service:jobs
checkout.latency:187|d|#service:checkout,region:eu
page.views:1|c|@0.1|#page:home
users.unique:alice|s

The types:

Code Type Behavior
c Count Increments; the Agent sums values over the flush interval
g Gauge Keeps the last value in the interval
h Histogram Collects values, reports avg, count, median, max, 95th percentile per Agent
ms Timer Same as histogram in DogStatsD
d Distribution Sends values for global server-side percentiles
s Set Counts unique values in the interval

@0.1 is a sample rate: the client sent only 10% of events, and the Agent scales counts back up (multiplies by 10) to estimate the true total. Use sampling to cut traffic for extremely hot code paths.

Client libraries

You rarely build packets yourself. Official clients exist for Python, Java, Go, Ruby, PHP, .NET, Node.js and more. A Python example with the datadog library:

Python
from datadog import initialize, statsd

initialize(statsd_host="127.0.0.1", statsd_port=8125)

statsd.increment("shop.orders.placed", tags=["env:prod", "payment:card"])
statsd.gauge("queue.depth", 42, tags=["service:jobs"])
statsd.distribution("checkout.latency", 187, tags=["service:checkout"])

with statsd.timed("db.query.time", tags=["db:orders"]):
    run_query()

Set DD_AGENT_HOST or pass the host and port, and add global tags with constant_tags. In containers, point the client at the node's Agent (host IP) or use a Unix domain socket, and enable DD_DOGSTATSD_NON_LOCAL_TRAFFIC on the Agent (see datadog.yaml configuration).

Choosing the right call

  • Counting something that happens: increment / c.
  • A level you sample: gauge / g.
  • A duration or size across many instances: distribution / d, because it gives correct global percentiles (see metric types).
  • Unique things (visitors): set / s.

Custom metrics and billing

A custom metric is any metric not from an included integration. Billing counts unique time series: each distinct combination of metric name and tag values (including host). Plans include a per-host allotment of custom metrics, and extra series are billed. Cardinality control is a design choice, not an afterthought (see tags). Specific practices:

  1. Use a stable, small set of tag values.
  2. Do not include request IDs, emails or timestamps in names or tags.
  3. Do not create metric names dynamically (orders_by_user_<id>).
  4. Check Metrics > Summary and the Plan & Usage page for the top metrics by cardinality.
  5. Use Metrics without Limits to keep only the tag keys you query.

Troubleshooting

  • Metric never appears: check that the Agent is running and DogStatsD is enabled; sudo datadog-agent status shows DogStatsD counters (packets received, parse errors).
  • Packet drops: a high rate of UDP packets can overflow the OS socket buffer; raise dogstatsd_so_rcvbuf, use a Unix socket, or sample.
  • Wrong values: counts reset at each flush, so a gauge is needed for levels.
  • Tags missing: the # section is comma-separated with no spaces.
  • Datagram too large: UDP packets should stay under the network MTU (the client batches smaller packets); use client buffering.
Python Stand-in: build and parse DogStatsD datagrams

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: aggregate one 10-second flush window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the second stand-in the sampled count |@0.5 adds 2 (1 divided by 0.5), so the three increments total 4.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which port does DogStatsD listen on by default?

    DogStatsD uses UDP port 8125.
  2. 2. What does |@0.1 mean in a datagram?

    The client sent about one in ten events, so the Agent multiplies counts by ten.
  3. 3. Which type gives correct global p99 latency across 50 pods?

    Distributions are aggregated server-side across all instances.
  4. 4. What drives custom metrics billing?

    Each distinct tag combination is a series, so unbounded tags cost money.

Summary

  • DogStatsD is a UDP server in the Agent (port 8125) that accepts name:value|type|@rate|#tags packets.
  • Types: count, gauge, histogram/timer, distribution, set; use distributions for global percentiles.
  • Client libraries wrap the protocol; calls are fire-and-forget.
  • Custom metrics are billed by unique series, so keep tags bounded.

Next: The Metrics Explorer and Query Language