Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท Agent and Integrations

Writing Custom Datadog Agent Checks in Python

Write a custom Datadog Agent check in Python: the AgentCheck class, checks.d and conf.d files, gauge, count, rate, service checks, error handling, and a local test harness.

When no integration fits (an in-house service, a queue depth stored in a file, a legacy tool), you can write a custom check: a small Python class the Agent runs on a schedule. It is the supported way to turn any number you can read into a Datadog metric.

Custom checks are real Python, but they need a real Agent. So the code below is what you would deploy, and the TryIt runs it against a stand-in base class that records what the check submits. The stand-in shows the behavior; it is not the Agent's AgentCheck.

Two files, same name

A custom check named queue_depth needs:

TEXT
/etc/datadog-agent/checks.d/queue_depth.py        # the code
/etc/datadog-agent/conf.d/queue_depth.d/conf.yaml # the configuration

The Python file name and the conf folder name must match. The config tells the Agent how often to run it and carries instance settings:

YAML
init_config:

instances:
  - queue_file: /var/spool/jobs/depth.txt
    min_collection_interval: 20
    tags:
      - service:jobs
      - env:prod

The check class

Python
from datadog_checks.base import AgentCheck, ConfigurationError

__version__ = "1.0.0"

class QueueDepthCheck(AgentCheck):
    def check(self, instance):
        path = instance.get("queue_file")
        if not path:
            raise ConfigurationError("queue_file is required")

        tags = instance.get("tags", [])
        try:
            with open(path) as f:
                depth = int(f.read().strip())
        except (OSError, ValueError) as err:
            self.service_check("queue_depth.can_read", self.CRITICAL,
                               tags=tags, message=str(err))
            return

        self.service_check("queue_depth.can_read", self.OK, tags=tags)
        self.gauge("jobs.queue.depth", depth, tags=tags)

Rules to remember:

  • Inherit from AgentCheck (import from datadog_checks.base).
  • Implement check(self, instance). The Agent calls it once per instance, every collection interval.
  • The module should contain one AgentCheck subclass.
  • Prefix metric names with your own namespace (jobs.queue.depth), never system. or datadog..
  • Do not sleep or block for long. A check that takes longer than its interval skips runs.
  • Catch expected errors and report them as service checks or raise ConfigurationError for bad config.

Submission methods

AgentCheck provides these (they mirror the metric types in metric types):

Method Use it for
self.gauge(name, value, tags=[...]) A value that goes up and down (queue depth, temperature)
self.count(name, value, tags=[...]) Number of events since the last run
self.monotonic_count(name, value, tags=[...]) A cumulative counter that only grows; Agent submits the difference
self.rate(name, value, tags=[...]) Per-second rate computed from successive values
self.histogram(name, value, tags=[...]) Distribution of values within the interval
self.service_check(name, status, tags=[...], message=...) Up/down health with OK, WARNING, CRITICAL, UNKNOWN
self.event({...}) A discrete event
self.log.info(...) Write to the Agent log

Pick the right type: submitting a cumulative counter as a gauge makes graphs useless unless you apply diff() later.

Run it and see it

Terminal
sudo -u dd-agent datadog-agent check queue_depth        # one run, prints metrics
sudo datadog-agent configcheck | grep -A5 queue_depth   # is the config loaded?
sudo datadog-agent status | grep -A8 queue_depth        # health and last error

The new metric jobs.queue.depth is a custom metric, which may count against your plan (see DogStatsD and custom metrics). Check the check output before leaving it running.

Testing the logic without an Agent

Because the interesting part is plain Python, test it with a fake base class. The stand-in below records what the check submits so you can assert on it.

Python Stand-in: run a custom check against a fake AgentCheck

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: why monotonic_count is not a gauge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Packaging and deployment

For one host, copying the two files is enough. For a fleet, ship them with configuration management (Ansible, Puppet, a Helm confd ConfigMap) or package the check as a wheel and install it with datadog-agent integration install. Keep the code in Git with tests, version it with __version__, and treat it like any other service: log errors, avoid secrets in the config, and monitor the check itself (datadog.agent.check_status).

A check that has to call a slow remote API is a bad fit; run a small exporter instead and send values through DogStatsD.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A custom check is named uptime_ext. What must match?

    The Python file name and the conf.d folder name identify the check and must agree.
  2. 2. Which method should you use for a cumulative counter that only increases?

    monotonic_count submits the increase since the previous run, so graphs show activity.
  3. 3. How should a check report that its target is unreachable?

    Service checks carry up/down status you can alert on.
  4. 4. What is the method the Agent calls every interval?

    AgentCheck subclasses implement check(self, instance).

Summary

  • A custom check is a Python AgentCheck subclass in checks.d plus a config in conf.d/<name>.d.
  • Implement check(self, instance); submit metrics with gauge, count, monotonic_count, rate, histogram, and health with service_check.
  • Test with datadog-agent check <name> and with a fake base class in unit tests.
  • Custom metrics can count toward billing; keep names namespaced and tags bounded.

Next: Containers and Kubernetes Monitoring