Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 35 ยท Operations and Cost

Monitoring and Alerts on DigitalOcean

Watch your Droplets with DigitalOcean Monitoring: the metrics agent, CPU, memory, disk and bandwidth graphs, alert policies, uptime checks and what to alert on.

You cannot fix what you cannot see. A server that quietly runs out of disk or memory fails at the worst moment, usually at night. DigitalOcean Monitoring collects metrics from your Droplets, draws graphs, and sends alerts by email or Slack when a metric crosses a threshold. Uptime checks test your site from outside, which catches the problems a metrics agent on the machine cannot (DNS broken, certificate expired, firewall blocking everything).

DigitalOcean is not available on this site, so the setup is shown as text, and the Python blocks are stand-ins that evaluate alert rules and uptime math on fabricated sample data.

The metrics agent

Droplets created with the Monitoring option enabled (a checkbox, or --enable-monitoring in doctl) get the agent installed automatically. On an existing Droplet install it with the script from the documentation, which currently looks like this:

Terminal
curl -sSL https://repos.insights.digitalocean.com/install.sh | sudo bash
sudo systemctl status do-agent

Read the script before piping it to a shell, and check the docs for the current command. The agent reports CPU, memory, disk usage, disk I/O, load average and public and private bandwidth every minute. Graphs are in the control panel, on the Droplet's Graphs tab, and the same data is available through the API.

Basic CPU, disk I/O and bandwidth graphs come from the hypervisor even without the agent, but memory and disk usage need it, so enable it everywhere.

Alert policies

An alert policy has four parts: a metric (such as CPU percent, memory percent, disk percent, public bandwidth), a condition (above or below a value), a window (5, 10, 30 minutes or 1 hour) and targets (specific Droplets or a tag). Notifications go to email addresses and optionally a Slack channel.

Terminal
doctl monitoring alert create \
  --type v1/insights/droplet/cpu \
  --compare GreaterThan --value 80 --window 5m \
  --tags web --emails [email protected] \
  --description "web CPU above 80% for 5 minutes" --enabled
doctl monitoring alert list

(The metric type names and the allowed windows are listed in the API documentation; check doctl monitoring alert create --help for the current values.)

The window is the key idea. A CPU spike of ten seconds is normal; an average above 80 percent for five minutes is a problem. This stand-in evaluates an alert rule against a series of one-minute samples.

Python Stand-in: windowed alert policy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single 92 percent reading at minute 3 does nothing; the sustained run starting at minute 6 trips the policy, a few minutes after it began. A longer window reduces false alarms but adds delay.

What to alert on

Alert on things a person must act on:

  • Disk above 85 to 90 percent (a full disk breaks databases and logs).
  • Memory above 90 percent for several minutes (the kernel may kill processes).
  • CPU above 80 percent for 10 minutes (capacity, or a runaway process).
  • Load average well above your vCPU count.
  • Bandwidth spikes, which can be an attack or a runaway download.
  • Uptime check failures and certificate expiry (see TLS).

Do not alert on everything: if every alert is ignored, the one real one is missed. Each alert should have a known action ("add disk", "restart the worker", "scale out"). Send critical alerts to a channel people actually read, and write a short runbook for each.

Uptime checks

An uptime check requests a URL (HTTP, HTTPS or ping) every minute from several regions and alerts when it fails or is slow, or when the SSL certificate is close to expiry. Use one for your public homepage and one for a health endpoint that touches the database. The percentage that matters for a service-level goal is simple arithmetic: how much downtime a given availability allows.

Python Stand-in: downtime budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A single Droplet rarely achieves 99.99 percent. Redundancy (Load Balancers, several Droplets, managed databases with standby nodes) is how you buy more nines.

Beyond the built-in tools

For application-level visibility you will usually add more:

  • Logs. Ship logs to a log service, or run a stack such as Loki or an Elasticsearch-compatible tool. App Platform keeps and streams logs for you.
  • Application metrics. Expose a /metrics endpoint and scrape it with Prometheus, then draw graphs in Grafana. Both run happily on a small Droplet.
  • Managed database metrics (connections, replication lag, cache hit ratio) are in each database cluster's Insights tab.
  • Kubernetes: use kubectl top, the Metrics Server, and a Prometheus stack for cluster metrics.
  • Third parties (Datadog, Grafana Cloud, Better Stack) are common when you outgrow the basics.

Review dashboards regularly, not only during incidents, and look at trends: a disk that grows 1 GB a day is a problem in six weeks, not tonight.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which metrics need the do-agent rather than coming from the hypervisor?

    Memory and disk usage are measured inside the machine, so they need the agent.
  2. 2. Why does an alert policy have a window?

    Averaging over a window filters brief spikes and reduces false alarms.
  3. 3. What does an uptime check test that the agent cannot?

    An external probe sees what visitors see, including failures outside the server.
  4. 4. How many minutes of downtime does 99.9 percent allow in 30 days?

    30 days is 43,200 minutes, and 0.1 percent of that is about 43 minutes.

Summary

  • Enable the monitoring agent on every Droplet for memory, disk and bandwidth metrics.
  • Alert policies use a metric, threshold, window and targets (by tag); alert only on conditions that need action.
  • Uptime checks test from outside and can watch certificate expiry.
  • Availability targets are downtime budgets; more nines require redundancy.

Next: Backups, Snapshots and Recovery