Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 70 ยท Monitoring, operations and cost

CloudWatch Metrics, Logs and Alarms

Learn CloudWatch: namespaces, dimensions, statistics, log groups, Logs Insights, alarms and dashboards, with a runnable metric aggregator and alarm state machine.

A system you cannot observe is a system you cannot run. Once your application is on EC2, Lambda or containers, you need answers to simple questions: is the CPU saturated, how many requests failed in the last five minutes, and who should be woken up when the error rate jumps? Amazon CloudWatch is the AWS service that collects numbers (metrics), text (logs) and reacts to both (alarms).

Almost every AWS service publishes metrics to CloudWatch for free or at low cost, so you often get useful graphs without installing anything. This lesson explains the data model, how statistics and periods work, how alarms decide when to fire, and how to search logs. Later lessons build on it: CloudTrail, Config and X-Ray cover audit and tracing, and Auto Scaling Groups use alarms to add or remove servers.

CloudWatch runs inside AWS and cannot run on this site. The CLI commands, query snippets and SDK code below are read-only. The runnable examples are small Python stand-ins that implement the same ideas (grouping data points, computing statistics per period, evaluating an alarm), so you can see how the numbers behave. They are simplified models, not the real service.

The metric data model

A metric is a time-ordered set of data points, for example the CPU utilization of one server. Each metric is identified by three things:

  • A namespace, a container that groups metrics from one source. AWS services use names that start with AWS/, such as AWS/EC2, AWS/Lambda and AWS/S3. Your own metrics go in a namespace you choose, such as MyShop/Checkout.
  • A metric name, such as CPUUtilization or Errors.
  • Up to 30 dimensions, which are name/value pairs that narrow a metric to a specific thing, for example InstanceId=i-0abc123.

Different dimension values make different metrics. CPUUtilization with InstanceId=i-0abc and CPUUtilization with InstanceId=i-0def are two separate metrics. A metric with no dimensions is also a different metric from the same name with dimensions: CloudWatch does not add them up for you. Services that publish aggregated views (for example per Auto Scaling group) do so as separate metrics with their own dimension.

Each data point has a timestamp, a value, and a unit (Percent, Count, Bytes, Seconds, and others). Here is how you read one metric with the CLI (read-only here):

Terminal
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc123def456 \
  --start-time 2026-05-01T10:00:00Z \
  --end-time 2026-05-01T11:00:00Z \
  --period 300 \
  --statistics Average Maximum

Two details are worth remembering. EC2 sends metrics every five minutes by default (basic monitoring) and every minute if you turn on detailed monitoring for a small fee. And EC2 cannot see inside the guest operating system, so memory use and disk space are not reported by default. To get them you install the CloudWatch agent, which publishes them as custom metrics.

Periods and statistics

Raw data points are rarely what you want. You ask CloudWatch to aggregate them over a period (a number of seconds, in multiples of 60 for standard metrics) using a statistic:

Statistic Meaning
Average Sum of values divided by the number of data points
Sum Total of all values (good for counts such as requests or errors)
Minimum, Maximum Lowest and highest value in the period
SampleCount How many data points arrived in the period
p90, p99 Percentiles (the value below which 90 or 99 percent of data points fall)

Pick the statistic that fits the question. For the number of failed requests use Sum. For latency, the average hides slow outliers, so use p99 or Maximum as well. A common beginner mistake is to alarm on Average of a Count metric and wonder why the numbers look tiny.

The stand-in below stores raw data points, groups them by dimension and period, and computes statistics. Nothing here is talking to AWS: it only shows what aggregation does.

Python A tiny metric store: dimensions, periods and statistics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at minute 2 of i-web1: one slow request (400 ms) lifts the average to 194 ms while the other two take under 100 ms, and Maximum and p99 show the outlier directly. The last line is the part that surprises newcomers: asking for the metric with no dimensions finds nothing, because a metric is identified by its exact dimension set.

Custom metrics and resolution

Your application can publish its own numbers with PutMetricData. Business metrics (orders placed, payments failed, queue depth in your own units) are often more useful than CPU. With the CLI or boto3 (read-only here):

Python
import boto3

cloudwatch = boto3.client("cloudwatch")
cloudwatch.put_metric_data(
    Namespace="MyShop/Checkout",
    MetricData=[{
        "MetricName": "OrdersPlaced",
        "Dimensions": [{"Name": "Environment", "Value": "prod"}],
        "Value": 1,
        "Unit": "Count",
    }],
)

Each distinct combination of namespace, name and dimensions is billed as a separate custom metric, so avoid high-cardinality dimensions such as a user ID or a request ID. A dimension with a million values means a million metrics. Put such details in logs instead.

Standard resolution metrics have a one-minute granularity. High-resolution custom metrics can be published at one-second granularity, which costs more. Older data is rolled up over time: data points with a period under 60 seconds are kept for a few hours, one-minute points for 15 days, five-minute points for 63 days and one-hour points for 455 days (about 15 months). If you need longer history, export it.

Logs: groups, streams and Logs Insights

Metrics tell you that something is wrong. Logs tell you why. CloudWatch Logs organizes text events like this:

  • A log group holds the logs of one application or component, for example /aws/lambda/checkout-handler. Retention (how long events are kept) and encryption are set per group. The default retention is never expire, which quietly grows your bill, so set a retention such as 30 days.
  • A log stream is a sequence of events from one source inside the group: one Lambda execution environment, one container, one server.
  • A log event is a timestamp plus a message.

Lambda sends its output to CloudWatch Logs automatically when its role allows it. On EC2 and in containers you configure the CloudWatch agent or a log driver.

flowchart LR
    L["Lambda function"] --> G["Log group<br/>/aws/lambda/checkout"]
    E["EC2 with agent"] --> G2["Log group<br/>/myshop/web"]
    G --> M["Metric filter:<br/>count ERROR lines"]
    G2 --> I["Logs Insights<br/>queries"]
    M --> A["CloudWatch alarm"]

Logs flow into groups. A metric filter turns matching lines into a metric, which an alarm can watch.

Logs Insights is a query language for log groups. A typical query finds the slowest requests in the last hour (read-only here, you would paste it into the console):

TEXT
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() as errors by bin(5m)
| sort errors desc
| limit 20

The commands are joined with pipes: fields picks columns, filter keeps matching events, stats aggregates (with bin(5m) as a time bucket), sort orders and limit caps the output. You pay per gigabyte of log data scanned, so narrow the time range and the log groups.

A metric filter turns log lines into a metric: for example, count each line that contains ERROR as one data point of a metric called ErrorLines. That lets you alarm on something your application only writes as text.

Alarms

An alarm watches one metric (or a math expression over metrics) and has one of three states:

  • OK: the metric is within the threshold.
  • ALARM: the threshold is breached.
  • INSUFFICIENT_DATA: not enough data points have arrived to decide.

The key settings are the period, the threshold, the evaluation periods (N, how many recent periods to look at) and the datapoints to alarm (M, how many of those N must breach). "3 out of 5" means the alarm fires only if at least three of the last five periods breached the threshold. This smooths out single spikes.

The treat missing data setting decides what a period without data means: missing (ignore it, the default), notBreaching, breaching or ignore (keep the previous state). A function that is idle most of the day produces no Errors data points at all, so notBreaching is usually right for error counts, while breaching suits a heartbeat metric that must always arrive.

When the state changes, the alarm can run actions: publish to an SNS topic (email, chat, pager), trigger an Auto Scaling policy, stop or reboot an EC2 instance, or create a Systems Manager item. A read-only CLI example:

Terminal
aws cloudwatch put-metric-alarm \
  --alarm-name web-high-cpu \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc123def456 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 5 \
  --datapoints-to-alarm 3 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:123456789012:ops-alerts

Here is the state machine in Python. Feed it the per-period values and see how "M out of N" and missing data change the result.

Python An alarm state machine with M out of N and missing data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "1 of 1" alarm follows every spike: ALARM at periods 2 and 3, OK at 4, ALARM again at 5 to 7, so it would page and recover repeatedly. The "3 of 5" alarm reports INSUFFICIENT_DATA until five periods exist, stays OK at period 4 (only two breaches in its window), goes to ALARM at period 5 when the third breach arrives, and stays there until fewer than three of the last five periods breach. The last block shows why treat missing data matters when a metric stops arriving: the same silence is INSUFFICIENT_DATA, OK or ALARM depending on the setting.

Real CloudWatch also has composite alarms (combine several alarms with AND/OR to cut noise), anomaly detection (a learned band instead of a fixed threshold) and metric math (for example errors divided by requests to get an error rate). All follow the same state model.

Synthetics canaries: test your site like a user

Metrics tell you what happened to real traffic. But if nobody visits your site at 3 a.m. and it is down, no metric moves, and you find out in the morning. CloudWatch Synthetics fixes this with canaries: small scripts that run on a schedule (every minute, every five minutes, or a cron expression) and exercise your application the way a user or a client would.

A canary is a Lambda function with extra tooling, written in Node.js (using Puppeteer) or Python (using Selenium). AWS provides blueprints so that you rarely start from scratch:

  • Heartbeat monitor: loads a URL, takes a screenshot, and checks that it succeeded. The simplest "is the page up" check.
  • API canary: calls a REST endpoint and checks the status code and the response body.
  • Broken link checker: crawls a page and tests every link on it.
  • GUI workflow builder: clicks through a flow such as log in, add to cart, view checkout.
  • Visual monitoring: compares screenshots against a baseline to catch unexpected layout changes.

Each run records its results as metrics (SuccessPercent, Duration) in the CloudWatchSynthetics namespace, stores logs, screenshots and HAR files in an S3 bucket, and can feed an alarm. A minimal Node.js heartbeat looks like this (read-only; it needs the Synthetics runtime):

JavaScript
const synthetics = require("Synthetics");

const heartbeat = async function () {
  const page = await synthetics.getPage();
  const response = await page.goto("https://www.example.com/health", { waitUntil: "domcontentloaded", timeout: 30000 });
  if (response.status() !== 200) {
    throw new Error(`Unexpected status ${response.status()}`);
  }
  await synthetics.takeScreenshot("health", "loaded");
};

exports.handler = async () => heartbeat();

Then alarm on the canary: for example, SuccessPercent < 100 for 2 of 3 runs, with an action that notifies the on-call person. Canaries are billed per run, so choose the frequency your service level needs. A canary every minute from one Region is a different cost from one every minute across five.

The stand-in below mimics the alarm logic for canary results. Real canaries run in AWS, but the "2 of 3 runs failed" evaluation is the same idea as the alarms earlier in this lesson.

Python Alarm on canary failures (2 out of 3)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A single failed run does not page anyone, but two failures in three runs do, and the alarm returns to OK when the site recovers.

CloudWatch or CloudTrail?

These two services are confused constantly, because both produce "logs" and both can trigger actions. The difference is the question they answer.

CloudWatch CloudTrail
Question "How is it running right now?" "Who did what, and when?"
Data Metrics, application and system logs, alarms A record of API calls and console actions in your account
Typical use Performance, capacity, health, error rate, alarms Security investigations, auditing, compliance, finding who deleted a resource
Example CPU of an instance, 5xx errors of a load balancer DeleteBucket was called by user alice at 14:02 from this IP
Triggers Alarms notify people or start Auto Scaling EventBridge rules react to specific API calls

They work together. CloudTrail can send its events to CloudWatch Logs, where a metric filter counts, say, console logins without MFA, and an alarm notifies the security team. Read more about CloudTrail in CloudTrail, Config and X-Ray. A third tool, AWS Config, answers a different question again: "what does this resource's configuration look like, and does it follow our rules?"

The CloudWatch agent in practice

As noted earlier, EC2 cannot see inside the guest operating system, so memory and disk metrics need the CloudWatch agent. It also ships log files. The agent reads a JSON configuration, which you can store in Systems Manager Parameter Store so every instance uses the same one (read-only):

JSON
{
  "metrics": {
    "append_dimensions": { "InstanceId": "${aws:InstanceId}" },
    "metrics_collected": {
      "mem":  { "measurement": ["mem_used_percent"] },
      "disk": { "measurement": ["used_percent"], "resources": ["/"] }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          { "file_path": "/var/log/nginx/error.log", "log_group_name": "/myshop/web", "log_stream_name": "{instance_id}" }
        ]
      }
    }
  }
}

The instance profile needs permission to write metrics and logs (the managed policy CloudWatchAgentServerPolicy covers it). Install the agent with Systems Manager so you do not have to log in to every machine.

Dashboards and good habits

A dashboard is a customizable page of graphs and numbers, shared across a team. A good dashboard answers the questions of an on-call person at a glance: traffic, errors, latency and saturation (CPU, memory, queue depth). Build one dashboard per service rather than one giant wall.

Habits that save money and pain:

  • Alarm on symptoms users feel (error rate, latency) before causes (CPU).
  • Send each alarm to somewhere a human reads, and write what to do in the alarm description.
  • Set log retention, and avoid logging secrets or full request bodies.
  • Keep dimensions low-cardinality, and publish only the custom metrics you will use.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You publish a metric with the dimension InstanceId=i-0abc and then query the same namespace and metric name with no dimensions. What do you get?

    Each combination of namespace, metric name and dimensions is its own metric. CloudWatch does not roll metrics up across dimension values unless something publishes the aggregate.
  2. 2. Which statistic is the best fit for an alarm on the number of failed requests per minute?

    For counts you want the total in the period, which is Sum. Average of a count metric gives the mean per data point and often hides the real volume.
  3. 3. An alarm is configured with evaluation periods 5 and datapoints to alarm 3. When does it go to ALARM?

    M out of N means at least M of the last N evaluated periods must breach. It smooths out brief spikes without waiting for every period to breach.
  4. 4. Why should you set a retention period on log groups?

    New log groups never expire by default. Choosing a retention such as 14 or 30 days deletes old events automatically and keeps costs predictable.

Summary

  • A metric is identified by namespace, name and dimensions; each distinct dimension set is a separate metric, and high-cardinality dimensions get expensive.
  • You read metrics by choosing a period and a statistic: use Sum for counts and Maximum or percentiles for latency.
  • Logs live in log groups and streams; set retention, and use Logs Insights queries or metric filters to find and count events.
  • An alarm has the states OK, ALARM and INSUFFICIENT_DATA, an "M out of N" rule and a missing-data policy, and can notify, scale or act.
  • EC2 does not report memory or disk by default: use the CloudWatch agent for those.

Next: CloudTrail, Config and X-Ray