Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 70 ยท Monitoring, operations and cost

CloudTrail, Config and X-Ray

Learn who did what with CloudTrail, whether resources comply with AWS Config rules, and where requests slow down with X-Ray, with runnable query stand-ins.

CloudWatch tells you how your system is behaving. Three other services answer different questions. CloudTrail answers "who did what, and when?" by recording API calls. AWS Config answers "what does this resource look like, did it change, and does it follow our rules?" X-Ray answers "where did this one slow request spend its time?" by tracing a request across services.

You will use all three the first time something goes wrong in a way graphs cannot explain: a security group was opened to the world, an S3 bucket vanished, or a checkout takes four seconds and nobody knows which downstream call is to blame. This lesson explains how each service works and gives you runnable stand-ins that query events, evaluate rules and read a trace. It builds on IAM Users, Groups and Roles and on the idea from Console, CLI and SDKs that every action is an API call.

CloudTrail, Config and X-Ray run inside AWS and cannot run on this site. CLI commands and JSON samples are read-only. The runnable examples are small Python stand-ins that work on sample data shaped like the real records, so you can practice the logic of filtering, evaluating and reading traces.

CloudTrail: the audit log

Because the console, the CLI and every SDK all call the same HTTPS APIs, AWS can record each call. CloudTrail does that. A record says which principal made the call, when, from which IP address, which API, with which parameters, and whether it succeeded.

There are three kinds of events you will meet:

  • Management events are control-plane operations: creating an instance, changing a security group, attaching a policy, signing in to the console. They are recorded by default.
  • Data events are high-volume operations on the data in a resource, such as GetObject and PutObject on S3 objects, or Lambda Invoke. They are off by default and cost extra, so you enable them only for the buckets or functions where you need them.
  • Insights events flag unusual activity, such as a sudden jump in RunInstances calls.

Without any setup, the console's Event history shows the last 90 days of management events in the current Region. That is useful for a quick investigation but not enough for compliance. For longer retention you create a trail, which delivers log files to an S3 bucket (and optionally to CloudWatch Logs). A trail can cover all Regions, and an organization trail can cover every account in an AWS Organization, delivered to one central bucket.

flowchart LR
    U["User, role or service"] --> API["AWS API call"]
    API --> CT["CloudTrail"]
    CT --> H["Event history<br/>90 days, management events"]
    CT --> T["Trail"]
    T --> S3["S3 bucket<br/>(long-term, encrypted)"]
    T --> CWL["CloudWatch Logs<br/>(alarms on events)"]

Every API call is recorded. A trail keeps the records beyond 90 days and can feed alarms.

Turn on log file integrity validation for the trail. CloudTrail then writes hourly digest files that are signed, so you can prove later that no log file was changed or deleted. Protect the bucket with a restrictive policy: the audit log is only worth anything if attackers cannot edit it.

A single CloudTrail event, trimmed to the fields you read most often (read-only):

JSON
{
  "eventTime": "2026-05-12T09:14:03Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "AuthorizeSecurityGroupIngress",
  "awsRegion": "eu-west-1",
  "sourceIPAddress": "203.0.113.25",
  "userAgent": "aws-cli/2.17.0",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::123456789012:assumed-role/dev-role/alice",
    "accountId": "123456789012"
  },
  "requestParameters": {
    "groupId": "sg-0abc123",
    "ipPermissions": { "items": [{ "fromPort": 22, "toPort": 22, "ipRanges": { "items": [{ "cidrIp": "0.0.0.0/0" }] } }] }
  },
  "errorCode": null,
  "readOnly": false
}

Key fields: userIdentity (who), eventSource plus eventName (which service and API), sourceIPAddress and userAgent (from where and with what tool), requestParameters (what they asked for) and errorCode (present when the call failed, for example AccessDenied). Note that role sessions show up as AssumedRole with the session name at the end of the ARN, which is how you tell who actually used a shared role.

To query logs you can use CloudTrail Lake (SQL over events), Amazon Athena over the S3 files, or CloudWatch Logs Insights if the trail also delivers there.

Querying events

Real investigations are mostly filtering. The stand-in below runs the same kind of questions over a handful of sample events: who opened SSH to the world, which calls failed, and what did one session do.

Python Query CloudTrail-style events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real CloudTrail has far more fields and you query them with SQL or Athena, but the habit is the same: start from a suspicious change (the security group), find the identity and source IP, then look at everything else that identity did around that time. Alarms can watch for the dangerous cases automatically: a metric filter on a trail's CloudWatch log group can count root logins, console logins without MFA or changes to security groups, and alert you the moment they occur.

AWS Config: what do my resources look like?

CloudTrail records actions. AWS Config records state. It keeps a configuration item for each supported resource (an EC2 instance, a security group, an S3 bucket, an IAM role): what its settings were, and how they changed over time. You can ask "what did this security group look like last Tuesday?" and "which resources are connected to it?"

On top of that history, Config evaluates rules. A rule is a check that returns COMPLIANT or NON_COMPLIANT for each resource. AWS provides over a hundred managed rules, for example:

  • s3-bucket-public-read-prohibited
  • encrypted-volumes (attached EBS volumes must be encrypted)
  • restricted-ssh (security groups must not allow SSH from 0.0.0.0/0)
  • root-account-mfa-enabled

You can also write custom rules, using a Lambda function or a declarative language called Guard. Rules run when a resource changes (change-triggered) or on a schedule (periodic). If a resource is non-compliant, Config can start a remediation, for example a Systems Manager Automation document that closes the open port. Conformance packs bundle many rules and remediations into a deployable unit, and an aggregator shows compliance across accounts and Regions.

A key point: Config is a detective control. It tells you something drifted, and with remediation it can fix it afterwards, but it does not stop the change at the moment of the API call. Preventive controls are IAM policies and SCPs, as in How Policy Evaluation Works. A mature setup uses both: prevent the worst things, detect everything else.

Enabling Config is a one-time recorder setup, read-only here:

Terminal
aws configservice put-config-rule --config-rule '{
  "ConfigRuleName": "restricted-ssh",
  "Source": { "Owner": "AWS", "SourceIdentifier": "INCOMING_SSH_DISABLED" }
}'

Config charges per configuration item recorded and per rule evaluation, so enable it deliberately in accounts that matter and record only the resource types you need.

Python Evaluate Config-style compliance rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the third value, NOT_APPLICABLE: a rule only looks at the resource types it was written for. In real Config, evaluation results are stored and shown on a compliance dashboard, so you see non-compliant resources without running anything yourself.

X-Ray: following one request

A modern request often crosses an API gateway, a function, a queue and a database. When it is slow, metrics show only that the average latency rose. AWS X-Ray shows one request's journey as a trace:

  • A trace is the whole request, identified by a trace ID that is passed along in the X-Amzn-Trace-Id HTTP header.
  • A segment is the work one service did for that request (for example, the Lambda function's invocation).
  • Subsegments break a segment down: a downstream call to DynamoDB, an HTTP call to another API, or a block of your own code.
  • Annotations are indexed key/value pairs you can search by (for example customer_tier=gold); metadata is extra data you can read on a trace but cannot search.

X-Ray builds a service map from traces: nodes for services, arrows for calls, with latency and error rates on each. Because tracing every request is costly, X-Ray samples: by default the first request each second plus five percent of the rest, and you can change this with sampling rules. Services such as Lambda and API Gateway can turn on active tracing with a setting, and applications are instrumented with an SDK or with OpenTelemetry. For new projects, AWS steers instrumentation toward OpenTelemetry, with X-Ray as the backend that stores and shows the traces.

flowchart LR
    C["Client"] --> G["API Gateway<br/>12 ms"]
    G --> L["Lambda: checkout<br/>380 ms"]
    L --> D["DynamoDB<br/>25 ms"]
    L --> P["Payment API<br/>310 ms"]

A trace shows where the time went: here the payment call is most of the Lambda function's 380 ms.

The stand-in below holds a trace as a list of segments with start times and durations, then finds the slowest subsegment and prints a simple timeline. Reading a trace is mostly this: sort by duration, check what overlaps, and see which part has no child to explain it.

Python Read a trace: find the slowest part

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The timeline makes the story visible: the payment call takes about three quarters of the whole request, so optimizing the Lambda code would barely help. A real X-Ray trace shows the same thing in the console, with colors and an error or throttling marker on any failing call.

Choosing the right tool

Question Service
Who deleted the bucket, and from which IP? CloudTrail
Did this security group change last week? Config (configuration history)
Which resources break our encryption rule? Config rules
Why is this one request slow? X-Ray
Is the error rate rising right now? CloudWatch metrics and alarms, see CloudWatch Metrics, Logs and Alarms

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which CloudTrail event type is NOT recorded by default and costs extra when enabled?

    Management events (control-plane actions, sign-ins and IAM changes) are recorded by default. Data events are high volume, so you turn them on per bucket or function.
  2. 2. You need to prove to an auditor that no CloudTrail log file was modified after delivery. What do you enable?

    Integrity validation writes signed digest files that let you check logs for tampering or deletion. Event history is only 90 days and does not prove integrity.
  3. 3. What is the main difference between AWS Config and CloudTrail?

    CloudTrail is the log of who called which API. Config keeps what each resource looked like and checks it against rules.
  4. 4. In X-Ray, what is the difference between an annotation and metadata?

    Annotations are key/value pairs that X-Ray indexes so you can filter traces by them. Metadata is extra detail you can read on a trace but cannot search.
  5. 5. A Config rule reports a security group as NON_COMPLIANT. What does that tell you?

    Config is a detective control. It reports the violation (and can start a remediation) but does not stop the change when it happens.

Summary

  • CloudTrail records API calls (who, what, when, from where). Management events are on by default, data events are opt-in, and Event history keeps 90 days.
  • Create a trail for long-term storage in S3, enable integrity validation, and protect the bucket from edits.
  • AWS Config keeps the configuration history of resources and evaluates managed or custom rules as COMPLIANT or NON_COMPLIANT, with optional remediation. It detects, it does not prevent.
  • X-Ray traces a request across services using trace IDs, segments and subsegments, builds a service map and samples requests to control cost.
  • Use CloudTrail for "who", Config for "what and compliant?", X-Ray for "where is the time going?" and CloudWatch for "how is it behaving now?".

Next: Systems Manager and Day-Two Operations