Skip to content
Tutorials Center
My learning

Topics

Lesson 67 of 70 ยท Practice, interviews and cheat sheet

Common Mistakes and Troubleshooting

Debug the AWS problems everyone hits: AccessDenied, security group and route failures, Lambda timeouts and surprise bills, with runnable diagnostic stand-ins.

Almost every AWS problem you will meet has a small number of causes, and the same few mistakes show up again and again: a missing permission, a blocked port, a function that ran out of time, a resource nobody remembered to delete. This lesson gives you a method for each. The key habit is to read the symptom carefully, because the exact words of an error message tell you which layer to look at.

The diagnostic tools below are read-only commands and runnable Python stand-ins that follow the same rules as the real services. They are simplified: real AWS has more layers and more error messages than a small program can show. Use them to practice the reasoning, then apply it with the real console, CloudTrail and VPC Reachability Analyzer.

A method that works

  1. Read the exact error. AccessDenied, UnauthorizedOperation, Connection timed out, Connection refused, Task timed out after 3.00 seconds and 502 Bad Gateway are different problems.
  2. Find the layer. Identity (who are you?), permission (may you?), network (can packets arrive?), application (does the program work?), quota or cost.
  3. Change one thing at a time, and check the result before the next change.
  4. Use the evidence AWS keeps: CloudTrail for API calls, VPC Flow Logs for traffic, CloudWatch Logs for your code, and the Reachability Analyzer or policy simulator for what-if checks.
flowchart TD
    S["Something fails"] --> Q{"What does the<br/>error say?"}
    Q -- "AccessDenied" --> P["Permissions:<br/>policies, SCPs, KMS, MFA"]
    Q -- "Timed out (network)" --> N["Network path:<br/>SG, NACL, route, IP"]
    Q -- "Connection refused" --> A["Application:<br/>not listening, wrong port"]
    Q -- "Task timed out (Lambda)" --> L["Time or network:<br/>timeout, NAT, cold start"]
    Q -- "Unexpected charges" --> C["Cost:<br/>NAT, idle, transfer, logs"]

Match the symptom to a layer before touching any setting.

AccessDenied

This is the most common error. Work through the layers in the order from How Policy Evaluation Works: look for an explicit deny first, then guardrails (service control policies, permission boundaries, session policies), then check that a grant exists on the right side (identity policy, resource policy, or both for cross-account calls).

Typical causes:

  • The action is missing from the identity policy, or the resource ARN is wrong (bucket versus object ARN, a wrong region or account ID).
  • The role that is actually used is not the role you edited. After aws sts assume-role or on an EC2 instance or Lambda function, the role's permissions apply, not your user's.
  • An SCP at the organization level or a permission boundary silently limits the role.
  • For encrypted data the KMS key policy or kms:Decrypt permission is also required.
  • A policy condition does not match: MFA missing, source IP outside the allowed range, a required tag absent, or a wrong aws:SourceVpce.
  • A resource policy denies the caller, for example a bucket policy that requires TLS or a specific VPC endpoint.
  • Newly created access can take a short time to become visible everywhere.

The first command to run is always the same, to find out who you are:

Terminal
aws sts get-caller-identity

Then look up the failing call in CloudTrail. The event contains the caller, the action and the error message, which for many services says whether an explicit deny or a missing allow caused it. The simulator tests a policy without making the call:

Terminal
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/app-role \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::acme-data/reports/q1.csv

Here is a stand-in that takes a description of the situation and prints the likely causes in debugging order, like a checklist that fills itself in. It models the same layers as the earlier evaluator; the real thing also needs the policy documents.

Python AccessDenied explainer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the order of the output: the deny comes first because nothing else matters when a deny applies. The Lambda case fails even though the identity policy is fine, because the data key lives under a KMS key whose policy does not trust the role. The first scenario has two findings, a guardrail and a missing resource grant, but the guardrail is the real blocker, because fixing the grant alone would still be denied by the SCP.

Security groups, routes and "it just hangs"

A connection that hangs and times out means packets were dropped silently. A connection that is refused means a packet reached the machine and nothing was listening, which is a good sign for the network and a bad one for the app.

Check the path in order:

  1. Address. Does the instance have a public IP or Elastic IP? A private instance cannot be reached from the internet directly.
  2. Route. The subnet's route table needs 0.0.0.0/0 to an internet gateway for a public subnet, or to a NAT gateway for outbound-only private subnets.
  3. Network ACL. NACLs are stateless: inbound and outbound rules must both allow the traffic, including ephemeral return ports (1024 to 65535).
  4. Security group. Stateful: only the inbound rule matters for the reply. Check the port and the source CIDR or source security group.
  5. The operating system and the app. Is the service running and bound to 0.0.0.0 rather than 127.0.0.1? Does a host firewall interfere?

The VPC Reachability Analyzer checks steps 2 to 4 for you without sending packets, and VPC Flow Logs show ACCEPT and REJECT records (see NACLs, Flow Logs and Network Security). Here is the same walk as a stand-in:

Python Why can't I reach my instance?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each case stops at the first failing layer, exactly like real troubleshooting: you fix the first problem, retry, and only then see the next one. The "sshd not running" case is the only one that produces a refusal, which tells you to stop editing network settings and look at the server instead.

A few related mistakes: opening port 22 to 0.0.0.0/0 (use Session Manager instead, see Connecting to Instances); forgetting that security group rules reference other groups by ID, which is cleaner than CIDRs; and peering two VPCs without adding routes on both sides.

Lambda timeouts and failures

Task timed out after 3.00 seconds is the default timeout speaking: a new function has 3 seconds unless you raise it (the maximum is 15 minutes). Raising it blindly is rarely the real fix. Ask why the function is slow:

  • A VPC function without internet access. The function waits on a connection to a public API until it gives up. Add a NAT gateway or a VPC endpoint.
  • Low memory. CPU grows with memory, so a CPU-bound function at 128 MB is slow. Test 512 MB or 1,024 MB; a bigger setting is sometimes cheaper overall because it finishes faster (Lambda scaling and cold starts).
  • Cold starts with a large package or heavy imports on first use.
  • A slow dependency, such as a database without connection reuse. Create clients outside the handler so warm invocations reuse them.
  • A mismatch with the caller. API Gateway waits about 29 seconds by default, so a function that needs 40 seconds can never answer a synchronous request. Use an asynchronous pattern (queue plus status endpoint).
  • Retries. Asynchronous invocations retry on error, so a failing function can run three times. Make handlers idempotent and configure a dead-letter queue or destination.

Read the REPORT line at the end of each invocation's logs: Duration, Billed Duration, Memory Size and Max Memory Used answer most sizing questions at a glance.

Surprise bills

Cost problems are usually resources that cost money while doing nothing, or traffic that crosses a charged boundary. The common culprits:

  • NAT gateways: an hourly fee plus a charge per GB processed, easily the biggest line in a small account.
  • Idle resources: forgotten instances, load balancers with no traffic, unattached EBS volumes, old snapshots, unassociated Elastic IPs.
  • Public IPv4 addresses, which are charged hourly even when attached.
  • Data transfer: internet egress and traffic between Availability Zones or Regions.
  • CloudWatch Logs with no retention setting, or debug logging left on in production.
  • S3: incomplete multipart uploads, huge numbers of tiny requests, and versioned buckets holding every old version.
  • Provisioned capacity (DynamoDB, provisioned concurrency) that nobody uses.

The defence is a budget with alerts, cost allocation tags and a monthly look at Cost Explorer (Cost Management). This stand-in scans an inventory the way a cleanup script would:

Python Find the resources that cost money for nothing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Eight of nine resources are flagged. Only the attached 100 GB volume is fine. Real scans use AWS Trusted Advisor, Cost Explorer's rightsizing recommendations, Compute Optimizer and the Cost Optimization hub, which have actual usage data behind them.

A troubleshooting checklist

  • What is the exact error text, region, account and time?
  • Who am I? (aws sts get-caller-identity)
  • Is it permissions? Deny first, then guardrails, then grants; also KMS and conditions.
  • Is it the network? Public IP, route, NACL (both directions), security group, DNS.
  • Is it the app? Listening on the right port and interface, logs in CloudWatch.
  • Is it a limit? Service quotas, throttling (429/ThrottlingException), Lambda concurrency.
  • Is it cost? Check Cost Explorer grouped by service and by usage type.
  • What changed recently? CloudTrail and AWS Config show the history.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Your connection to an EC2 instance hangs and then times out. What is the most likely class of problem?

    A silent timeout means nothing answered, which points to a blocking firewall rule or routing. An application that is not listening typically produces an immediate "connection refused".
  2. 2. A Lambda function that worked outside a VPC now times out when calling a public API from private subnets. What is the likely fix?

    A function in a private subnet has no path to the internet. Permissions are not the problem; the missing route is.
  3. 3. An AccessDenied occurs on an S3 object encrypted with a customer managed KMS key, even though the role has s3:GetObject. What is missing?

    Reading an SSE-KMS object requires permission to use the key as well. Check both the role's policy and the key policy.
  4. 4. Which of these quietly costs money every month even when nothing uses it?

    A NAT gateway is billed per hour as long as it exists, plus per GB it processes. Idle ones are a classic source of surprise bills.
  5. 5. Why is the order of an AccessDenied investigation "explicit deny first"?

    Policy evaluation checks for explicit denies first, and they cannot be overridden. Adding more allows is useless while a deny applies.

Summary

  • Read the exact symptom first: AccessDenied, timeout, refused, Lambda timeout and unexpected charge each point to a different layer.
  • For AccessDenied, check deny, guardrails, grants, KMS and conditions in that order, and confirm which identity actually made the call.
  • Network timeouts mean dropped packets: check address, route, NACL (both directions), security group; a refusal means the app is not listening.
  • Lambda timeouts come from small timeouts, missing NAT access, low memory, cold starts and slow dependencies; match them to the caller's own limits.
  • Surprise bills come from NAT gateways, idle resources, public IPv4 addresses, data transfer, unbounded logs and incomplete uploads; budgets and tags catch them early.

Next: AWS Interview Questions