Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 45 ยท Production Practice

Common Mistakes and Troubleshooting

Avoid the classic Google Cloud mistakes: leaked keys, public buckets, surprise bills, permission denied and quota errors, forgotten resources, and cleanup.

Most Google Cloud disasters are not exotic. They are the same handful of mistakes, made by new and experienced teams alike: a service account key pushed to a public repository, a bucket opened to everyone "just for a minute", a test cluster left running over a holiday. The good news is that each one has a cheap, boring fix that you can set up in advance.

This lesson lists the mistakes that cause the most pain, shows how to recognize each one, and ends with a small checker that scans a configuration for them. The real console, gcloud and billing data need an account, so the commands are read-only; the checker is a stand-in that shows the logic of such an audit.

Five ways to get hurt

Leaked keys Key files in code or chat.
Public buckets allUsers on a bucket.
Runaway bills No budget, no limits.
Forgotten resources Idle VMs and disks keep billing.

All five are preventable with a little setup.

1 / 2

Leaked credentials

A service account key is a JSON file containing a private key. Anyone who has the file is that service account until the key is deleted. Keys leak through Git commits, container images, chat messages, screenshots and shared drives, and automated scanners find public ones within minutes. The usual result is attackers starting expensive VMs for cryptocurrency mining on your bill.

Prevention, in order of value:

  • Do not create keys. Attach a service account to the workload (Cloud Run, a VM, GKE with Workload Identity) so the platform supplies short-lived tokens. For local work use gcloud auth application-default login.
  • Enable the organization policy that disables key creation (see organization policies).
  • Keep application secrets in Secret Manager, never in source, images or environment files in Git.
  • Add secret scanning to your repository and a .gitignore entry for *.json key files and .env.

If a key does leak, act in this order: disable or delete the key, check the audit logs for what it did, delete any resources you did not create, rotate anything the account could read, then fix the root cause.

Terminal
# List keys on a service account, then delete the leaked one
gcloud iam service-accounts keys list \
  --iam-account[email protected]
gcloud iam service-accounts keys delete KEY_ID \
  --iam-account[email protected]

Public buckets and over-broad access

The principals allUsers (anyone on the internet) and allAuthenticatedUsers (anyone with any Google account, which is not "people in my company") are the usual cause of data leaks. A bucket that holds a website must be readable by everyone; a bucket with customer exports must not be.

Safer habits:

  • Turn on public access prevention for buckets (or enforce it by organization policy) and make exceptions explicit.
  • Use uniform bucket-level access so IAM is the only control, not a mix of IAM and per-object ACLs.
  • Share one file with a time-limited signed URL instead of making a bucket public (see Cloud Storage).
  • Avoid the basic roles Owner, Editor and Viewer for people and services; they grant far more than needed. Least privilege means a predefined or custom role on the narrowest resource that works.
Terminal
# Find out who can access a bucket, and look for allUsers
gcloud storage buckets get-iam-policy gs://my-exports-bucket

# Block public access for the bucket
gcloud storage buckets update gs://my-exports-bucket --public-access-prevention

Runaway bills and forgotten resources

Cloud billing is pay-as-you-go (what is cloud computing), so anything left running keeps costing money. The typical offenders:

Forgotten thing Why it keeps billing
Stopped VM's disks A stopped VM stops the compute charge, but its persistent disks and reserved IPs still cost money
Static external IP not attached Reserved addresses that are not in use are charged
GKE cluster or Cloud SQL instance Billed for the hours it exists, used or not
Load balancer forwarding rule Billed per rule and per traffic, even with no backends
Snapshots and old container images Storage accumulates silently
BigQuery scans SELECT * on a huge table scans every byte and, on-demand, you pay for them

Defenses: set a budget with alerts on day one (budgets), put every experiment in its own project so you can delete the whole thing, label resources with owner and purpose, and set max instances on Cloud Run and autoscalers. Remember that a budget alert only notifies; it does not stop spending unless you build automation for that.

"Permission denied" and "quota exceeded"

When a call fails with 403 PERMISSION_DENIED, work through a short checklist:

  1. Who is calling? Run gcloud auth list. Often it is your personal account when you assumed it was a service account, or the reverse.
  2. Which project? gcloud config get project. A correct role on the wrong project still fails.
  3. Is the API enabled? A message like "API has not been used in project โ€ฆ before or it is disabled" needs gcloud services enable.
  4. Which permission is missing? The error names it. Find a predefined role containing it and grant that role on the right resource. See IAM.
  5. Is something above the project denying it? An organization policy, a deny policy, or VPC Service Controls can block a call even when IAM allows it.
  6. Did you wait? IAM changes usually take effect within a minute or two, sometimes longer.
Terminal
gcloud auth list
gcloud config get project
gcloud services enable run.googleapis.com
gcloud projects add-iam-policy-binding my-project-id \
  --member="serviceAccount:[email protected]" \
  --role="roles/cloudsql.client"

A quota is a limit on how much of something a project may use, such as CPUs per region or API requests per minute. Quota errors mention quota or RESOURCE_EXHAUSTED and are not bugs: view usage under IAM & Admin โ†’ Quotas, request an increase, or spread work across regions. A related but different error is "zone resource pool exhausted": Google has no capacity of that machine type in that zone right now, so try another zone or machine type.

Other repeat offenders

  • Everything in one project. Production, tests and experiments together make permissions and cleanup hard. See projects and the hierarchy.
  • Wrong region. Resources far from users or from each other add latency and cross-region data charges (regions and zones).
  • Default service account. Compute Engine's default account often has a broad role; create dedicated accounts instead.
  • Open firewall rules. 0.0.0.0/0 on SSH or database ports invites scanning; use IAP tunneling or private IPs.
  • No backups tested. A backup you have never restored is a hope, not a backup.

Cleanup

When an experiment ends, delete the project: gcloud projects delete my-sandbox-id removes (after a grace period of about 30 days during which it can be restored) everything in it. If the project must stay, list what exists and delete deliberately.

Terminal
gcloud compute instances list
gcloud compute disks list
gcloud compute addresses list
gcloud sql instances list
gcloud container clusters list
gcloud storage buckets list

A misconfiguration checker (stand-in)

Real audit tools such as Security Command Center and Cloud Asset Inventory read your live resources. This stand-in does the same kind of thing on plain objects: each rule inspects a resource description and returns a finding. The data and rules are invented for teaching, not a real API.

JS Stand-in: scan resources for common mistakes

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the public site-assets bucket produced no finding: being public is correct for a website, which is why real tools need the purpose or an allow-list to avoid false alarms. The stopped-VM finding is the quiet one: nobody is using the machine, yet the disk keeps charging.

A permission-denied triage (stand-in)

This second stand-in turns the checklist above into code. Given what you know about a failing call, it names the most likely cause. It does not call Google; it just encodes the order in which to look.

JS Stand-in: triage a 403

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A stopped Compute Engine VM has been off for a month. What still costs money?

    Stopping a VM ends the compute charge, but attached disks and reserved addresses keep billing until you delete them.
  2. 2. What is the best way to let a Cloud Run service call other Google Cloud APIs?

    Attached service accounts give short-lived credentials without any key file that could leak.
  3. 3. Why is allAuthenticatedUsers on a bucket dangerous?

    It sounds like 'logged-in users of my organization' but covers every Google account in the world.
  4. 4. You get PERMISSION_DENIED even though the role looks right. What should you check?

    A correct role on the wrong project or identity, a disabled API, or a blocking policy all produce the same error.

Summary

  • Avoid service account keys; use attached accounts, and if a key leaks, delete it first and audit afterwards.
  • Never grant allUsers or allAuthenticatedUsers unless the data is truly public; prefer signed URLs and public access prevention.
  • Set a budget alert, put experiments in their own projects, and delete what you stop using, including disks and IPs.
  • For 403 errors check identity, project, enabled API, missing permission and policies; quota errors are limits you can raise or avoid.
  • Deleting a whole project is the cleanest cleanup.

Next: Project: Host a Static Website