Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 45 ยท Operations and Delivery

Cloud Logging, Monitoring and Alerting

Learn Cloud Logging, Cloud Monitoring, log-based metrics, dashboards, alerting policies, uptime checks, SLOs and Error Reporting, with a runnable logger and alert rule.

An application that nobody can observe is an application that fails quietly. Once you run code on Google Cloud, you need to answer three questions at any hour: what happened (logs), how is it doing (metrics), and who should wake up (alerts). Google Cloud's operations suite covers these with Cloud Logging, Cloud Monitoring and Error Reporting, and most of it works from the first minute without installing anything.

This lesson shows how a log entry is shaped, how logs turn into metrics, how to build dashboards and alerting policies, and how to describe reliability with a service level objective. The stand-ins at the end let you run a structured logger and an alert rule yourself.

Three questions, three tools

Logging What happened, in detail.
Monitoring How the system is doing over time.
Alerting Tell a human when it matters.
Error Reporting Group crashes into a short list.

Logs explain, metrics measure, alerts call you.

1 / 2

Log entries and severity

Almost everything on Google Cloud writes to Cloud Logging: platform and audit logs automatically, and your own application through its standard output. A log entry has a timestamp, the resource that produced it (for example a Cloud Run revision), a severity, and a payload, which is plain text or, much better, structured JSON.

The severity levels, from least to most serious, are DEFAULT, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT and EMERGENCY. Use them with care: if everything is ERROR, nothing stands out.

On Cloud Run, Cloud Run functions, App Engine and GKE, a line printed to stdout becomes a log entry. If that line is a JSON object, Logging parses it: the severity and message keys are recognized, and your other keys become searchable fields.

JavaScript
// Node.js: one JSON object per line on stdout
console.log(JSON.stringify({
  severity: "ERROR",
  message: "payment failed",
  orderId: 103,
  code: "card_declined",
}));

Structured logs let you ask exact questions later (jsonPayload.code="card_declined") instead of searching text with guesses. Include identifiers, durations and the name of the operation, but never passwords, tokens or full card numbers: logs are widely readable and kept for a long time.

You search in the Logs Explorer with a small query language:

TEXT
resource.type="cloud_run_revision"
severity>=ERROR
jsonPayload.code="timeout"

Where logs live matters for cost and retention. Entries go into log buckets: the _Required bucket holds admin audit logs for a fixed long period at no charge, and the _Default bucket holds the rest, with a default retention of 30 days that you can change (longer retention costs more). The Log Router copies entries that match a filter to other destinations through sinks: Cloud Storage for archives, BigQuery for analysis, or Pub/Sub for streaming. You can also use exclusion filters to drop noisy entries before they are stored, which is a common way to control the bill.

Log-based metrics

Sometimes the number you need only exists in a log: how many "payment failed" entries per minute? A log-based metric turns matching entries into a time series. A counter metric counts entries; a distribution metric extracts a number (such as a latency) from each entry and records its spread. Once defined, it appears in Monitoring like any other metric and can drive charts and alerts.

Terminal
# Read-only: needs a real project.
gcloud logging metrics create payment_failures \
  --description="Failed payments" \
  --log-filter='severity>=ERROR AND jsonPayload.message="payment failed"'

Keep the labels on such metrics few and low-cardinality (for example an error code, not an order ID), or the number of separate time series explodes.

Cloud Monitoring: metrics and dashboards

Cloud Monitoring stores metrics: numbers sampled over time. Google services publish many on their own, such as request counts and latency for Cloud Run, CPU for VMs and queue depth for Pub/Sub, with no setup. You can add custom metrics from your code, and use Managed Service for Prometheus if your team already speaks Prometheus.

A dashboard is a page of charts for the metrics you care about. Useful dashboards are short: traffic, errors, latency and saturation (the "four golden signals") for each important service. Google also provides ready-made dashboards for its own products, which is a good place to start, and you can describe your own as code so that every environment gets the same one.

A chart shows one metric and aggregates it: for instance, request count summed across instances and aligned to one-minute buckets, or the 95th percentile of latency. Prefer percentiles to averages for latency: an average hides the slow requests that real users actually notice.

Alerting policies and uptime checks

An alerting policy watches metrics and acts on a condition, such as "error rate above 5% for 3 minutes". A good policy has three parts:

  1. A condition: the metric, a threshold and a duration. The duration stops one noisy data point from waking someone.
  2. Notification channels: email, SMS, mobile push, Slack, PagerDuty or Pub/Sub or webhooks that start automated responses.
  3. Documentation: a short note in the alert telling the person what it means and where to start looking.

An incident opens when the condition is met and closes when it stops being met. Alert on symptoms users feel (errors, slowness, unavailability) more than on causes (CPU at 80%), because many causes are harmless and many problems have causes you did not predict.

An uptime check is a probe: Google sends an HTTP, HTTPS or TCP request to your endpoint from several locations around the world every minute or so, and records whether it answered correctly and how fast. Attach an alert to it and you learn about an outage before your customers do. Remember that it tests the public path, including DNS and the load balancer.

Service level objectives

A service level indicator (SLI) is a measurement of good service, for example "the fraction of requests that return a non-error response in under 500 ms". A service level objective (SLO) is a target for it over a period, such as "99.5% over 28 days". The difference between perfect and the target is your error budget: the amount of failure you can afford. If you have plenty of budget left, ship changes boldly; if it is nearly spent, slow down and fix reliability.

Monitoring can define SLOs on services and alert on burn rate, how quickly the budget is being used. A fast burn pages someone now; a slow burn opens a ticket. This is better than a fixed threshold because it ties alerts to what users are promised.

Error Reporting

Error Reporting collects exceptions and stack traces from your logs, groups the same error together, counts occurrences, shows when it first appeared and can notify you of new error types. Printing a stack trace as an ERROR log entry, or using the client library, is enough on most platforms. Instead of scrolling through ten thousand lines, you see "NullPointerException in checkout, 412 times since the 14:05 deploy".

Logging, Monitoring and alerts all cost money at scale: ingestion past a free monthly allowance, retention beyond the defaults and custom metrics are billed. Check the current price list, exclude noisy debug logs in production, and watch your billing for surprises.

Stand-in: a structured logger and queries

Cloud Logging cannot run on this page. This stand-in is a small logger that writes JSON objects (the same shape you would print on Cloud Run), keeps them in memory with a counter instead of real timestamps, and answers queries similar to severity>=WARNING. It is not the Logging query language or API, only the idea that structured fields make questions easy.

JS Stand-in: structured logs and filters

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last part is a log-based counter in miniature: the same entries, grouped and counted, become a metric you could chart or alert on.

Stand-in: an alert rule and an error budget

The second model feeds one data point per minute (requests and errors) through a rule: "error rate above 5% for 3 minutes in a row". The rule opens an incident when the streak is long enough and closes it when the rate returns to normal. At the end it computes the error budget for a 99.5% SLO. This imitates the logic of an alerting policy; real policies work on aligned time series and have more options.

JS Stand-in: alert rule with a duration, and an SLO budget

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Two things stand out. The alert did not fire at minute 3 (one bad minute) but at minute 5, after the third in a row, which is the point of a duration. And the incident used more than ten times the error budget of this eight-minute sample, which is exactly the kind of fact an SLO makes visible.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is structured JSON logging better than plain text on Cloud Run?

    Logging parses JSON lines from stdout, using severity and message and exposing other keys as fields.
  2. 2. What does a log-based counter metric do?

    It turns matching entries into a time series that dashboards and alerting policies can use.
  3. 3. Why do alerting conditions usually include a duration?

    Requiring the condition to hold for a period filters out brief spikes.
  4. 4. A 99.9% SLO over a period leaves what for failures?

    The gap between 100% and the target is the error budget: here 0.1% of requests may fail.
  5. 5. What does Error Reporting do?

    It aggregates identical errors, shows first-seen time and counts, and can notify you of new errors.

Summary

  • Logs explain, metrics measure and alerts notify; Cloud Logging, Monitoring and Error Reporting cover each.
  • Write structured JSON logs with a severity to stdout, and never log secrets.
  • Log-based metrics turn log entries into charts and alerts; sinks and exclusions control where logs go and what they cost.
  • Alert on user-visible symptoms, with a duration, a notification channel and a short note on what to do.
  • Uptime checks watch your public endpoint; SLOs and error budgets turn reliability goals into numbers.

Next: Infrastructure as Code with Terraform