Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Administration

Cluster Monitoring, Metrics and Alerts in OpenShift

Use the built-in OpenShift monitoring stack: Prometheus, Alertmanager, PromQL basics, user workload monitoring, ServiceMonitors, PrometheusRules and alert routing.

OpenShift installs a complete monitoring stack: Prometheus collects metrics, Alertmanager routes alerts, Grafana-style dashboards are in the web console under Observe, and the Cluster Monitoring Operator keeps it all configured. You get node, pod and platform metrics immediately. For your own applications you switch on user workload monitoring.

The product is not available here. Commands and YAML are read-only; the Python stand-ins evaluate PromQL-like queries and alert rules on sample data.

What is monitored by default

The openshift-monitoring project runs Prometheus, Alertmanager, node-exporter, kube-state-metrics and more. Platform alerts, such as KubeNodeNotReady, etcdInsufficientMembers or KubePodCrashLooping, are pre-defined.

Terminal
oc get pods -n openshift-monitoring
oc adm top nodes
oc adm top pods -n shop

In the console: Observe > Dashboards (cluster and workload dashboards), Observe > Metrics (run PromQL), Observe > Alerting (firing alerts, silences, rules).

Metrics and PromQL

Prometheus stores time series identified by a metric name and labels, like container_cpu_usage_seconds_total{namespace="shop", pod="shop-api-1"}. Metric types:

  • Counter: only goes up (total requests). Use rate() to see change per second.
  • Gauge: goes up and down (memory in use).
  • Histogram: buckets of observations, for latency percentiles.

Typical queries:

PROMQL
sum(rate(container_cpu_usage_seconds_total{namespace="shop"}[5m])) by (pod)
sum(container_memory_working_set_bytes{namespace="shop"}) by (pod)
sum(rate(http_requests_total{job="shop-api", code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="shop-api"}[5m]))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

rate(counter[5m]) is the per-second increase averaged over five minutes. Selecting by regular expression uses =~.

Python Stand-in: what rate() computes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

User workload monitoring

To scrape your own applications, enable the second Prometheus instance in openshift-user-workload-monitoring:

YAML
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

Then your app exposes /metrics (Prometheus text format) and you declare how to scrape it with a ServiceMonitor:

YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: shop-api
  namespace: shop
spec:
  selector:
    matchLabels:
      app: shop-api
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

The service must carry the matching label and a named port. Use PodMonitor when there is no service. Targets appear under Observe > Targets; check there when metrics are missing.

Alerting rules

A PrometheusRule defines alerts as PromQL expressions that must stay true for some time:

YAML
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: shop-alerts
  namespace: shop
spec:
  groups:
    - name: shop.rules
      rules:
        - alert: ShopApiHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="shop-api",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="shop-api"}[5m])) > 0.05
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "More than 5% of shop-api requests fail"

An alert goes through states: inactive, pending (condition true but not yet for the for duration) and firing. The for clause suppresses short blips.

Python Stand-in: pending then firing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Alertmanager: routing

Alertmanager groups, de-duplicates and routes alerts to receivers (email, Slack, PagerDuty, webhooks), using labels such as severity and namespace. Configure it in the alertmanager-main secret in openshift-monitoring (or the console under Administration > Cluster Settings > Alertmanager):

YAML
route:
  receiver: team-chat
  group_by: [alertname, namespace]
  routes:
    - matchers: ['severity="critical"']
      receiver: pager
receivers:
  - name: team-chat
    slack_configs:
      - channel: "#ops"
        api_url_file: /etc/alertmanager/secrets/slack-url
  - name: pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-key

Silences mute an alert for a period, for example during planned maintenance, without changing its rule. Keep the secret values for receivers in Secrets, never in plain config.

Python Stand-in: route alerts to receivers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Habits that make monitoring useful

  • Alert on symptoms users feel (errors, latency), not on every CPU spike.
  • Give each alert a summary and a runbook link.
  • Set retention and storage for Prometheus with persistent volumes, or metrics vanish when a pod restarts.
  • Watch the monitoring stack itself: a down Prometheus means silent failures.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What do you enable to scrape metrics from your own applications?

    User workload monitoring runs a second Prometheus for project metrics.
  2. 2. What does rate(http_requests_total[5m]) return?

    rate() turns a growing counter into a per-second change.
  3. 3. What is the purpose of for: 10m in an alert rule?

    Until the duration passes the alert is pending, which suppresses short blips.
  4. 4. What tells Prometheus how to scrape a service?

    ServiceMonitor selects services and describes the port, path and interval.

Summary

  • OpenShift ships Prometheus, Alertmanager and dashboards; use Observe in the console.
  • PromQL works on labeled time series; counters need rate().
  • Enable user workload monitoring, expose /metrics, add a ServiceMonitor.
  • PrometheusRules define alerts; Alertmanager routes them and supports silences.
  • Alert on user-visible symptoms and keep Prometheus data on persistent storage.

Next: Troubleshooting pods and reading logs