Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Administration

Troubleshooting Pods, Events and Logs in OpenShift

A systematic guide to fixing broken workloads: pod states such as Pending, CrashLoopBackOff and ImagePullBackOff, oc logs, events, oc debug, must-gather and node problems.

Something is broken: the route shows an error, the pod will not start, or the app restarts every minute. Good troubleshooting is a routine, not guesswork. You start from the symptom, read what the cluster already knows (status, events, logs), and narrow the cause layer by layer. This lesson gives you that routine and the exact oc commands for each step.

The commands are read-only examples (no cluster on this site). The Python stand-ins are small diagnostic helpers that mimic how you reason about the output.

Step 1: what state is it in?

Terminal
oc get pods
oc get pods -o wide                 # node, IP
oc get events --sort-by=.lastTimestamp | tail -20
oc describe pod shop-api-6d8f7-abcde

The STATUS column tells most of the story:

Status Meaning Look at
Pending Not scheduled or waiting for a volume Events: insufficient CPU, unbound PVC, taints, quota
ContainerCreating Image pulling or volume mounting Events: mount errors, pull progress
ImagePullBackOff / ErrImagePull Cannot pull the image Image name/tag, registry secret, network
CrashLoopBackOff Container starts, exits, restarts with delays oc logs --previous, exit code
Error / Completed A job-like container exited Exit code, logs
OOMKilled (in last state) Memory limit exceeded Raise limit or fix leak
Running but not ready Readiness probe failing Probe path/port, app logs
Terminating (stuck) Finalizers or a stuck node oc describe, node status
Python Stand-in: map a status to the next command

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: events and describe

oc describe pod ends with an Events list, the first place to look. Typical messages:

TEXT
Warning  FailedScheduling  0/6 nodes are available: 3 Insufficient memory, 3 node(s) had untolerated taint.
Warning  FailedMount       MountVolume.SetUp failed for volume "data": persistentvolumeclaim "x" not found
Warning  BackOff           Back-off restarting failed container
Warning  Unhealthy         Readiness probe failed: HTTP probe failed with statuscode: 503

Events expire after about an hour, so collect them early.

Step 3: logs

Terminal
oc logs deploy/shop-api                 # current container, one of the pods
oc logs shop-api-6d8f7-abcde -c sidecar # a specific container
oc logs shop-api-6d8f7-abcde --previous # the crashed instance
oc logs -f deploy/shop-api --tail=100   # follow
oc logs -l app=shop-api --since=15m     # all pods with a label

--previous is the key to CrashLoopBackOff, because the current container has no output yet. Exit codes help: 1 generic error, 137 killed (OOM or SIGKILL), 139 segmentation fault, 143 SIGTERM.

Python Stand-in: decode an exit code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 4: get inside

Terminal
oc rsh shop-api-6d8f7-abcde                      # shell in the running container
oc exec shop-api-6d8f7-abcde -- env | sort
oc port-forward svc/shop-api 8080:80             # test from your laptop
oc debug deploy/shop-api                         # copy of the pod with a shell
oc debug node/worker-1                           # shell on a node (chroot /host)
oc debug deploy/shop-api --as-user=0             # needs an SCC that allows root

oc debug starts a copy of the pod with the entry point replaced by a shell, so you can inspect a pod that crashes at startup. For a missing tool in a minimal image, debug with another image: oc debug deploy/shop-api --image=registry.access.redhat.com/ubi9/ubi.

Step 5: layer by layer for "I cannot reach my app"

flowchart TD
  A["Browser fails"] --> B{"Route exists and admitted?"}
  B -- no --> B1["oc get route; check host conflicts"]
  B -- yes --> C{"Service has endpoints?"}
  C -- no --> C1["selector, readiness, pod state"]
  C -- yes --> D{"Pod answers inside the cluster?"}
  D -- no --> D1["oc logs; oc rsh; port and bind address"]
  D -- yes --> E["NetworkPolicy, TLS mode or router logs"]

Work from the outside in, one layer at a time.

A frequent cause: the app listens on 127.0.0.1 instead of 0.0.0.0, so the pod cannot be reached from other pods even though it runs fine.

Step 6: events about the platform

If many things fail at once, look at the platform:

Terminal
oc get clusteroperators              # anything Degraded?
oc get nodes                         # NotReady?
oc describe node worker-1            # conditions: MemoryPressure, DiskPressure
oc adm top nodes
oc get csr                           # pending certificate requests

Collect a diagnostic archive for Red Hat support or a colleague:

Terminal
oc adm must-gather
oc adm inspect ns/shop

must-gather writes resource dumps and logs to a local directory. It may contain sensitive information, so review before sharing.

A triage helper

Python Stand-in: summarize pods from `oc get pods` output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A pod is in CrashLoopBackOff. Which command shows why the last run failed?

    --previous prints logs of the crashed container instance.
  2. 2. Where do you read why a pod is stuck Pending?

    Scheduler messages about resources, taints or volumes appear in the pod events.
  3. 3. Exit code 137 usually means

    137 = 128 + 9 (SIGKILL); check the last state for OOMKilled.
  4. 4. What does oc debug deploy/NAME do?

    It runs a debug copy of the pod so you can inspect it even when the real container crashes.

Summary

  • Read status, then events, then logs, then get inside the container.
  • --previous and exit codes explain crash loops; describe explains Pending.
  • Test the path outside-in: route, service endpoints, pod, policies.
  • oc debug and oc rsh help inspect; must-gather collects evidence.
  • Check cluster operators and nodes when many workloads fail at once.

Next: Cluster updates and machine configuration