Skip to content
Tutorials Center
My learning

Topics

Lesson 62 of 78 ยท Operations and Observability

Troubleshooting Playbook

A practical Kubernetes troubleshooting playbook: Pending, ImagePullBackOff, CrashLoopBackOff, OOMKilled, empty Service endpoints, DNS failures and kubectl debug.

Sooner or later a pod will not start, a Service will not answer, or a deploy will hang halfway. The good news is that Kubernetes failures are very regular: the same dozen problems cause most incidents, and each one leaves clear evidence in a status field or an event. This lesson is a playbook. You learn what each status means, where to look and what to do.

All kubectl commands and manifests here are read-only, because this site has no cluster. The runnable Python examples are stand-ins: a diagnostic decision tree over pod status, and a check that explains why a Service has no endpoints. They encode the same reasoning you would do by hand.

The three first commands

Whatever is wrong, start by narrowing down with three commands, in this order.

Terminal
kubectl get pods -o wide                  # status, restarts, node, IP
kubectl describe pod web-7c9d8b5f6-abcde  # spec, conditions, and the Events at the bottom
kubectl logs web-7c9d8b5f6-abcde          # what the app printed
  • get shows the status column (for example Pending, CrashLoopBackOff). It is a summary, not a phase: more on that below.
  • describe ends with an Events section that is often the whole answer: "0/3 nodes are available: 3 Insufficient memory", "Back-off pulling image", "Liveness probe failed".
  • logs shows the container's stdout and stderr. For a container that already crashed, use --previous to see the last run's output.

Events are stored for only about an hour. To see recent ones across the namespace, sorted by time:

Terminal
kubectl get events --sort-by=.lastTimestamp
kubectl get events --field-selector type=Warning

Where to look first

get pods Read the status and restart count.
describe Read the Events at the bottom.
logs Use --previous after a crash.
debug Open a shell next to a broken pod.
1 / 2

Pending: the pod is not placed

A pod stays Pending when the scheduler cannot find a node, or when something it needs is missing. The Events tell you which:

  • 0/3 nodes are available: 3 Insufficient cpu means the requests do not fit anywhere. Lower the request, add nodes, or let the cluster autoscaler add one (how the scheduler works).
  • node(s) had untolerated taint means nodes are reserved for something else (taints and tolerations).
  • didn't match Pod's node affinity/selector points to a typo in a nodeSelector or affinity rule.
  • pod has unbound immediate PersistentVolumeClaims means the claim has no matching volume or StorageClass (PersistentVolumes).
  • A namespace quota may block creation entirely; then the ReplicaSet shows exceeded quota events and no pod exists at all.

ImagePullBackOff and ErrImagePull

The kubelet could not pull the image. describe shows the real error. The usual causes are a typo in the image name or tag, a private registry without an imagePullSecret, an expired registry token, or a node with no network path to the registry. Check the exact image string first; it is the most common cause by far. You can test pulling from a node or with a throwaway pod.

Terminal
kubectl describe pod web-7c9d8b5f6-abcde | grep -A5 -i "failed"
kubectl get pod web-7c9d8b5f6-abcde -o jsonpath='{.spec.containers[*].image}'

CrashLoopBackOff

The container starts and exits, again and again. Kubernetes restarts it with an increasing delay (10 s, 20 s, 40 s, up to 5 minutes), which is what BackOff means (pod lifecycle). The status is not the cause; it only tells you the restart loop is running. Find the cause in:

  1. kubectl logs <pod> --previous for an exception or a missing environment variable.
  2. kubectl describe pod, field Last State: look at the Exit Code and Reason.
  3. Failing liveness probes, which kill a healthy but slow application (probes).

Useful exit codes: 1 is an application error, 127 is "command not found" (a wrong command or image), 137 is SIGKILL (often the OOM killer), and 143 is SIGTERM.

OOMKilled

Last State: Terminated, Reason: OOMKilled, Exit Code: 137 means the container went over its memory limit and the kernel killed it (requests and limits). Raise the limit, or fix a leak. Remember that the limit counts the whole container, not just your heap: a JVM with -Xmx equal to the limit will be killed. Memory is not compressible like CPU: a container over its CPU limit is merely throttled and slowed, but one over its memory limit dies.

Running but not Ready, and Services with no endpoints

A pod can be Running and still receive no traffic because its readiness probe fails: it shows 0/1 in the READY column. Services only send traffic to ready pods (Services). If a Service has no endpoints, check it in this order:

Terminal
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl get svc web -o jsonpath='{.spec.selector}'
kubectl get pods --selector app=web --show-labels

The causes are almost always one of: the Service selector does not match the pod labels, the pods are not Ready, or targetPort is wrong so the connection is refused although the endpoint exists.

DNS failures

If a pod cannot resolve web.shop.svc.cluster.local, test from inside the cluster. Check that CoreDNS pods run in kube-system, that the pod's /etc/resolv.conf points to the cluster DNS Service, and that a network policy is not blocking UDP and TCP port 53 egress (DNS and discovery).

Terminal
kubectl run tmp --rm -it --image=busybox:1.37 --restart=Never -- nslookup web.shop
kubectl -n kube-system get pods -l k8s-app=kube-dns

kubectl debug and ephemeral containers

Many production images have no shell or tools at all (distroless, scratch). You cannot exec into them to run curl or nslookup. Ephemeral containers solve this: kubectl debug adds a temporary container with your tools into a running pod, sharing its network and, optionally, its process namespace.

Terminal
# Add a debug container to a running pod and attach to it
kubectl debug -it web-7c9d8b5f6-abcde --image=busybox:1.37 --target=web

# Debug a crashing pod by making a copy with a different command
kubectl debug web-7c9d8b5f6-abcde -it --copy-to=web-debug --container=web -- sh

# Open a shell on a node (mounts the node's filesystem at /host)
kubectl debug node/worker-1 -it --image=busybox:1.37

Ephemeral containers cannot be removed from a pod once added; they last until the pod is replaced. Copying a pod is safer when you need to change its command without touching the original.

A decision tree as code

Experienced operators follow a mental flowchart. The stand-in below encodes one as a function from a pod's status fields to a diagnosis and the next command to run. It is simple on purpose: real pods have more fields, but the shape of the reasoning is the same.

Python A diagnostic decision tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that web-1 is reported as an image problem even though its phase is Pending: the container waiting reason is more specific than the phase, so the function must check it first. Real clusters work the same way: the phase is coarse, the container state and events carry the detail.

Why a Service has no endpoints

The second stand-in explains an empty Service by comparing the selector with pods and checking readiness and ports, in the same order that the endpoints controller works.

Python Explain an empty Service

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only web-a serves. The other three each fail for a different reason. Real EndpointSlices omit pods that are not ready, but they do not check your port: a wrong targetPort still produces an endpoint that refuses connections, which is why that case is tricky. The stand-in checks the port to show the symptom, but you must test it yourself with a request.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A pod's status is CrashLoopBackOff. What is the best first step to find the cause?

    CrashLoopBackOff only says that the restart loop is running. The previous container's logs and its exit code or reason show why it died.
  2. 2. What does Exit Code 137 with Reason OOMKilled mean?

    137 means killed by SIGKILL, and OOMKilled shows the kernel did it because of the memory limit. CPU limits only throttle, they do not kill.
  3. 3. A Service has no endpoints but the pods are Running. Which cause is most likely?

    Endpoints are built from pods that match the selector and are Ready. A label typo or a failing readiness probe leaves the list empty.
  4. 4. Why would you use an ephemeral container with kubectl debug?

    Distroless images have no shell. An ephemeral container brings in tools and shares the pod's network, without changing the original spec.

Summary

  • Start with kubectl get pods, describe (read the Events) and logs (use --previous after a crash).
  • Pending means not scheduled or waiting for a resource: read the scheduler event for resources, taints, affinity or PVCs.
  • ImagePullBackOff is a name, tag, credentials or network problem; CrashLoopBackOff is a symptom, and the exit code and previous logs tell you the cause.
  • OOMKilled (exit 137) means the memory limit was exceeded; CPU limits only throttle.
  • An empty Service means a selector mismatch, pods that are not Ready, or a wrong port; DNS issues come from CoreDNS, resolv.conf or network policies.
  • kubectl debug with ephemeral containers lets you inspect pods that have no shell.

Next: Cluster Operations and Upgrades, where you keep the cluster itself healthy and current.