Skip to content
Tutorials Center
My learning

Topics

Lesson 75 of 78 ยท Practice

Exercises: Scheduling, Security and Troubleshooting

Eight scenario exercises with full solutions: pods stuck in Pending, taints, selectors, Pod Security, RBAC, probes, PodDisruptionBudgets and broken manifests to diagnose.

The basic exercises asked you to write manifests from a description. Real work is the opposite: somebody hands you a broken manifest, a pod that will not start or a Forbidden error, and you have to work out why. This lesson trains that skill with eight scenarios. Each one shows the symptom and the evidence, asks you to find the cause, and only then shows the solution.

There is no cluster on this site, so all manifests and kubectl output are read-only; the output was written by hand to match what a real cluster prints. Three runnable Python stand-ins let you practice the reasoning itself: a tiny scheduler that explains why each node rejected a pod, a Pod Security checker and a Service selector tester. They are models of the real logic, not Kubernetes.

A method for every scenario

Observe Read status, events and logs before changing anything.
Locate Which layer failed - scheduling, image, start, network or access?
Prove Confirm the cause with one more command.
Fix once Change the manifest, not the live object.

Try each exercise on paper first, then open the solution.

1 / 2

If a word below is unfamiliar, the lesson that explains it is linked in the solution. A good order is to read the symptom, write down your guess, and only then continue.

Exercise 1: The pod that stays Pending

A team added a GPU node pool. Their training job never starts:

TEXT
$ kubectl get pod trainer-0
NAME        READY   STATUS    RESTARTS   AGE
trainer-0   0/1     Pending   0          12m

$ kubectl describe pod trainer-0
...
Events:
  Warning  FailedScheduling  12m  default-scheduler
    0/4 nodes are available: 1 node(s) had untolerated taint
    {dedicated: gpu}, 3 node(s) didn't match Pod's node affinity/selector.

The pod spec contains nodeSelector: {accelerator: nvidia}. The GPU node carries the label accelerator=nvidia and the taint dedicated=gpu:NoSchedule. What is missing, and what should you add?

Solution. The selector narrowed the choice to the GPU node, but that node repels every pod that does not tolerate its taint. The pod needs a matching toleration:

YAML
spec:
  nodeSelector:
    accelerator: nvidia
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule

Notice the division of labor: the taint keeps ordinary pods away, the toleration merely allows the pod in, and the selector is what actually pulls it onto the GPU node. A toleration alone does not attract a pod. See Taints and Tolerations and Node Selectors, Affinity and Anti-Affinity.

Exercise 2: Which node can take it?

The event text in exercise 1 is the scheduler's summary of its filter phase. The first stand-in reproduces that phase so you can test your own scenarios. Change the pod or the nodes and see how the reasons change.

Python A scheduler that explains its refusals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Run it and compare the first two pods: the only difference is the toleration, and it flips the result from Pending to scheduled. The real scheduler also scores the nodes that pass (spreading, image locality and more); the stand-in simply picks the roomiest. Try giving web-1 a nodeSelector with a typo such as zone: bee and read the message.

Exercise 3: The typo that looks fine

YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      nodeSelector:
        disk-type: ssd
      containers:
        - name: api
          image: registry.example.com/api:1.4

All three pods are Pending. kubectl get nodes --show-labels shows the nodes carry disktype=ssd. What is wrong?

Solution. The selector key is disk-type but the node label is disktype. Kubernetes does not validate selector keys against existing labels, because labels can appear later, so the manifest is accepted and the pods wait forever. The event says didn't match Pod's node affinity/selector. Fix the key (or relabel the nodes with kubectl label node <name> disk-type=ssd) and the pods schedule on their own, because the scheduler retries Pending pods. Lesson: when "nothing matches", copy label keys from --show-labels instead of typing them.

Exercise 4: A Service with no endpoints

TEXT
$ kubectl get svc shop
NAME   TYPE        CLUSTER-IP     PORT(S)   AGE
shop   ClusterIP   10.96.41.200   80/TCP    3m

$ kubectl get endpointslices -l kubernetes.io/service-name=shop
NAME         ADDRESSTYPE   PORTS   ENDPOINTS   AGE
shop-x7k2q   IPv4          <unset> <unset>     3m

The Service has selector: {app: shop, tier: web}. The pods have the labels app=shop and tier=frontend, and all of them are Running and Ready. Why does the Service have no endpoints?

Solution. A Service selects pods by labels, and every key in its selector must match. tier: web does not equal tier=frontend, so no pod qualifies. Pod readiness is irrelevant until the labels match. The same symptom appears when the pods exist but are not Ready (their IPs are listed as not ready and receive no traffic) and when the pods are in another namespace. The stand-in below tests all three causes.

Python Why does this Service have no endpoints?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real cluster the three checks map to kubectl get pods --show-labels -n prod, kubectl get endpointslices and kubectl get pods -A. See Services and Labels, Selectors and Annotations.

Exercise 5: Rejected by Pod Security

A namespace is labelled pod-security.kubernetes.io/enforce=restricted. Applying this Deployment gives an error instead of pods:

YAML
containers:
  - name: app
    image: registry.example.com/app:2.0
    securityContext:
      runAsUser: 0
TEXT
Error from server (Forbidden): pods "app-6d8f..." is forbidden:
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false,
unrestricted capabilities, runAsNonRoot != true, seccompProfile

Rewrite the container so it passes.

Solution. The restricted profile requires four things in the container's security context, and a non-root user:

YAML
containers:
  - name: app
    image: registry.example.com/app:2.0
    securityContext:
      runAsNonRoot: true
      runAsUser: 10001
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault

seccompProfile can also be set once at pod level. The image must really run as that user, so a file it writes must be in a directory that user owns or in a volume. See Pod Security and Security Contexts. Try your own variants in the checker:

Python A mini Pod Security checker (restricted)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The checker covers the headline rules only. The real restricted profile also limits volume types, host ports and more, and the baseline level is much looser.

Exercise 6: Forbidden

A CI pipeline runs as the service account ci in namespace team-a and fails:

TEXT
Error from server (Forbidden): pods is forbidden: User
"system:serviceaccount:team-a:ci" cannot list resource "pods"
in API group "" in the namespace "team-a"

Grant it the least privilege that lets it list and get pods in that namespace, and show how to verify.

Solution. A Role defines the permission and a RoleBinding attaches it to the subject, both namespaced:

YAML
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: team-a
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-pod-reader
  namespace: team-a
subjects:
  - kind: ServiceAccount
    name: ci
    namespace: team-a
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
Terminal
kubectl auth can-i list pods -n team-a \
  --as=system:serviceaccount:team-a:ci      # yes
kubectl auth can-i delete pods -n team-a \
  --as=system:serviceaccount:team-a:ci      # no

A ClusterRoleBinding to a broad role would also have fixed the error, and would have given the pipeline access to every namespace: the wrong answer. See Authentication and RBAC.

Exercise 7: Three crashes, three causes

Match each status to its most likely cause, then to the first command you would run.

TEXT
a)  api-5c9f   0/1  CrashLoopBackOff  6   9m     last state: Terminated, Exit Code 1
b)  cache-0    0/1  OOMKilled         3   4m     last state: Terminated, Exit Code 137
c)  web-7d4b   0/1  ImagePullBackOff  0   2m

Solution.

  • a) The process starts and exits with an error, over and over; the kubelet restarts it with growing delays (up to five minutes). Run kubectl logs api-5c9f --previous to read the output of the last crashed container, because the current one has no logs yet.
  • b) Exit code 137 is 128 + 9, SIGKILL, and the reason OOMKilled says the kernel killed the process for exceeding its memory limit. Compare kubectl top pod with resources.limits.memory, then raise the limit or fix the leak.
  • c) The node cannot pull the image: a wrong tag or registry name, a private registry without an imagePullSecret, or no network route. kubectl describe pod web-7d4b shows the exact pull error in Events.

The troubleshooting lesson has a full decision tree: Troubleshooting Playbook.

Exercise 8: The rollout that never finishes

YAML
containers:
  - name: api
    image: registry.example.com/api:2.1
    ports:
      - containerPort: 8080
    readinessProbe:
      httpGet:
        path: /healthz
        port: 80
      periodSeconds: 5

After kubectl apply, kubectl rollout status deploy/api hangs on "Waiting for deployment... 1 out of 3 new replicas have been updated". The old pods keep serving. Why does the rollout stop, and why is that actually good news?

Solution. The application listens on 8080, but the probe asks port 80. Every new pod stays Running but never Ready, so the Deployment controller will not retire old pods: with the default maxUnavailable of 25%, it only removes old capacity as new pods become ready. Users never saw an outage, which is exactly what readiness gates are for. Fix the port and the rollout continues; if you cannot, kubectl rollout undo deploy/api returns to the previous ReplicaSet. See Probes and Rolling Updates and Rollbacks.

When you are stuck, compare the three ports: the container's real listening port, containerPort (informational), the probe port and the Service targetPort. Mismatched ports cause a large share of "it deploys but does not work" tickets.

Where to go from here

Repeat the exercises by breaking things on purpose in a local cluster (Running a Local Cluster): add a taint, mistype a label, drop the limits too low. Reading your own broken manifests is the fastest way to build the instinct these scenarios train.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A pod has a toleration for dedicated=gpu:NoSchedule but still lands on an ordinary node. Why?

    Taints repel, tolerations allow, selectors and affinity attract. To dedicate GPU nodes you need both a taint on the node and a toleration plus selector on the pod.
  2. 2. A Service has no endpoints although all pods are Ready. What is the most likely cause?

    Endpoints come from pods in the Service's namespace whose labels match every key of the selector and that are Ready.
  3. 3. What does exit code 137 with reason OOMKilled tell you?

    137 = 128 + 9 (SIGKILL). With OOMKilled the kernel's out-of-memory killer ended the process because it crossed the container memory limit.
  4. 4. A rollout hangs because new pods never become Ready. What happens to users?

    Rolling updates remove old pods only as new ones become ready, and Services send traffic only to Ready pods.
  5. 5. Which command shows the logs of the previous, crashed instance of a container?

    --previous (or -p) prints the output of the last terminated container, which is the one that holds the error in a CrashLoopBackOff.

Summary

  • Diagnose in layers: scheduling, image, start, readiness, network, access.
  • A Pending pod's Events list, node by node, why scheduling failed: taints, selectors, resources.
  • Taints repel, tolerations allow, selectors and affinity attract.
  • A Service selects pods in its own namespace by labels, and only Ready pods receive traffic.
  • Pod Security restricted needs non-root, no privilege escalation, dropped capabilities and a seccomp profile.
  • Grant RBAC with a namespaced Role and RoleBinding, and verify with kubectl auth can-i --as.
  • Exit code 137 plus OOMKilled means a memory limit was exceeded; use --previous for crash logs.

Next, test yourself out loud with Kubernetes Interview Questions.