Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 78 ยท Operations and Observability

Debugging Applications in a Cluster: Port-Forward, Ephemeral Containers and Remote Debuggers

Debug a running Kubernetes app: kubectl port-forward to reach Pods privately, kubectl debug for tools, attaching IDE debuggers (Java JDWP, Node, Python, Go) and safer practices.

Troubleshooting taught you to read Pod status, events and logs to find out why a Pod is unhealthy. This lesson is about a different situation: the Pod is healthy, but your application misbehaves, and you need to look inside it. You want to call a service that is not exposed, run a tool next to a process that has no shell, or attach your IDE's debugger to code running in the cluster.

The tools are kubectl port-forward, kubectl exec, kubectl debug with ephemeral containers, and the remote debug protocols of your language. There is no cluster on this site, so every command is read-only; the two runnable examples are small Python models of port mapping and of a debugger option string.

A private door: kubectl port-forward

Most Pods are reachable only inside the cluster. kubectl port-forward opens a tunnel from a port on your machine to a port in a Pod, through the API server. No Service, Ingress or firewall change is needed, and it uses your own RBAC permissions.

Terminal
kubectl port-forward pod/web-7c9d8b5f6-abcde 8080:80
# in another terminal
curl http://127.0.0.1:8080/

The syntax is LOCAL:REMOTE. The command stays in the foreground and prints Forwarding from 127.0.0.1:8080 -> 80; press Ctrl+C to close it. You can forward to other resource types:

Terminal
kubectl port-forward deploy/web 8080:80          # picks one Pod of the Deployment
kubectl port-forward svc/web 8080:80             # picks one Pod behind the Service
kubectl port-forward -n shop pod/db-0 5432:5432  # a database, in another namespace
kubectl port-forward pod/web-abc 8080:80 9229    # several ports at once
kubectl port-forward pod/web-abc :80             # let kubectl choose a free local port

Details that surprise people:

  • One Pod only. Forwarding to a Deployment or Service resolves it to a single Pod when the command starts. There is no load balancing, and if that Pod is replaced during a rollout the tunnel breaks.
  • Localhost only by default. The tunnel listens on 127.0.0.1 (and ::1). Add --address 0.0.0.0 to share it with your network, which also exposes whatever is behind it to everyone on that network.
  • TCP only. UDP is not supported.
  • Port numbers are checked locally. If the local port is taken, the command fails immediately.

This model parses the port arguments the way kubectl does:

Python How port-forward maps ports

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use port-forward for admin pages, databases, metrics endpoints (/metrics), health checks and debug ports. Do not use it as a way to publish an application: it is a developer tool, tied to your terminal session.

Looking around: exec, logs and copy

When the image has a shell, plain exec is the quickest tool:

Terminal
kubectl exec -it web-7c9d8b5f6-abcde -- sh               # interactive shell
kubectl exec web-7c9d8b5f6-abcde -- env                  # one command
kubectl exec web-7c9d8b5f6-abcde -c sidecar -- ls /data  # a specific container
kubectl logs web-7c9d8b5f6-abcde -f --tail=50            # follow the log
kubectl logs web-7c9d8b5f6-abcde --previous              # the crashed container's last log
kubectl cp web-7c9d8b5f6-abcde:/app/dump.json ./dump.json   # copy a file out

kubectl cp needs the tar binary in the container, so it fails on minimal images. In that case use an ephemeral container (below) or write the data to a volume you can read.

Tools without a shell: kubectl debug

Hardened images (distroless, scratch) contain no shell and no tools, so exec has nothing to run. Troubleshooting introduced kubectl debug with ephemeral containers; here is how to use it for application debugging:

Terminal
# add a debug container that shares the target's process namespace
kubectl debug -it web-7c9d8b5f6-abcde --image=nicolaka/netshoot --target=web

# inside it: see the app's processes, files and network
ps aux
ls /proc/1/root/app          # the target container's filesystem, via its PID 1
curl -s localhost:8080/healthz
ss -tlnp

Key points:

  • --target=web makes the debug container join the process namespace of the web container, so you can see its processes and read its files under /proc/<pid>/root. The network namespace is shared by the whole Pod, so localhost is the app.
  • Ephemeral containers cannot be removed from a Pod and are not restarted; they last until the Pod is replaced.
  • To change how the app starts without touching the real Pod, make a copy: kubectl debug web-abc -it --copy-to=web-debug --container=web -- sh creates web-debug with the same spec but your command. Delete it when done.
  • Recent versions have --profile (general, baseline, restricted, netadmin, sysadmin) that sets the debug container's privileges. Pick the least that works; restricted fits namespaces with the restricted Pod Security level (Pod Security).
  • kubectl debug node/worker-a -it --image=busybox starts a Pod on a node with the node's filesystem mounted at /host, for node-level problems (Nodes).

Attaching a remote debugger

Sometimes logs are not enough, and you want breakpoints. Every major runtime has a remote debugging protocol: the app listens on a debug port, and your IDE connects. In Kubernetes the connection travels through port-forward, so the debug port is never published.

Java (JDWP)

Start the JVM with the Java Debug Wire Protocol agent. The easiest way, without changing the image, is the standard environment variable JAVA_TOOL_OPTIONS:

YAML
containers:
  - name: api
    image: registry.example.com/team/api:2.3.1
    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
    ports:
      - name: debug
        containerPort: 5005

Then forward the port and attach your IDE (IntelliJ: Run > Attach to Process or a "Remote JVM Debug" configuration with host localhost and port 5005):

Terminal
kubectl port-forward deploy/api 5005:5005

Two settings matter. suspend=n lets the app start normally; suspend=y freezes it until a debugger attaches, which fails readiness probes and restarts the Pod unless you also relax them. The address decides who can connect: * means every client that can reach the Pod, and anyone connected can execute arbitrary code in your JVM. Binding to 127.0.0.1:5005 is safer, and kubectl port-forward still works because it connects from inside the Pod's network namespace. The model below prints the option string with warnings:

Python Building a JDWP option safely

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other languages

Runtime How to enable Forward
Node.js node --inspect=127.0.0.1:9229 server.js (or NODE_OPTIONS=--inspect=127.0.0.1:9229); attach from VS Code or Chrome DevTools kubectl port-forward pod/web 9229:9229
Python python -m debugpy --listen 127.0.0.1:5678 app.py (the debugpy package must be in the image); attach from VS Code port 5678
Go Build with -gcflags="all=-N -l" and run dlv exec ./app --headless --listen=127.0.0.1:2345 --api-version=2 --accept-multiclient port 2345
.NET Use vsdbg from the IDE through kubectl exec as the transport, or Visual Studio's Kubernetes attach exec-based
PHP Xdebug connects from the Pod to your IDE (reverse), so the IDE must be reachable from the cluster; a tunnel tool is easier see below

For compiled languages, debug builds have optimizations turned off and are bigger and slower; use them on a dev or staging cluster, not production.

Safety rules for debug ports

  • Never expose a debug port through a Service, Ingress or LoadBalancer. A debug protocol is remote code execution by design.
  • Prefer localhost binding plus port-forward.
  • Use a separate dev or staging namespace, and restrict who can port-forward and exec with RBAC (pods/portforward and pods/exec are separate permissions, RBAC).
  • Do not leave the debug option in the committed manifest. Add it through a Kustomize overlay or a Helm value used only in dev (Kustomize).
  • Pause with care. A breakpoint in a request thread stalls that request, and a paused JVM may fail its probes. Scale to one replica, or take the Pod out of its Service (remove the label) while you debug it (Services).

Developing against the cluster

Instead of attaching to a Pod, you can run the code on your laptop and let it behave as if it were in the cluster:

  • Port-forward the dependencies your local process needs (kubectl port-forward svc/postgres 5432) and run the app locally against them.
  • Telepresence and mirrord are tools that connect your local process to the cluster network, and can intercept or mirror traffic for a chosen Pod so that requests reach your laptop. They need an agent installed in the cluster, so check what your organization allows.
  • Skaffold and Tilt rebuild and redeploy on every change (skaffold dev, skaffold debug also configures the debug ports), which removes the build-push-apply loop (Local clusters).

Choose by loop speed: a local process with port-forwarded dependencies is the fastest; rebuilding images is the slowest but the most faithful to production.

A debugging checklist

  1. kubectl get pod -o wide, then kubectl describe pod and kubectl logs --previous for the basics (Troubleshooting).
  2. kubectl port-forward to the Pod and call the endpoint yourself, to separate application bugs from Service or Ingress problems.
  3. kubectl exec if a shell exists; otherwise kubectl debug --target.
  4. Attach a debugger through a forwarded, localhost-bound port on a non-production copy.
  5. Remove debug options, ephemeral access and copies when you are done.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You run kubectl port-forward svc/web 8080:80 and then a rollout replaces the Pods. What happens to the tunnel?

    port-forward resolves the Service to one Pod when it starts. It does not balance load or follow rollouts.
  2. 2. Why does kubectl debug --target=web let you see the web container's processes?

    --target shares the process namespace, so ps shows the app and /proc/<pid>/root exposes its filesystem. The Pod's network namespace is already shared.
  3. 3. What is the risk of starting a JVM with a JDWP agent bound to address=*:5005?

    A debug protocol is remote code execution. Bind to localhost and use port-forward, never expose it with a Service.
  4. 4. Which is the right way to reach a debug port in a Pod?

    port-forward keeps the port private and uses your RBAC permissions.
  5. 5. Which RBAC resources control who can use port-forward and exec?

    They are separate subresources, so you can allow reading Pods without allowing shells or tunnels.

Summary

  • kubectl port-forward LOCAL:REMOTE opens a private tunnel to one Pod, for admin pages, databases and debug ports; it is a developer tool, not a way to publish an app.
  • exec, logs --previous and cp cover images with shells; kubectl debug --target adds an ephemeral container with tools to hardened images, and --copy-to debugs a copy.
  • Remote debuggers (JDWP, --inspect, debugpy, Delve) attach through a forwarded, localhost-bound port; never expose a debug port with a Service or Ingress.
  • Restrict pods/portforward and pods/exec with RBAC, and keep debug options out of production manifests.
  • For fast development loops, run the code locally against port-forwarded dependencies, or use Telepresence, mirrord, Skaffold or Tilt.

Next, learn how to ship everything you built with GitOps.