Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 111 ยท Containers in Depth

Debugging Containers

A practical workflow for broken containers: read exit codes, logs and inspect output, override the entrypoint, attach a debug sidecar to shell-less images and watch events.

Sooner or later a container will not start, will start and immediately stop, will run but not answer, or will be killed for no obvious reason. The good news is that Docker gives you a small set of tools that, used in the right order, find the cause of nearly every problem within minutes. The bad news is that many people reach for the wrong one first, typically docker exec, which only works on a container that is still running.

This lesson gives you a workflow rather than a list of flags: first read what Docker already knows (status, exit code, logs), then look at the configuration the container actually received, then get inside (by override, by exec or by a sidecar), and finally watch the system's events. Every command and message below was produced by a real daemon with small images. The tools from Logs and exec and Listing and Inspecting are assumed.

The debugging ladder

TEXT
Does the container exist?            docker ps -a
  |
  v
What state is it in, and how did it end?   docker inspect  (State: exit code, OOMKilled, Error)
  |
  v
What did it say?                     docker logs
  |
  v
What did it get as configuration?    docker inspect (Entrypoint, Cmd, Env, Mounts, Ports)
  |
  v
Can I reproduce it interactively?    docker run --entrypoint sh   /   docker exec
  |
  v
No shell inside?                     sidecar with --pid / --network container:NAME
  |
  v
Something outside the container?     docker events, docker stats, daemon logs

Climb it one rung at a time. Most problems are solved in the first three steps.

Step 1: status and exit code

Start with docker ps -a, because docker ps shows only running containers and a crashed one will simply be missing.

CONSOLE
$ docker run -d --name api myapp
$ docker ps -a --filter name=api --format 'table {{.Names}}\t{{.Status}}\t{{.Command}}'
NAMES     STATUS                              COMMAND
api       Exited (1) Less than a second ago   "python app.py"

The number in Exited (1) is the exit code of the main process. It is the single most useful clue, so memorize this table:

Code Usual meaning
0 The program finished normally. Not an error, but if it is a server it should not have exited.
1 The program failed (generic error, uncaught exception). Read the logs.
2 Often a command-line usage error or a shell builtin misuse.
125 Docker itself failed: bad option, unknown image, port conflict. The container never ran.
126 The command was found but cannot be executed (permission denied, wrong file type).
127 The command was not found (typo, binary missing from the image, wrong PATH).
137 Killed by SIGKILL (128 + 9): docker kill, a stop that timed out, or the out-of-memory killer.
139 Killed by SIGSEGV (128 + 11): the program crashed with a memory fault.
143 Ended by SIGTERM (128 + 15): a stop where the program did not install a handler and was not PID 1.

Codes above 128 mean "died from signal (code - 128)". Codes 125 to 127 come from Docker or the shell before or while starting your program. Let us see the three start-up failures with real messages, because their wording is worth recognizing.

CONSOLE
$ docker run --rm alpine:3.22 nosuchcmd
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "nosuchcmd": executable file not found in $PATH

Run 'docker run --help' for more information
$ echo $?
127

That is exit code 127: the command does not exist in the image. Typical causes are a typo, a binary that is in the build stage but not in the final stage of a multi-stage build, or a shell script with Windows line endings (the interpreter line then reads #!/bin/sh\r, and the "missing file" is the interpreter).

CONSOLE
$ docker run --rm alpine:3.22 /etc/passwd
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/etc/passwd": permission denied

Run 'docker run --help' for more information
$ echo $?
126

Code 126: the file exists but is not an executable (a script missing chmod +x, or a directory). And code 125 means Docker never got as far as starting anything:

CONSOLE
$ docker run --rm nosuchimage:1
Unable to find image 'nosuchimage:1' locally
docker: Error response from daemon: pull access denied for nosuchimage, repository does not exist or may require 'docker login'

Run 'docker run --help' for more information
$ echo $?
125
$ docker run --rm -p 99999:80 alpine:3.22
docker: invalid hostPort: 99999

Run 'docker run --help' for more information
$ echo $?
125

Notice what these have in common: the error text is printed by the docker client, not by your program, and the exit status is 125 to 127. Anything else is your application's own exit code.

docker inspect knows how it ended

docker inspect stores more than the exit code. The State block records whether the out-of-memory killer was involved and whether Docker itself reported an error:

CONSOLE
$ docker run --name c1 alpine:3.22 sh -c 'echo working; exit 3'
working
$ docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err=[{{.State.Error}}]' c1
exited exit=3 oom=false err=[]

Compare with a container that exceeded its memory limit:

CONSOLE
$ docker run --name oom --memory 32m --memory-swap 32m python:3-alpine python -c "x = bytearray(200*1024*1024); print('allocated')"
$ echo $?
137
$ docker inspect -f 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' oom
exit=137 oom=true

Both are 137, but only OOMKilled=true proves the kernel killed the process for using too much memory (the cure is in Resource Limits). An exit code of 137 with OOMKilled=false points to a docker kill, a stop that timed out (see Signals, PID 1 and --init), or another process killing it.

Step 2: read the logs

docker logs shows what the main process wrote to standard output and standard error, and it works on stopped containers too. That is why you should not use --rm while debugging: it deletes the logs with the container.

CONSOLE
$ docker logs api
Traceback (most recent call last):
  File "/app/app.py", line 4, in <module>
    url = os.environ["DATABASE_URL"]
          ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "<frozen os>", line 729, in __getitem__
KeyError: 'DATABASE_URL'

Here is the cause: the app requires an environment variable that was never passed. Supplying it is the fix:

CONSOLE
$ docker run --rm -e DATABASE_URL=postgres://db/app myapp
connecting to postgres://db/app

If the logs are empty, the program probably writes to a file instead of stdout, or buffers its output. Python, for example, buffers stdout when it is not a terminal, so a crash can lose the last lines; set PYTHONUNBUFFERED=1 or use print(..., flush=True). The convention in containers is to log to stdout and stderr so that docker logs and log drivers can collect it (see Logging and Monitoring).

Step 3: check the configuration the container really got

Dockerfile, Compose file and docker run flags are your intentions. docker inspect shows what the daemon actually applied. For a wrong-command bug, compare the entrypoint and command:

CONSOLE
$ docker inspect -f 'entrypoint={{json .Config.Entrypoint}} cmd={{json .Config.Cmd}} workdir={{.Config.WorkingDir}}' myapp
entrypoint=null cmd=["python","app.py"] workdir=/app

null means no entrypoint is set. Other fields worth knowing: .Config.Env (all variables), .Config.User, .HostConfig.Binds and .Mounts (what is mounted where), .NetworkSettings.Ports (published ports), .HostConfig.RestartPolicy and .State.Health (healthcheck results). Run docker inspect api without -f once and scroll through it; you will learn where things live.

The restart loop

A container with a restart policy hides crashes because it keeps coming back. These two signs give it away:

CONSOLE
$ docker run -d --name loop --restart always myapp
$ docker ps -a --filter name=loop --format '{{.Names}} | {{.Status}}'
loop | Restarting (1) Less than a second ago

Restarting (1) means "crashed with exit code 1 and will be started again". With a bounded policy you can see how many attempts were made, and the container finally stops:

CONSOLE
$ docker run -d --name bounded --restart on-failure:3 myapp
$ docker inspect -f 'restarts={{.RestartCount}} status={{.State.Status}} exit={{.State.ExitCode}}' bounded
restarts=3 status=exited exit=1

Debug a looping container the same way as any other: docker logs (it keeps the output of earlier attempts), and if needed, docker update --restart=no loop to stop the cycle while you investigate. See The Container Lifecycle for how the policies behave.

Step 4: reproduce it interactively

Override the entrypoint

When a container dies at once, there is nothing to exec into. Start a new container from the same image but replace the entrypoint with a shell, so nothing crashes:

CONSOLE
$ docker run --rm --entrypoint sh myapp -c 'ls -l /app; python --version'
total 4
-rwxr-xr-x    1 root     root            95 ... app.py
Python 3.15.1

In a real session you would add -it and omit -c ... to get a prompt: docker run --rm -it --entrypoint sh myapp. From the prompt you can run the application's command by hand (python app.py), look at the files that are actually in the image, check env, test network access to other services, and fix the issue interactively before encoding the fix in the Dockerfile.

--entrypoint replaces the image's ENTRYPOINT (the arguments after the image name become its arguments). If the image defines only a CMD, simply putting a command after the image name replaces CMD: docker run --rm -it myapp sh. See CMD and ENTRYPOINT. Add the same -e, -v and --network options you normally use, otherwise you debug a different situation than the failing one.

Exec into a running container

When the container is running but misbehaving, docker exec starts an extra process inside it, sharing its filesystem, network and environment:

CONSOLE
$ docker exec -it web sh
$ docker exec web ps -o pid,ppid,args
$ docker exec --user root web id -u
0

The last form opens a root shell even if the container normally runs as a non-root USER, which is handy for installing a tool temporarily (apk add curl). Remember that anything you install this way disappears when the container is removed (see Container Storage) and is not part of your image.

Look at a crashed container's files

A stopped container keeps its writable layer until you remove it. docker diff lists what changed compared with the image (A added, C changed, D deleted), and docker cp copies files out:

CONSOLE
$ docker run --name crashed alpine:3.22 sh -c 'echo "step 1 ok" > /tmp/progress.txt; echo "step 2 failed" >&2; exit 2'
step 2 failed
$ echo $?
2
$ docker diff crashed
C /tmp
A /tmp/progress.txt
$ docker cp crashed:/tmp/progress.txt - | tar -xO
step 1 ok

(docker cp ... - streams a tar archive to standard output; tar -xO prints the file.) If you need a shell inside the crashed state itself, docker commit crashed snap turns the container into a throwaway image you can start with docker run --rm -it snap sh. This is good for debugging and bad for anything else, as explained in Save, Load, Export, Import. Delete the snapshot image afterwards.

Step 5: containers with no shell

Minimal images (scratch, distroless) contain only your program. That is great for security and size, and awkward for debugging:

CONSOLE
$ docker exec bare sh
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

You cannot install a shell into it, but you can start a second container that joins the first one's namespaces and brings its own tools. --pid container:NAME shares the process view, and --network container:NAME shares the network stack (the same IP, interfaces and localhost). The target here is a tiny Go web server built FROM scratch:

CONSOLE
$ docker run --rm --pid container:bare busybox:1.37 ps
PID   USER     TIME  COMMAND
    1 root      0:00 /server
   45 root      0:00 ps
$ docker run --rm --network container:bare busybox:1.37 wget -qO- http://localhost:8080/
hello from bare
$ docker run --rm --network container:bare busybox:1.37 netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 :::8080                 :::*                    LISTEN      

The helper sees the app as PID 1, reaches the server over localhost and sees the listening port. Because the process namespace is shared, the helper can even read the target's environment and its filesystem through /proc:

CONSOLE
$ docker run --rm --pid container:bare busybox:1.37 sh -c 'cat /proc/1/environ | tr "\0" "\n"'
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=71b3d23974a3
APP_NAME=bare
HOME=/
$ docker run --rm --pid container:bare busybox:1.37 ls /proc/1/root/
dev
etc
proc
server
sys

/proc/1/root/ is the target's root filesystem, so you can read config files that exist inside it. This works only for processes you are allowed to ptrace (as root with the default capabilities it does). Docker Desktop also offers docker debug <container>, which attaches a toolbox shell to a container that has none. It is a Docker Desktop feature (not part of plain Docker Engine), so treat it as a read-only example here:

CONSOLE
$ docker debug bare          # read-only example, not run here

Which is better, an --entrypoint override or a sidecar? Use the override when the container will not start (the sidecar needs a running target) and the sidecar when the problem only shows up in the live process and you must not restart it.

Keep one debug image around with your favorite tools (curl, dig, ping, tcpdump, strace, jq) and use it as the sidecar. Pulling a general-purpose toolbox is faster than rebuilding your production image.

Step 6: events, stats and the daemon

Sometimes the container's own view is not enough: it was killed, and nobody knows by whom or when. docker events is the daemon's audit log. It streams container, image, network and volume events, and with --since and --until it replays history and stops. Here we ran one container that crashed and one that was out of memory:

CONSOLE
$ docker events --since 5m --until 0s --filter container=app --filter container=hog --format '{{.Type}} {{.Action}} {{.Actor.Attributes.name}}'
container create app
container start app
container die app
container create hog
container start hog
container oom hog
container die hog
$ docker events --since 5m --until 0s --filter container=hog --filter event=die --format '{{.Action}} exitCode={{.Actor.Attributes.exitCode}}'
die exitCode=137

The oom event just before die confirms the memory kill. Without --until, docker events keeps running and prints events live, which is useful in a second terminal while you reproduce a problem. Filter by --filter event=health_status to watch healthchecks, or --filter type=network for connect and disconnect events. Event history is limited, so --since can only reach back as far as the daemon remembers.

Two more quick checks for a container that is running but slow or stuck:

CONSOLE
$ docker stats --no-stream --format '{{.Name}} {{.MemUsage}}'
bare 1.617MiB / 11.63GiB
$ docker top web

docker stats shows live CPU, memory, network and process counts (the last memory number is the limit; with no --memory it is the host's total). docker top lists the processes from the host's side. A container pegged at 100% CPU or growing in memory points to the application, not Docker. If Docker itself misbehaves (it cannot pull, build or start anything), look at the daemon's logs: journalctl -u docker on a systemd Linux host, or Troubleshoot in Docker Desktop.

Common causes, by symptom

Symptom First thing to try
Exits immediately with 0 The command is not a long-running process (for example bash without -it, or a server started in the background with &). Run it in the foreground.
Exits with 1 and a stack trace docker logs; usually missing configuration, wrong credentials or an unreachable dependency.
127 or 126 Wrong command, missing binary in the final stage, Windows line endings in a script, script not executable.
137 right after start OOMKilled? Raise --memory or fix the leak.
Starts, but port does not answer Is the app listening on 0.0.0.0 (not 127.0.0.1) inside the container, and is the port published with -p?
Works with docker run, fails in Compose docker compose config shows the merged file; compare env, command and working directory.
Healthy on laptop, fails in CI Different architecture or file permissions; compare docker inspect output.
"No such file or directory" for a file you copied Check WORKDIR and .dockerignore (see Build Context and .dockerignore).

A frequent trap is an application bound to 127.0.0.1 inside the container. Published ports reach the container's external interface, so a server that listens only on its own loopback is unreachable from outside, even though docker exec and a localhost check inside the container work. Bind to 0.0.0.0.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. docker ps shows nothing, but you just started a container that was supposed to run a web server. What should you run first?

    docker ps lists only running containers. docker ps -a shows the exited one with its exit code, and docker logs still works on it. docker exec needs a running container.
  2. 2. A container ends with exit code 127 immediately after docker run. What does that indicate?

    127 means the executable could not be found (executable file not found in $PATH). 126 means found but not executable, 137 would be SIGKILL, and a failed pull is Docker's own error 125.
  3. 3. How can you tell that exit code 137 was caused by the kernel's out-of-memory killer, not by docker kill?

    Both end with SIGKILL (137), but Docker records OOMKilled=true in the container state and emits an oom event when the memory limit was the cause.
  4. 4. A distroless container has no shell, so docker exec NAME sh fails. Which approach lets you inspect its live network and processes?

    A helper container can join the target's PID and network namespaces and bring its own tools (ps, wget, netstat), without touching the target's image.
  5. 5. Why is it a bad idea to debug with docker run --rm?

    A stopped container keeps its logs, state and writable layer until removed. With --rm they vanish the moment the process exits, leaving nothing to read.

Summary

  • Follow the ladder: docker ps -a, then docker inspect (exit code, OOMKilled, Error), then docker logs, then the real configuration, then an interactive reproduction.
  • Exit codes: 125 is Docker itself, 126 not executable, 127 command not found, 137 SIGKILL (stop timeout or OOM), 143 SIGTERM; above 128 means death by signal (code - 128).
  • Do not use --rm while debugging; stopped containers keep logs and files, and docker diff, docker cp and docker commit can recover evidence.
  • When a container will not start, replace its entrypoint: docker run --rm -it --entrypoint sh IMAGE. When it runs, use docker exec.
  • For shell-less images, run a helper with --pid container:NAME and --network container:NAME; docker debug is the built-in alternative (read-only example).
  • docker events --since ... --until ... tells you what the daemon did and when: create, start, oom, die and healthcheck events.
  • Apps must bind to 0.0.0.0 inside the container to be reachable through published ports.

Next: Cleaning Up