Debugging Docker Containers

Most Docker debugging goes wrong the same way: someone stares at a broken container, guesses at a fix, restarts it, and repeats until something changes or they give up. The container is not a black box, though. Docker exposes almost everything you need through a small set of commands, and if you run them in the right order you turn "it does not work" into a specific answer within a minute or two. This guide gives you that order - the systematic read - and then walks the classic failures (immediate exits, OOM kills, unreachable published ports, restart loops) with the actual fixes.

The method: STATUS, then logs, then inside

When a container misbehaves, resist the urge to restart it. Restarting destroys the evidence and, if the cause is a bug in your image or config, just reproduces the same broken state. Work the problem in a fixed order instead:

1. Look at STATUS. docker ps -a tells you whether the container is running, restarting, or exited - and with what exit code. That single line usually narrows the cause to a category.
2. Read the logs. docker logs shows what the process printed on stdout/stderr. Most crashes announce themselves here in plain text.
3. Get inside (or inspect from outside). If it is running, docker exec gets you a shell to poke around. If it exited or has no shell, docker inspect and docker events tell you how it was configured and what happened to it.

That is the whole loop: STATUS narrows the category, logs give the application's own account, and inside/inspect confirms the specifics. Almost every section below is a special case of this sequence.

Reading STATUS and exit codes

docker ps shows only running containers. When something is broken, you almost always want docker ps -a, which includes stopped and exited ones - the container that died two seconds after start will not appear otherwise.

docker ps -a

CONTAINER ID   IMAGE        STATUS                        NAMES
a1b2c3d4e5f6   myapp:1.4    Exited (1) 12 seconds ago     web
9f8e7d6c5b4a   redis:7      Up 3 minutes (healthy)        cache
3c2b1a0f9e8d   worker:2.0   Restarting (137) 4 sec ago    worker

The STATUS column is the first real clue, and the exit code in parentheses is the second. The number is the container's exit code, and a handful of values come up constantly:

• Exit 0 - the process ended cleanly, on purpose. If a container you expected to keep running shows Exited (0), the problem is usually that its main process was a one-shot command that finished, or a server that had nothing to do and returned. A container lives only as long as its PID 1; when that process returns, the container stops, success or not.
• Exit 1 (or other small non-zero) - the process crashed or errored out. A generic application failure: an unhandled exception, a missing file, a bad config value. The logs will almost always say what.
• Exit 125 - Docker itself failed to run the container (bad flag, invalid option). The error is from the daemon, not your app.
• Exit 126 - the command was found but is not executable (permission bits, or it is not actually a binary).
• Exit 127 - command not found. The entrypoint or command path does not exist in the image. Very common after a typo in CMD, or when you expected a shell or tool that the base image does not include.
• Exit 137 - the container was killed with SIGKILL (128 + 9). Most often this is the OOM killer: the container hit its memory limit and the kernel killed it. It can also be a docker kill or a stop that exceeded its grace period. Check docker inspect for "OOMKilled": true to confirm.
• Exit 143 - terminated by SIGTERM (128 + 15), the normal graceful stop signal. Usually expected (someone ran docker stop).

So the read on that docker ps -a output above is immediate: web errored (exit 1, go read its logs), worker is stuck in a restart loop after being OOM-killed (137, raise its memory limit or fix the leak), and cache is fine.

docker logs: the application's own story

Once STATUS points you at a container, docker logs is where the application tells you what went wrong in its own words. It captures whatever the container's main process wrote to stdout and stderr.

docker logs web                 # everything the container has printed
docker logs -f web              # follow live, like tail -f
docker logs --tail 50 web       # just the last 50 lines
docker logs -f --tail 100 web   # follow, starting from the last 100 lines
docker logs -t web              # prefix each line with a timestamp
docker logs --since 10m web     # only the last 10 minutes

For a crashed container, the last few lines are usually the payoff - the stack trace or error message printed on the way out. --tail keeps you from scrolling through hours of startup noise, and -f is what you leave running in one terminal while you reproduce a problem in another.

One caveat that trips people up: docker logs only shows what the process writes to stdout/stderr. If your app logs to a file inside the container instead, docker logs will be empty and you will wrongly conclude the container is silent. The containerized convention is to log to stdout/stderr; if an app insists on a file, you will need to exec in and read that file directly (below). This is also why a container can look "broken and silent" when it is actually logging fine to the wrong place.

docker exec: getting a shell inside

If a container is running but behaving oddly, the fastest way to understand it is to step inside and look around - check files, environment variables, network reachability, running processes.

docker exec -it web /bin/bash    # interactive shell (if bash exists)
docker exec -it web /bin/sh      # fall back to sh - almost every image has it
docker exec web env              # dump environment variables without a shell
docker exec web ls -la /app      # run a single command and exit
docker exec web cat /app/config.yaml

The -it flags give you an interactive terminal (-i keeps stdin open, -t allocates a TTY). Once inside, the usual questions are: are the environment variables what I set (env), is the config file actually there and correct (cat), can this container reach its dependencies (wget, curl, nc, or getent hosts db), and is the process actually running (ps aux, if ps exists).

When there is no shell

Minimal and distroless images are increasingly common - they ship your binary and nothing else, on purpose, for a smaller attack surface. There is no bash, no sh, often no ls. docker exec -it web /bin/sh just returns exec: "/bin/sh": stat /bin/sh: no such file or directory. Do not conclude the container is broken; it simply has no shell to give you. You have a few options:

• Inspect from outside instead of getting in. docker inspect, docker logs, and docker top web (which lists the container's processes from the host, no in-container ps needed) get you most of what you wanted without a shell.
• Copy files out with docker cp and read them on the host: docker cp web:/app/config.yaml ./config.yaml.
• Attach a debug container that shares the target's namespaces. docker debug (Docker Desktop) or, more portably, run a throwaway container in the same PID and network namespace:

docker run -it --rm --pids-limit 100 \
  --network container:web \
  --pid container:web \
  nicolaka/netshoot sh

That gives you a full toolbox (curl, dig, tcpdump, ss) seeing the same network and process view as the target container, without adding anything to the production image. nicolaka/netshoot is the standard image for this.

docker inspect: the full configuration

docker inspect dumps the complete configuration and state of a container (or image, volume, or network) as JSON. This is the ground truth for "how was this container actually set up" - not how you think you ran it, but what Docker recorded. It is invaluable when a container exited before it could log anything, or when the bug is a mismatch between intended and actual config.

docker inspect web                          # the whole thing (long)
docker inspect --format '{{.State.Status}}' web
docker inspect --format '{{.State.ExitCode}}' web
docker inspect --format '{{.State.OOMKilled}}' web    # true if killed for memory
docker inspect --format '{{.State.Error}}' web
docker inspect --format '{{json .Mounts}}' web        # volumes and bind mounts
docker inspect --format '{{json .NetworkSettings.Ports}}' web
docker inspect --format '{{json .Config.Env}}' web    # environment variables

The --format flag with a Go template pulls out just the field you want instead of scrolling pages of JSON. The high-value fields for debugging:

• .State - Status, ExitCode, OOMKilled, Error, and start/finish timestamps. This is how you confirm an OOM kill, or read the exit code without staring at docker ps -a.
• .Mounts - every volume and bind mount, with source and destination. The place to check when a container "cannot find" a file that you are sure you mounted - often the host path is wrong or the mount landed at a different destination than the app expects.
• .NetworkSettings - the container's IP, the network it is on, and the port mappings. Central to the "published port but unreachable" case below.
• .Config.Env and .Config.Cmd - the environment and the command it was actually started with.

docker stats: resource pressure

When a container is slow, getting killed, or thrashing, docker stats gives you a live view of resource use - CPU, memory (against its limit), network, and disk I/O.

docker stats            # live, all running containers
docker stats web        # just one
docker stats --no-stream    # a single snapshot, then exit (good for scripts)

CONTAINER   CPU %    MEM USAGE / LIMIT     MEM %   NET I/O        BLOCK I/O
web         3.2%     412MiB / 512MiB       80.5%   1.2MB / 800kB  0B / 0B
worker      99.4%    120MiB / 256MiB       46.9%   0B / 0B        4MB / 0B

The MEM USAGE / LIMIT column is the one to watch for exit-137 mysteries. If a container repeatedly climbs toward its limit and then dies with 137, you have your answer: it is hitting the memory ceiling and being OOM-killed. Either the limit is too low for the real workload, or the app is leaking - docker stats over a few minutes tells you which (a steady climb that never plateaus is a leak). A container pinned at high CPU that is not doing useful work is the other classic: a hot loop, or a process starved and throttled because its limit is too tight.

docker events: what the daemon saw

docker logs is the container's story; docker events is Docker's story. It streams lifecycle events from the daemon in real time - creates, starts, dies, kills, OOMs, health-status changes, network connects. It is the tool for catching when and why something happened, especially for a container that restarts too fast to inspect.

docker events                                   # live stream of everything
docker events --filter container=web            # just one container
docker events --filter event=die                # only death events
docker events --since 1h                         # replay the last hour
docker events --filter event=oom                 # OOM kills specifically

Leave docker events running in a terminal while you reproduce a restart loop, and you will see the exact sequence - die, then start, then die again - with timestamps and the exit code on each die. When a container churns too fast for docker ps -a to catch it mid-cycle, events is how you observe the loop.

Healthchecks

A healthcheck is a command Docker runs periodically inside the container to decide whether it is actually working, not just running. Without one, "the process is up" is all Docker knows; with one, you get a real healthy / unhealthy signal in docker ps and can make dependent services wait for readiness.

You define it in the Dockerfile (or compose file):

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

The command's exit code is the verdict: 0 means healthy, 1 means unhealthy. --start-period gives a slow-booting app grace time before failures count, and --retries sets how many consecutive failures flip it to unhealthy.

When a container shows (unhealthy) in docker ps, read the recorded probe output - Docker keeps the last few results:

docker inspect --format '{{json .State.Health}}' web

That shows the current health status and the last handful of check results with their output and exit codes, which usually explains the failure directly (connection refused, wrong path, timeout). Two common mistakes to check for: a healthcheck that curls a port the app is not actually listening on, and one whose curl/wget binary does not exist in a minimal image (so the check "fails" only because the probe command is missing).

The classic case: published port but I cannot reach it

You ran the container with -p 8080:80, docker ps shows the mapping, but curl localhost:8080 refuses the connection. This is the single most common Docker networking confusion, and it almost always comes down to one of a few things.

First, confirm the mapping is what you think:

docker ps                    # PORTS column shows e.g. 0.0.0.0:8080->80/tcp
docker port web              # explicit mapping for this container

-p 8080:80 means host port 8080 forwards to container port 80. Now the checklist, in order of likelihood:

• The app inside is listening on the wrong interface. This is the big one. Many apps default to binding 127.0.0.1 (localhost) inside the container. From the container's own point of view that is fine, but Docker's port forwarding arrives from outside that loopback, so nothing answers. The app must listen on 0.0.0.0 (all interfaces) inside the container to be reachable through a published port. Fix it in the app config, not in Docker. Verify from inside: docker exec web sh -c 'netstat -tln' (or ss -tln) - you want to see 0.0.0.0:80, not 127.0.0.1:80.
• The container port is wrong. If the app actually listens on 3000 but you published -p 8080:80, you forwarded to a dead port. The container-side number in -p HOST:CONTAINER must match what the app binds. docker exec web ss -tln shows the truth.
• The app is not up yet, or crashed. docker ps -a and docker logs web - a refused connection often just means the process behind the port never started.
• You are testing from the wrong place. Between two containers on a user-defined bridge network, you do not use published host ports at all - you reach the other container by its name on the container port (http://web:80), and -p is irrelevant. Published ports are for host-to-container, not container-to-container. (On the default bridge, name resolution does not work; use a named network.)

Ninety percent of the time it is the first item: the app is bound to 127.0.0.1 inside the container. Make it bind 0.0.0.0 and the published port starts working.

The classic case: container keeps restarting

A container stuck in Restarting (...) in docker ps -a, or flickering between up and down, is a crash loop: it starts, its main process dies, and a restart policy (--restart unless-stopped or always, or restart: in compose) brings it back, over and over. The restart policy is not the bug - it is faithfully doing its job. Something is killing the process on each start.

Work it like this:

docker ps -a                  # confirm Restarting, note the exit code
docker logs --tail 50 web     # what did it print before dying?
docker inspect --format '{{.State.ExitCode}}' web
docker inspect --format '{{.State.OOMKilled}}' web
docker events --filter container=web    # watch the die/start cycle live

The exit code from the loop points straight at the category:

• Exit 1 and an error in the logs - an application-level failure on startup: a missing or malformed config, an environment variable that is not set, a required file not mounted, or a dependency (database, cache) that is not reachable yet. Read the logs; they almost always name it. If a dependency is simply slow to come up, the fix is often a healthcheck plus depends_on: condition: service_healthy in compose so the app does not start before what it needs.
• Exit 137 with OOMKilled: true - the container exceeds its memory limit on every run and gets killed. Raise --memory / mem_limit to fit the real workload, or fix the leak. Confirm with docker stats (memory climbing to the limit) and the inspect flag above.
• Exit 127 - command not found: the entrypoint or CMD path does not exist in the image. Check for a typo, or a binary you expected the base image to have but it does not.
• Exit 0 but still restarting - the main process finishes cleanly and the restart policy dutifully starts it again. This means your container's PID 1 is a short-lived command, not a long-running server. The fix is to make the container's main process the thing that stays running (the actual server), not a script that sets something up and exits.

The trap to avoid: repeatedly running docker restart web and hoping. The container is already restarting itself; another manual restart just runs the same broken start again. Read the exit code and the logs, fix the underlying cause (config, memory, command, or a dependency ordering), then let it come back up healthy.

The shape of it

A misbehaving container is not opaque - it just needs to be read in order. Start with docker ps -a for STATUS and the exit code, which sorts the problem into a category (clean exit, app crash, OOM kill, command-not-found). Go to docker logs for the application's own account of what went wrong. Then get closer: docker exec for a shell when the container is running, docker inspect for the recorded configuration when it is not, docker stats when resources are the suspect, and docker events to watch a fast loop cycle. The two classic incidents both yield to this: the unreachable published port is almost always an app bound to 127.0.0.1 instead of 0.0.0.0 inside the container, and the endless restart is a crash loop whose exit code names the cause. Read STATUS, read logs, look inside, fix the root - not the symptom - and the container stops being a mystery.