Skip to content
Tutorials Center
My learning

Topics

Lesson 56 of 111 ยท Networking

Troubleshooting Networks

Debug Docker network problems like a pro: inspect networks, test DNS and ports from a helper container, spot loopback binds, port conflicts and wrong-network mistakes.

"It works on my machine" gets a Docker-flavored twist: "the container is running, but the other container cannot reach it." The good news is that container networking failures fall into a short list of causes, and a calm, step-by-step routine finds the culprit in a few minutes. Most of the time the answer is one of five things: wrong network, wrong name, wrong port, wrong bind address, or a port conflict on the host.

This lesson gives you that routine, the tools to run it (even for images that contain no tools at all), and a catalogue of real error messages with their causes. Every output comes from real containers, with IDs and addresses shortened or variable.

The debugging ladder

Work from the bottom up. Each step answers one question, and you stop at the first "no".

TEXT
1. Is the target container running, and healthy?        docker ps, docker logs
2. Are both containers on the SAME network?             docker network inspect
3. Does the NAME resolve?                               nslookup / getent hosts
4. Is something LISTENING on that port, on 0.0.0.0?     netstat -tln (inside the target)
5. Can I CONNECT from the client's side?                curl / wget / nc from the client's namespace
6. Is it the host side (published ports, firewall)?     docker port, curl from the host

Let us set up a small lab with a few deliberate faults and walk through the ladder:

CONSOLE
$ docker network create n1
$ docker network create n2
$ docker run -d --name web --network n1 nginx:alpine
$ docker run -d --name ok --network n1 python:3-alpine python -m http.server 8000
$ docker run -d --name loc --network n1 python:3-alpine python -m http.server 8000 --bind 127.0.0.1

web is a healthy web server. ok serves on port 8000 on all interfaces. loc serves on port 8000 too, but only on its own loopback, which is a very common mistake.

Step 1 and 2: running, and on the same network

First check that the container did not crash. A stopped container has no DNS name at all:

CONSOLE
$ docker stop web
$ docker run --rm --network n1 alpine:3 ping -c1 -W1 web
ping: bad address 'web'
$ docker start web

If a name does not resolve, the cheapest explanation is "the container is not running", so start with docker ps -a and docker logs <name> (see Logs and exec).

Next, list who is attached to a network:

CONSOLE
$ docker network inspect n1 --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
loc 172.21.0.3/16
web 172.21.0.2/16
ok 172.21.0.4/16

And see a single container's networks, addresses and gateway:

CONSOLE
$ docker inspect web --format '{{range $n,$c := .NetworkSettings.Networks}}{{$n}} {{$c.IPAddress}} gw={{$c.Gateway}}{{"\n"}}{{end}}'
n1 172.21.0.2 gw=172.21.0.1

The full JSON also lists the container's DNS names (its name plus its short ID, and any aliases) under DNSNames. If the client and target do not share a network, you found the bug:

CONSOLE
$ docker run --rm --network n2 alpine:3 wget -T2 -qO- http://web
wget: bad address 'web'
$ docker run --rm --network n2 alpine:3 nslookup web
*** Can't find web: No answer

Fix: connect one of them to the other's network with docker network connect, or, better, put both on a shared network from the start (see User-Defined Bridge Networks).

Step 3 to 5: a toolbox in the target's namespace

You will want curl, dig, nc and netstat, but production images are often minimal and have none of them. Do not install tools into the broken container. Instead start a helper container that shares its network namespace:

CONSOLE
$ docker run --rm -it --network container:web alpine:3 sh
/ # apk add --no-cache curl bind-tools
/ # dig +short ok
172.21.0.4
/ # curl -s -o /dev/null -w "%{http_code}\n" http://ok:8000/
200
/ # curl -sS -m3 http://loc:8000/
curl: (7) Failed to connect to loc:8000 after 1 ms: Could not connect to server

Because of --network container:web, the helper has exactly the same interfaces, IP, DNS settings and view of the network as web. Whatever fails here fails for web. When you leave the shell, the helper disappears and the target is untouched. (More on this mode in Host, None, Macvlan and Overlay.)

For ready-made all-in-one toolboxes, many people use a large community image full of network tools. A plain alpine plus apk add is smaller and good enough for nearly everything.

Keep a one-liner in your shell history, for example docker run --rm -it --network container:NAME alpine:3 sh. It is the single most useful networking debug command, because it works for any container no matter how stripped down.

Which of the three failures is it?

The helper above showed two distinct symptoms. Compare what the tools say:

Symptom Meaning Next check
bad address, Can't find, could not translate host name Name resolution failed Is the target running and on this network? Typo in the name?
Connection refused immediately Name resolved; the host answered "nobody listens here" Wrong port, or server bound to loopback, or server still starting
Long wait, then timed out Packets vanish Different network (by IP), firewall rule, or an unresponsive process

Is the server listening where you think?

The most deceptive problem is a service bound to 127.0.0.1. From inside the container itself it works perfectly, so developers conclude the app is fine, yet every neighbour gets "Connection refused". Look at the listening sockets inside the target:

CONSOLE
$ docker exec loc netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.11:33067        0.0.0.0:*               LISTEN      
$ docker exec ok netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.11:33167        0.0.0.0:*               LISTEN      

Read the Local Address column. 127.0.0.1:8000 means loopback only: unreachable from outside. 0.0.0.0:8000 means all interfaces. The 127.0.0.11 line is Docker's own DNS resolver (its port number changes each time).

If the image has no netstat, use the shared-namespace helper and run netstat -tln there, or read /proc/net/tcp directly (0100007F:1F40 is hex for 127.0.0.1:8000).

Fix: configure the application to bind to 0.0.0.0 (for Node, listen(3000, '0.0.0.0'); for Flask, --host 0.0.0.0; for Python's http.server, drop --bind or use --bind 0.0.0.0).

Step 6: the host side

If other containers reach the service but your browser does not, the issue is in the published port.

CONSOLE
$ docker port web

An empty answer means nothing is published. Compare the mapping Docker shows with the port the server really uses:

CONSOLE
$ docker ps --format 'table {{.Names}}\t{{.Ports}}'
NAMES     PORTS
p1        127.0.0.1:18861->80/tcp

Read it as host address:host port -> container port. A few things to verify:

  • The right-hand side must be the port the server listens on inside the container (80 for nginx).
  • A mapping bound to 127.0.0.1 is reachable only from the host itself. To reach it from other machines you need 0.0.0.0 (which is also less safe, see Publishing Ports).
  • A container attached only to an --internal network cannot publish ports.
  • Test from the host with curl -v http://127.0.0.1:<port>/ before blaming the browser.

Port already allocated

The classic start-up error is a port clash on the host:

CONSOLE
$ docker run -d --name p1 -p 127.0.0.1:18861:80 nginx:alpine
$ docker run -d --name p2 -p 127.0.0.1:18861:80 nginx:alpine
docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint p2 (ba6ea7efe88a): Bind for 127.0.0.1:18861 failed: port is already allocated

Only one process can own a host port. Find the owner with docker ps --filter publish=18861, or with netstat -ano | findstr :18861 on Windows and ss -ltnp | grep 18861 on Linux. Then either stop the other process or choose another host port (the container port can stay 80). Note that the failed container p2 still exists in the Created state; remove it with docker rm p2.

Reading the evidence Docker keeps

A few inspection commands settle arguments quickly:

CONSOLE
$ docker exec ok ip route
default via 172.21.0.1 dev eth0 
172.21.0.0/16 dev eth0 scope link  src 172.21.0.4 
$ docker exec ok tail -2 /etc/hosts
ff02::2	ip6-allrouters
172.21.0.4	fa716a304eee
$ docker exec ok cat /etc/resolv.conf | grep nameserver
nameserver 127.0.0.11
  • ip route shows the container's default gateway (the bridge). No default route means no way out.
  • /etc/hosts shows the container's own name and any --add-host entries.
  • /etc/resolv.conf shows which resolver it uses: 127.0.0.11 on user-defined networks, the host's resolver on the default bridge.

You can also watch network events as they happen:

Terminal
docker events --filter type=network

It prints a line each time a container connects to or disconnects from a network, which is useful when an orchestrator rewires things behind your back.

A catalogue of common messages

Message Usual cause What to do
bad address 'x' (busybox), could not translate host name (psql), Temporary failure in name resolution, Try again Target not running, wrong name, different network Check docker ps, spelling, docker network inspect
Connection refused Nothing listening on that port, or bound to loopback netstat -tln in the target; bind to 0.0.0.0; check the port
No route to host Container is on an unrelated network or a firewall rejects Verify subnet and gateway; look at firewall rules
Network unreachable No route at all (--network none, --internal) Pick a network with a route out
Bind for ...: port is already allocated Host port taken Change host port or stop the other owner
network ... has active endpoints Removing a network that still has containers docker network disconnect or remove the containers
invalid pool request: Pool overlaps with other one on this address space A new network's subnet collides with an existing one Choose another --subnet
curl: (52) Empty reply from server / (56) Connected, but the server closed the connection Check the app logs; often HTTP vs HTTPS confusion

When the host is the problem

If everything inside Docker looks right, look outward:

  • Firewalls. On Linux, Docker inserts its own iptables/nftables rules for published ports, and these often bypass friendly front-ends such as ufw (a port you "blocked" in ufw may still be open). Put your restrictions in the DOCKER-USER chain, or bind the published port to 127.0.0.1. See DNS, Proxies and Firewalls.
  • VPNs. Corporate VPNs can capture the private ranges Docker likes (172.16.0.0/12, 192.168.0.0/16). Symptoms are containers that lose connectivity when the VPN connects. Fix by giving Docker networks a different --subnet, or by setting default-address-pools in the daemon configuration.
  • Docker Desktop. Containers run in a VM, so "localhost" and host networking mean the VM; published ports are forwarded to your machine. Restarting Docker Desktop fixes more mysterious network states than it should.
  • IPv6 and DNS. If a lookup returns an IPv6 address the container cannot route, applications may hang before falling back to IPv4. Test with curl -4.

A systematic checklist

When a connection fails, copy this list into your notes:

  1. docker ps -a and docker logs for both sides.
  2. docker network inspect for the shared network.
  3. From a helper in the client's namespace: resolve the name.
  4. From the same helper: connect to the port with curl or nc -zv.
  5. In the target: netstat -tln, look for 0.0.0.0 (or the container IP), not 127.0.0.1.
  6. For host access: docker port, then curl from the host.
  7. Only then suspect firewalls, VPNs and proxies.

Clean up

CONSOLE
$ docker rm -f web ok loc p1 p2
$ docker network rm n1 n2

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A neighbour container gets 'Connection refused' on port 8000 of your app, but the app answers fine when you curl localhost:8000 from inside its own container. What is the most likely cause?

    A server bound to loopback accepts only connections made from inside its own namespace. Different networks would give a name resolution error, not a refusal.
  2. 2. What is the best way to run curl and dig against a container whose image has no tools?

    The helper shares the target's network namespace, so tests run from it behave like tests from the target, and the target stays unmodified.
  3. 3. Which command shows every container attached to a network, with its IP address?

    docker network inspect lists attached containers under Containers. docker network ls shows only the networks themselves, and docker port shows published ports of one container.
  4. 4. Starting a container fails with 'Bind for 127.0.0.1:18861 failed: port is already allocated'. What should you do?

    Only one process can own a host port. The left side of -p is the host port; change it (or free it) while the container port can stay the same.
  5. 5. A lookup of a container name gives 'bad address', and docker ps shows the target Exited. Why?

    A container's DNS entry exists only while it is attached to the network, which requires it to be running. Check docker logs for why it stopped.

Summary

  • Troubleshoot in order: running, same network, name resolves, port listening on 0.0.0.0, connection works, then host-side publishing and firewalls.
  • Use docker network inspect and docker inspect to see networks, IPs and DNS names.
  • Debug with a helper started via --network container:NAME; do not modify the broken container.
  • bad address is a name problem, Connection refused means no listener, a timeout means dropped packets.
  • Check netstat -tln inside the target: 127.0.0.1 is the classic trap.
  • "Port is already allocated" means a host port clash; change the host side of -p.
  • Firewalls, VPN subnets and Docker Desktop's VM are the usual outside suspects.

Next: DNS, Proxies and Firewalls