Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 38 ยท Networking and Overlay

Network Troubleshooting and Tuning

A practical guide to debugging Swarm networking: DNS, VIPs, ports, MTU, subnet clashes, ingress, stale entries, plus tuning knobs for the data path.

Networking problems in a cluster feel mysterious because many layers are involved: DNS, VIPs, IPVS, VXLAN, the physical network and firewalls. The good news is that the same short checklist solves the vast majority of cases. This lesson turns the previous three lessons into a diagnostic routine.

Work from the outside in

Ask these questions in order. Stop at the first "no".

  1. Is the service healthy? docker service ls and docker service ps name. If tasks are not running, the network is not the problem yet.
  2. Do the containers share a network? docker service inspect --format '{{json .Spec.TaskTemplate.Networks}}' name. No shared overlay means no DNS and no route.
  3. Does the name resolve? Inside a container: getent hosts api or nslookup api. Failure means a missing network, a typo, or a service that was removed.
  4. Does the VIP answer? Connect to the resolved address and port (nc -zv api 8000 where available).
  5. Does it work between specific tasks? Try tasks.api and talk to one task IP directly. If a direct task IP works but the VIP does not, suspect IPVS state or health checks removing tasks.
  6. Does it work across nodes? If calls succeed only when both containers are on the same node, the VXLAN path is broken: ports, MTU or firewall.

A debugging container

Attach a toolbox container to the same network (the network must be attachable):

CONSOLE
$ docker run --rm -it --network backend nicolaka/netshoot

Any image with curl, dig, nc and tcpdump works. Alternatively use docker exec into a task's container. Remember that the Alpine nslookup is minimal; a toolbox gives better answers.

Common failures and fixes

Symptom Likely cause Fix
Name does not resolve Containers on different networks Attach both to a shared overlay
Works on one node, times out across nodes UDP 4789 or 7946 blocked Open ports in all firewalls and cloud security groups
Small requests work, large ones hang MTU mismatch (VXLAN overhead) Lower the overlay MTU: --opt com.docker.network.driver.mtu=1400
Random timeouts after long idle Idle connection dropped by IPVS or a firewall Enable TCP keep-alive in clients and pools
Published port unreachable on a node Host firewall, or no healthy task Check docker service ps; check the node's firewall for the published port
"address already in use" for a published port Another service or host process owns the port Pick another port or use host mode on specific nodes
New tasks cannot start: "could not find an available IP" Overlay subnet exhausted Create a larger network (default /24 has ~250 addresses) and move services
Client IP is wrong Ingress source NAT Host mode or forwarded headers
Strange behaviour after node IP change Stale gossip entries Restart Docker on the node, or leave and rejoin the swarm

MTU in detail

VXLAN adds about 50 bytes to each packet. If your physical network has MTU 1500 (the usual), overlay containers should use 1450. Docker sets this automatically for the default overlay, but on networks that already encapsulate traffic (a VPN, some cloud overlays with MTU 1400 or 1450) packets get dropped silently when they do not fit. Symptoms: TLS handshakes stall, ping works, big downloads freeze. Test with ping -M do -s 1400 target from the container, then set a lower MTU on the network.

Address space planning

Each task consumes one address on every overlay it joins, the VIP consumes one per service, and the ingress network consumes addresses per published service and per node. A default /24 holds about 250. Plan networks with room to grow:

CONSOLE
$ docker network create -d overlay --subnet 10.30.0.0/22 big-backend

A /22 holds around 1000. You cannot resize a network: you create a new one and move services (docker service update --network-rm old --network-add new name), one at a time. To change the default address pools for new networks, initialize the swarm with docker swarm init --default-addr-pool 10.40.0.0/16 --default-addr-pool-mask-length 24.

Tuning and good practice

  • Pin the ports and firewall rules in your infrastructure code, not in someone's memory.
  • Separate frontend and backend networks to limit blast radius and simplify debugging.
  • Prefer service names over IPs in configuration, always.
  • Use health checks so IPVS drops broken tasks quickly.
  • Keep keep-alives on (TCP keep-alive under 10 minutes) for connection pools.
  • Don't publish databases on the mesh. Keep them on an internal overlay without --publish.
  • Consider --opt encrypted only where needed, and measure the overhead.

Looking at the node

Advanced inspection runs on the node itself, as root. Docker keeps network namespaces in /var/run/docker/netns. Tools like nsenter can enter one to inspect IPVS (ipvsadm -ln) or the VXLAN interfaces. Treat this as a last resort and read only; changing these rules by hand will be overwritten by Docker.

Stand-in: MTU calculator

Quick arithmetic to pick an overlay MTU from the physical MTU, using the roughly 50-byte VXLAN overhead (add more for IPsec encryption, around 50 to 70 bytes depending on the mode; use the number as a rough guide).

Python Stand-in: overlay MTU

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: connection diagnosis walk

This function follows the checklist at the top. Provide observations as booleans and it tells you the first problem to fix, which is the habit to build.

Python Stand-in: the checklist as code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Containers communicate fine on one node but time out across nodes. What do you check first?

    Cross-node traffic uses VXLAN on UDP 4789 and discovery on 7946; a firewall block produces exactly this symptom.
  2. 2. Small requests succeed but large transfers hang. A likely cause is:

    Packets that exceed the effective MTU are dropped, so only large payloads stall.
  3. 3. Tasks cannot start with 'could not find an available IP'. What happened?

    A small subnet runs out of addresses; create a larger network and move the services.

Summary

  • Debug outside in: tasks, shared network, DNS, VIP, task IP, cross-node path.
  • Most cross-node failures are blocked ports or MTU.
  • Plan subnets generously; networks cannot be resized.
  • Keep health checks and TCP keep-alives enabled, and never publish databases.

Next: Docker Swarm Secrets: Create and Use Them