Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 38 ยท Networking and Overlay

Overlay Networks

Learn how overlay networks connect containers across Swarm nodes with VXLAN, how to create, attach and encrypt them, and how ingress and docker_gwbridge fit in.

On one host, containers talk over a bridge network. In a swarm, the two containers that need to talk may sit on different machines. An overlay network solves this: it is a virtual network that spans all nodes of the swarm, so containers attached to it can reach each other by name as if they were on the same switch, no matter where the scheduler placed them.

How it works

Overlay networks are built on VXLAN: each container's Ethernet frame is wrapped in a UDP packet (port 4789) and sent to the node that hosts the destination container. The sender and receiver nodes unwrap and wrap transparently. Swarm's gossip protocol (port 7946) spreads the table that says "container IP X lives on node Y", so no central router is needed.

flowchart LR
  subgraph N1["Node 1"]
    A["web task 10.0.1.3"]
  end
  subgraph N2["Node 2"]
    B["api task 10.0.1.4"]
  end
  A -->|"VXLAN over UDP 4789"| B

Containers use overlay addresses; between nodes the packets travel inside UDP on the physical network.

Creating an overlay network

Run on a manager:

CONSOLE
$ docker network create --driver overlay --subnet 10.20.0.0/24 backend
$ docker network ls --filter driver=overlay

Then attach services:

CONSOLE
$ docker service create --name api --network backend myorg/api:1
$ docker service create --name db  --network backend postgres:17

Details worth knowing:

  • The network is created on the manager first. It appears on a worker only when a task on that worker needs it, which keeps the state small. So docker network ls on a worker may not show it until a task starts there.
  • Containers on the same overlay reach each other by service name (and task name) through the built-in DNS (next lesson).
  • Containers on different overlays cannot reach each other. Use networks as segmentation: a frontend network for web and API, a backend network for API and database, so the web tier never touches the database.
  • Choose --subnet yourself if the default range (often 10.0.0.0/24 and following) clashes with your corporate network or VPN.

Using networks as segments

flowchart LR
  P["proxy"] --- F["frontend network"]
  F --- A["api"]
  A --- B["backend network"]
  B --- D["db"]

The proxy never shares a network with the database.

A service can join several networks: --network frontend --network backend.

Attachable overlays

By default only services can join overlay networks. To let a standalone container (started with docker run) join, create the network as attachable:

CONSOLE
$ docker network create -d overlay --attachable debug-net
$ docker run --rm -it --network debug-net alpine sh

This is useful for one-off debugging containers that need to reach services by name. Remove such networks when done, because any user with Docker access on a node can attach to them.

Encryption

CONSOLE
$ docker network create -d overlay --opt encrypted secure-net

Control-plane traffic is always encrypted, but application traffic on an overlay is not unless you ask. With --opt encrypted, Swarm creates IPsec (ESP) tunnels between the nodes, rotating keys periodically. Requirements and costs:

  • IP protocol 50 (ESP) must be allowed between nodes, plus UDP 4789.
  • Some CPU overhead, sometimes significant on busy links.
  • Not supported on Windows nodes for overlay encryption in older releases; check your Docker version's documentation.
  • Encryption is per network, decided at creation.

If your nodes sit on an untrusted network (the public internet, shared cloud networks), encrypt. Inside a private VLAN many teams accept the overhead trade-off differently, or encrypt at the application level with TLS.

Special networks you will see

  • ingress: a built-in overlay used by the routing mesh for published ports (lesson).
  • docker_gwbridge: a bridge on each node connecting overlay containers to the outside world, giving them outbound internet access via NAT.
  • bridge and host: per-node networks, not part of the swarm overlay.

Inspecting

CONSOLE
$ docker network inspect backend
$ docker network inspect --format '{{json .Containers}}' backend

On a manager, inspect shows the subnet, the gateway, the attached services' VIPs (under Services with --verbose) and peers. Useful when an address conflict is suspected.

Troubleshooting checklist

  1. Are ports 7946 (TCP and UDP) and 4789 (UDP) open between all nodes?
  2. Is MTU consistent? VXLAN adds about 50 bytes; if the physical network MTU is small (VPN, some clouds), large packets stall. Set --opt com.docker.network.driver.mtu=1400 when creating the network if needed.
  3. Is the service actually attached to the network (docker service inspect)?
  4. Does the subnet overlap with a host network?
  5. Does the task's own node have the network (create a task there)?

Stand-in: subnet planner

Overlapping subnets cause baffling problems. This stand-in uses Python's ipaddress module to check a plan for overlaps and compute capacity (usable addresses are fewer than the subnet size, because each task and the VIP consume one address).

Python Stand-in: subnet overlap check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: which containers can talk?

A container can reach another only if they share a network. This graph check models the segmentation idea.

Python Stand-in: network reachability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Which technology carries overlay traffic between nodes?

    Overlay packets are encapsulated in VXLAN and sent over UDP port 4789.
  2. 2. How do you let a standalone docker run container join an overlay network?

    Only services can use an overlay unless it was created as attachable.
  3. 3. Is traffic between containers on an overlay encrypted by default?

    Data-plane encryption is opt-in per network and uses IPsec ESP.

Summary

  • Overlay networks span the swarm using VXLAN (UDP 4789) and gossip (7946).
  • A network appears on a node only when a task there uses it.
  • Use separate overlays to segment tiers; make them attachable only for debugging.
  • Encrypt with --opt encrypted; watch MTU and subnet overlaps.

Next: Service Discovery and VIPs