Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 78 ยท Services and Networking

Pod Networking and CNI

Understand the Kubernetes network model, how CNI plugins such as Calico, Cilium and Flannel give Pods IPs, how pod CIDRs are split, and overlay versus routed networks.

Every other networking feature in Kubernetes, from Services to Ingress to NetworkPolicy, rests on one promise: every Pod has its own IP address and can reach every other Pod directly. Kubernetes states that promise but does not implement it. The work of wiring up virtual network interfaces, assigning addresses and moving packets between machines is done by a network plugin that follows the Container Network Interface (CNI) standard.

This lesson explains the rules of the Kubernetes network model, what a CNI plugin actually does when a Pod starts, how the cluster's address space is carved into per-node ranges, and the main trade-off between overlay and routed designs. You will finish by running a small address allocator to see the arithmetic.

The Kubernetes network model

The model is short enough to memorize:

Four rules of the network model

One IP per Pod All containers in a Pod share it.
No NAT between Pods A Pod sees the real source IP of its peer.
Node to Pod works Agents on a node can reach its Pods.
Plugin decides how The cluster only defines the contract.

Services and NetworkPolicy are built on top of these rules.

1 / 2

Compare this with plain Docker on one host, where containers sit behind a bridge and NAT and you publish ports to reach them (see Docker network drivers). Kubernetes avoids port mapping between Pods: two Pods can both listen on port 8080 because they have different IPs. Pod IPs are still ephemeral; when a Pod is replaced it gets a new address, which is exactly why Services exist.

What happens when a Pod starts

When the kubelet needs a new Pod, it asks the container runtime to create a sandbox: a tiny container (historically called the pause container) whose only job is to own the Pod's network namespace. All the real containers then join that same namespace, which is why they share an IP and talk over localhost (see Pods).

Then the runtime calls the CNI plugin:

sequenceDiagram
  participant K as kubelet
  participant R as container runtime
  participant C as CNI plugin
  K->>R: Create Pod sandbox
  R->>R: Create network namespace
  R->>C: ADD (namespace, Pod name, config)
  C->>C: Create veth pair, assign IP, set routes
  C-->>R: Result: IP 10.244.1.7/24
  R-->>K: Sandbox ready
  K->>R: Start containers in the namespace

The CNI plugin is called with ADD when a sandbox is created and DEL when it is removed.

The CNI standard is deliberately small. The runtime finds the plugin configuration as JSON files in a directory (conventionally /etc/cni/net.d), runs the plugin executable (conventionally from /opt/cni/bin) with an operation (ADD, DEL, CHECK) and the details in environment variables and standard input, and reads a JSON result with the assigned addresses. The usual building block is a veth pair, a virtual cable with one end inside the Pod's namespace and the other on the node, usually attached to a bridge or routed directly.

You can look at what a node has configured (on a real node, not on this site):

Terminal
ls /etc/cni/net.d
cat /etc/cni/net.d/10-flannel.conflist
ip route

A freshly built cluster shows nodes as NotReady and CoreDNS Pods as Pending until a CNI plugin is installed. Without one, no Pod can get a network.

Pod CIDRs: how addresses are divided

The cluster has one large range for Pods, the cluster CIDR, for example 10.244.0.0/16. It is split so that each node owns a slice, its pod CIDR, often a /24. Pods on that node get addresses from the node's slice. With a /16 cluster range and /24 slices you get up to 256 nodes with about 250 Pods each; the kubelet's default limit is 110 Pods per node.

Terminal
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR

The slices must not overlap with each other, with the Service CIDR (the virtual IPs of Services, see How Services Really Work), or with your company's own networks if Pods need to reach them. Changing the cluster CIDR later is painful, so plan the sizes at creation time, especially in the cloud where you may also be limited by the provider's subnet rules.

Some plugins do not use fixed per-node slices. Cilium's cluster-pool mode and cloud-native plugins such as the AWS VPC CNI hand out addresses from other sources (for example, secondary IPs from the VPC), which makes Pod IPs routable inside the cloud network at the cost of address space and per-instance limits.

Overlay versus routed networking

The remaining question is how a packet gets from a Pod on node A to a Pod on node B.

Overlay Routed (underlay)
Idea Wrap Pod packets in another packet (VXLAN, Geneve, IP-in-IP) addressed node to node Make the physical or cloud network know the route to each node's pod CIDR
Setup Works on almost any network Needs routing control: BGP, cloud route tables or VPC-native IPs
Cost A little CPU and about 50 bytes less MTU per packet Usually the fastest, with no encapsulation
Debugging Packets inside tunnels are harder to see Pod IPs visible in network tools

An overlay is the safe default when you do not control the network. Routed designs shine when you want line-rate performance or need Pod IPs visible to outside systems. Many plugins can do both: Calico can run BGP-routed or encapsulated, depending on what the network allows.

If large transfers hang while small requests work, suspect an MTU mismatch: the tunnel adds headers, so the Pod interface MTU must be lower than the node's. Good plugins set this for you; custom installs sometimes do not.

The common plugins

  • Flannel: the simplest. Gives each node a pod CIDR and connects them with a VXLAN overlay. It has no NetworkPolicy support, which makes it popular for labs and small clusters.
  • Calico: routes Pods with BGP or encapsulates with VXLAN/IP-in-IP, enforces NetworkPolicy and offers its own richer policy objects. It has an optional eBPF data plane.
  • Cilium: built on eBPF in the Linux kernel. It enforces policies (including identity- and DNS-aware rules), can replace kube-proxy, and provides observability through Hubble.
  • Cloud plugins: the AWS VPC CNI, Azure CNI and GKE's dataplane give Pods addresses from the cloud network.

Managed Kubernetes services pick one for you. On a self-managed cluster you choose it before installing workloads, because switching later usually means rebuilding nodes.

An allocator you can run

This stand-in models what the control plane and a simple plugin do with addresses: slice the cluster CIDR into node CIDRs, then hand out the next free Pod IP on a node. It uses Python's ipaddress module and keeps the first usable address for the node's bridge, as many simple plugins do.

Python Splitting a cluster CIDR into node ranges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real Pod IPs are not reused immediately by every plugin, since a quick reuse can confuse caches and connections; this model reuses the lowest free address to keep the output easy to read. What is real: the slicing arithmetic and the "one slice per node, pods inside the slice" structure. What is a stand-in: the allocation policy, which each plugin implements in its own way.

Now check whether a route plan fits. A cluster CIDR must be large enough, and must not collide with the Service range or the networks you need to reach:

Python Checking a network plan for overlaps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overlap between pods and datacenter is the kind of mistake that appears months later as "some servers are unreachable from Pods".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statement is part of the Kubernetes network model?

    Each Pod has one IP, shared by its containers, and Pod-to-Pod traffic works without address translation.
  2. 2. A new cluster's nodes show NotReady and CoreDNS is Pending. What is the most likely missing piece?

    Without a network plugin, the kubelet cannot create Pod sandboxes with networking, so nodes stay NotReady and Pods cannot start.
  3. 3. What does a node's pod CIDR represent?

    The cluster CIDR is divided into per-node slices. Pods on a node take their IPs from its slice.
  4. 4. Why might you choose a routed network instead of an overlay?

    Routed designs skip the tunnel headers and are faster, but they require control over routing, such as BGP or cloud route tables.

Summary

  • The network model: one IP per Pod, no NAT between Pods, nodes can reach their Pods.
  • A CNI plugin implements the model; the runtime calls it with ADD and DEL when sandboxes are created and removed.
  • The cluster CIDR is sliced into per-node pod CIDRs; plan sizes early and avoid overlaps with Services and external networks.
  • Overlays (VXLAN, Geneve) work anywhere at a small cost; routed networks are faster but need routing control.
  • Flannel is simple without policy support, Calico and Cilium add policy enforcement, cloud plugins use the provider network.

Next, learn how to give Pods configuration without rebuilding images in ConfigMaps.