Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 78 ยท Services and Networking

How Services Really Work: kube-proxy

Look inside Kubernetes Services: kube-proxy, virtual IPs, DNAT, iptables vs IPVS vs nftables modes, connection tracking and session affinity.

Earlier you learned that a Service has a virtual IP that no network card owns, and yet connecting to it works. Someone has to make that trick happen on every node, and keep it correct while Pods appear and disappear. That job belongs to kube-proxy, a small agent that runs on every node. Despite the name, it usually does not proxy any traffic itself: it programs the node's kernel, and the kernel does the forwarding.

Understanding this layer turns several mysteries into ordinary facts: why a ClusterIP cannot be pinged, why long-lived connections stay on one Pod, why large clusters see slow Service updates, and why some networking plugins can replace kube-proxy completely. You will also run a Python model of a NAT table that picks a backend the same way the iptables rules do.

The commands and rule listings here are read-only and written from knowledge of how the Linux modes behave; nothing here ran on a cluster. The Python stand-ins model the selection logic only, not packet processing in the kernel.

The job of kube-proxy

Watch Services and EndpointSlices from the API.
Program Write forwarding rules into the node kernel.
Keep in sync Update the rules when Pods change.

The kernel moves the packets, not kube-proxy.

1 / 2

The virtual IP is a rule, not a device

When a Pod connects to the Service address 10.96.14.20:80, the packet leaves with that destination. Somewhere on the path, in the node's kernel, a rule recognizes the destination, picks one of the real endpoints and rewrites the destination address to 10.244.2.7:8080. This rewrite is called DNAT (destination network address translation). The reply packets are rewritten back so the client believes it spoke to the Service address all along.

sequenceDiagram
  participant C as Client Pod
  participant K as Node kernel rules
  participant B as Backend Pod
  C->>K: connect to 10.96.14.20:80
  K->>K: choose endpoint, rewrite destination (DNAT)
  K->>B: 10.244.2.7:8080
  B-->>K: reply
  K-->>C: reply, source rewritten to 10.96.14.20:80

The client never learns the real Pod address. The kernel remembers the rewrite for the life of the connection.

Because the address exists only as a rule that matches TCP, UDP or SCTP flows to a particular port, there is no device to answer a ping, and no process listening on the virtual IP. This is why ping against a ClusterIP fails while curl succeeds.

kube-proxy itself is typically deployed as a DaemonSet in kube-system. It watches Services and EndpointSlices and reacts to every change:

Terminal
kubectl get daemonset kube-proxy -n kube-system
kubectl get configmap kube-proxy -n kube-system -o yaml | grep -i mode

Some clusters do not run it at all, because the CNI plugin (for example one based on eBPF) implements Services itself.

iptables mode

In iptables mode, kube-proxy writes a tree of rules into the nat table. For each Service there is a chain, and for each endpoint another chain that performs the DNAT. A listing for the web Service with three endpoints looks roughly like this (names and hash suffixes shortened; read-only):

TEXT
-A KUBE-SERVICES -d 10.96.14.20/32 -p tcp --dport 80 -j KUBE-SVC-WEB

-A KUBE-SVC-WEB -m statistic --mode random --probability 0.33333 -j KUBE-SEP-1
-A KUBE-SVC-WEB -m statistic --mode random --probability 0.50000 -j KUBE-SEP-2
-A KUBE-SVC-WEB -j KUBE-SEP-3

-A KUBE-SEP-1 -p tcp -j DNAT --to-destination 10.244.1.5:8080
-A KUBE-SEP-2 -p tcp -j DNAT --to-destination 10.244.2.7:8080
-A KUBE-SEP-3 -p tcp -j DNAT --to-destination 10.244.1.9:8080

The probabilities look odd until you realize the rules are checked in order, one after another. The first rule fires one time in three. If it does not fire, the second rule gets a half chance (it faces only two remaining endpoints), and the last rule catches everything left over. The result is an even one-third split: first gets 1/3, second gets 2/3 times 1/2 = 1/3, third gets the remaining 1/3.

The randomness is applied once, when a new connection starts. The kernel's connection-tracking table (conntrack) then remembers the chosen rewrite, so every later packet of that connection follows the same path. This has an important consequence: load balancing is per connection, not per request.

Strengths and limits of iptables mode:

  • It is simple, ships everywhere and is battle tested.
  • Rules are evaluated linearly, so cost grows with the number of Services and endpoints. With tens of thousands of Services, rule updates and lookups get slow, and kube-proxy rewrites large rule sets on each change.
  • Balancing is random, with no awareness of how busy a Pod is.

IPVS mode

IPVS is a layer-4 load balancer built into the Linux kernel. In this mode kube-proxy creates an IPVS "virtual server" for each Service address and adds the Pods as "real servers". Lookups use hash tables, so the cost stays nearly flat as the cluster grows, and IPVS offers more scheduling algorithms than random choice: round robin, least connections, source hashing and others. kube-proxy still uses a little iptables and ipset for tasks like masquerading. IPVS needs kernel modules and has been a source of subtle bugs; recent Kubernetes releases are moving away from it, so check the release notes of your version before choosing it for a new cluster.

nftables mode

nftables is the Linux kernel's replacement for iptables. The kube-proxy nftables mode keeps the same idea (rules that DNAT to endpoints) but uses nftables features such as maps and verdict maps, so lookups are fast and updates change only what is needed. It became generally available in Kubernetes 1.33 and is the intended long-term replacement for the iptables mode on modern kernels. Rules written by one mode are not visible to the others, and you select the mode in the kube-proxy configuration.

Mode Lookup cost Balancing choices Remarks
iptables grows with the rule count random The long-standing default
IPVS roughly constant many algorithms Needs kernel modules; being phased out
nftables roughly constant random The modern default direction

Session affinity

Sometimes a client must keep talking to the same Pod, for example with an application that holds in-memory sessions. Set it on the Service:

YAML
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

With ClientIP, connections from the same source address go to the same endpoint until the timeout (three hours by default) passes without traffic. The default is None. Affinity is a convenience, not a guarantee: if the chosen Pod disappears, the client is moved, and many clients behind one NAT address all land on the same Pod. Stateless applications and shared session storage are the more robust solution.

What the "per connection" rule means in practice

Because the choice happens when a connection opens, one long-lived connection stays on one Pod for its whole life. HTTP/2 and gRPC clients typically multiplex all requests over a single connection. Result: with such traffic, a Service spreads connections evenly but requests can be extremely lopsided, and newly added Pods (after a scale-up) receive nothing from existing clients. The usual fixes are client-side balancing, a proxy or Ingress that balances per request, or a service mesh. Reconnecting after a maximum connection age also helps.

Stand-in: a NAT table picking a backend

The first Python example rebuilds the probability chain you just read and applies it to 3,000 new connections. It also remembers a choice per client address, like sessionAffinity: ClientIP. It uses a seeded random generator, so the counts are repeatable, but your cluster will vary.

Python The iptables probability chain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three counts land near 1,000 each, even though no rule counts anything: the cascading probabilities do the work.

Stand-in: why long connections skew the load

The second example compares two ways of sending traffic. Four clients each send 100 requests. In the first run each client opens one connection and reuses it; in the second every request opens a new connection. Halfway through, the Service scales from two to four backends, but the existing connections were already pinned.

Python Per-connection balancing and scale-up

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the first line, the new backends C and D receive no traffic at all, since every client's connection was opened before the scale-up. In the second line all four share the load once they exist. This is the behavior behind many "I scaled up but nothing changed" tickets.

Common mistakes

  • Debugging a Service by pinging its ClusterIP.
  • Expecting request-level balancing from a Service with HTTP/2 or gRPC clients.
  • Scaling a Deployment and assuming existing clients will rebalance.
  • Depending on sessionAffinity instead of making the application stateless.
  • Running a very large cluster in iptables mode without checking rule-sync latency, or relying on IPVS without reading the release notes of your Kubernetes version.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does kube-proxy mainly do on each node?

    kube-proxy watches Services and EndpointSlices and installs iptables, IPVS or nftables rules; the kernel then does the DNAT.
  2. 2. In iptables mode with three endpoints, why is the second rule's probability 0.5 and not 0.33?

    The first rule takes 1/3. The second rule is reached by the remaining 2/3 and takes half of them, which is again 1/3 overall.
  3. 3. A gRPC client keeps one connection open and the Deployment is scaled from 2 to 6 Pods. What happens to that client's traffic?

    Endpoint selection happens per connection and conntrack pins the rewrite, so an existing connection stays where it is until it is closed.
  4. 4. Why does ping to a ClusterIP not get a reply?

    The Service address is matched by rules for specific protocols and ports. Nothing owns the address, so there is nothing to answer an echo request.
  5. 5. What does sessionAffinity: ClientIP do?

    Connections from the same source address are sent to the same endpoint until the affinity timeout (default three hours of inactivity) expires.

Summary

  • kube-proxy runs on every node, watches Services and EndpointSlices, and programs the kernel; some CNI plugins replace it.
  • A ClusterIP is a DNAT rule, not a device: connections to it are rewritten to a chosen Pod IP.
  • iptables mode chains probabilistic rules, IPVS uses a kernel load balancer, and nftables is the modern successor.
  • Balancing happens per connection, remembered by conntrack, so long-lived connections can be uneven and do not move after a scale-up.
  • sessionAffinity: ClientIP pins a client to one endpoint, but stateless design is more robust.

Next: routing web traffic by host and path with Ingress and Ingress Controllers.