Chapter 10: The Kubernetes Networking Model - CNI, kube-proxy & Services

“Kubernetes networking is a promise the platform keeps with a thousand little machines: every pod can reach every pod, everywhere, on a flat network - and the promise is kept by CNI plugins, iptables chains, and increasingly, eBPF.”

Everything in Parts I-III ran on one host. This chapter opens the cluster: what Kubernetes promises about networking, how the CNI layer keeps that promise at pod level, how Services add a stable name-and-port abstraction, and how kube-proxy implements it with iptables and IPVS - the machinery eBPF was built to replace. If Part IV has a thesis, it is this: Kubernetes networking is a translation problem - translating the cluster’s logical model (pods, services, policies) into the kernel’s physical model (interfaces, IP addresses, NAT tables) - and every layer of that translation is a place where eBPF can be faster and more correct than the machinery it replaces.

10.1 The Four Promises

The Kubernetes networking model is famously short - four rules, from the documentation, with no implementation details:

1. Every pod gets its own IP address, cluster-wide, unique.
2. Every pod can reach every other pod at that IP, on any node, without NAT in between.
3. Every pod can reach every service (and every node, and the outside world).
4. The pod’s view of the network is flat: it does not know about nodes, tunnels, or the network that carries its traffic.

The consequences are what make the model hard. Because pods move between nodes (rescheduling, scaling), their IPs must be portable: the network must carry pod-to-pod traffic across node boundaries. Because pods can be created and destroyed at will, address allocation must be dynamic. And because rule 4 forbids NAT between pods, the cluster needs an overlay or a flat L2/L3 fabric - which is where CNI comes in.

10.2 CNI: The Interface Between Runtime and Network

CNI (Container Network Interface) is the plugin contract between the container runtime and a network implementation. The runtime calls the CNI plugin binary with a JSON spec on stdin (“add this container to the network”); the plugin must create the pod’s network: a veth pair (one end in the pod’s network namespace, one end on the host), an IP address from its allocation pool, routes, and any encapsulation. The pod’s network namespace is then one hop from the host - which is why the pod’s view of the network (rule 4) is preserved: the pod sees its veth, the host sees the other end, and everything beyond is the plugin’s business.

The two families of CNI implementations map to the two models from Chapter 1:

• Routing-based (Calico, Cilium in routing mode): the plugin assigns real IPs and programs routes on the host (and the fabric), so pod-to-pod traffic is routed like any IP traffic. Fast, no encapsulation tax, but the fabric must be route-aware.
• Overlay-based (Flannel, Weave, Cilium in VXLAN mode): the plugin wraps pod traffic in an encapsulation (VXLAN/Geneve) and tunnels it across the cluster. Works on any fabric, at the price of header overhead and a per-packet encapsulation cost.

The packet path inside a node is the Chapter 1 path with names: pod veth -> host veth -> tc hooks -> routing -> (encapsulation if overlay) -> node NIC. Every hook in this book is on that path, and every CNI that uses eBPF (Cilium is the famous one) is putting the Chapter 5-9 programs exactly there.

10.3 Services: The Stable Abstraction

Pods die; the service stays. A Service is a stable virtual IP (ClusterIP) and port that fronts a set of pods selected by label; when a client connects to the service, the cluster routes the connection to one of the backing pods. The routing is the whole story: the service does not exist as a network object. It is a rule - “connections to 10.96.0.10:80 go to one of {pod-a, pod-b, pod-c}” - that some component must turn into packet-level decisions on every node.

That component has been kube-proxy, and its history is the history of the networking problem eBPF solves:

• Userspace mode (the original): kube-proxy owned the service IP and proxied connections in userspace. Correct, slow, and now ancient.
• iptables mode (the default for a decade): kube-proxy programs the kernel’s netfilter with a chain per service and per pod. Each packet to a service walks a linear chain of rules; the chain for a cluster with many services is long, and the first packet of a connection pays for all of it. Correct, and the source of “why is my cluster’s first-packet latency bad” mysteries.
• IPVS mode: kube-proxy programs the kernel’s IPVS (IP Virtual Server) tables instead - a hash-based LB engine - replacing the linear walk with a lookup. Better, but still netfilter, still a userspace agent reconciling state, and still per-node policy that must be kept in sync.

The recurring cost is translation: a userspace agent (kube-proxy) watches the API server, computes rules, and pushes them into the kernel through a slow, indirect mechanism. Every watch event, every resync, every partial failure is a chance for the kernel’s view to lag the cluster’s truth. That is the job eBPF takes over in Chapter 11.

10.4 The iptables Walk: What a Packet Actually Does

To appreciate the replacement, walk what a packet does today under kube-proxy in iptables mode:

client pod -> service ClusterIP 10.96.0.10:80
  -> OUTPUT chain (the pod's own node)
  -> KUBE-SERVICES: match 10.96.0.10
  -> KUBE-SVC-XXXX (one chain per service): random selection among pods
     -> KUBE-SEP-AAAA (endpoint A): DNAT to pod-a:8080
  -> the packet is now addressed to pod-a
  -> FORWARD chain, then the CNI's data path (veth, routes, overlay)
  -> pod-a's node, pod-a's veth, pod-a's process

Every service adds a chain; every endpoint adds a chain; the -m statistic --probability randomisation is a chain decision. The cost model is the one Chapter 6 established: each rule is a linear comparison, and the first packet of every connection walks from the top. With hundreds of services, the first-packet walk is the tail latency you can measure. The eBPF replacement (next chapter) is the same decision - “10.96.0.10:80 goes to pod-a” - but as a hash lookup in a BPF map: one lookup, O(1), before the packet ever reaches netfilter.

10.5 DNS and the Service Ecosystem

One more piece of the model before Cilium: DNS. Pods resolve service names through the cluster’s CoreDNS, and the resolution is the control plane of service discovery - it happens rarely. The data plane - routing connections to the resolved IP - happens per packet. The separation matters because it is the same separation this book has drilled since Chapter 3: control plane (what resolves, what changes) and data plane (what executes per packet). Cilium’s innovation is to compress more of the control plane’s output into BPF maps so the data plane does less work, and to observe the data plane (Hubble, Chapter 12) so the control plane’s mistakes are visible.

10.6 Service Types: ClusterIP, NodePort, LoadBalancer, ExternalName

The Service abstraction has four flavours, and each one changes the translation problem - which is why the eBPF replacement must handle all of them, not just ClusterIP:

• ClusterIP: the default. A virtual IP reachable only inside the cluster. The rule is “connections to 10.96.0.10:80 go to a backend pod”.
• NodePort: a port opened on every node (e.g. 30080), forwarded to the ClusterIP, then to a backend. The translation problem gains one hop: node -> ClusterIP -> pod.
• LoadBalancer: the cloud provider provisions an external LB (an ELB/NLB, or MetalLB in bare metal) whose backends are the nodes; traffic arrives as NodePort traffic. One more hop in the provider’s control plane, the same kernel machinery underneath.
• ExternalName: no IP at all - a DNS CNAME. The data plane never sees it; the translation is pure DNS.

The engineering consequence: a kube-proxy replacement must implement the whole family - external traffic policy (ExternalTrafficPolicy: Local preserves the client IP but pins the service to one node), session affinity (stickiness by client IP), and health checking of backends. When Chapter 11 shows Cilium’s service maps, remember that each service type is a different key shape in those maps - ClusterIP keys, NodePort keys, and the external-TrafficPolicy decisions are map entries, not code paths.

10.7 What You Can Do Today, With What You Know

Before moving on, notice that you already know how to build a kube-proxy replacement from the previous five chapters:

• Service routing at the socket layer: CgroupSockAddr (Chapter 9) rewrites the connect to a backend pod - exactly the socket-level LB of Chapter 9.4.
• Service routing at the packet layer: a tc or XDP program (Chapters 5-6) does the ClusterIP->pod DNAT with a hash map keyed by the service tuple - exactly the iptables walk, but O(1).
• Session affinity: a per-client hash in the map (Chapter 4’s HASH with a client-IP key) instead of -m statistic randomness.
• Observability: the ring buffers (Chapter 4) and tracepoints (Chapter 7) report who connected to what - the beginnings of Hubble.

The hands-on verification that makes the model concrete: on any cluster with kube-proxy in iptables mode, iptables -t nat -L KUBE-SERVICES -n shows the chain walk of Section 10.4 - one line per service, one chain per endpoint, and the statistic rules that randomise. Count the lines, then time the first packet of a new connection (hping3 -S or nc -vz against a fresh ClusterIP). That measured walk is what Chapter 11 replaces with a hash lookup.

Chapter 11 shows the production version of exactly this, from the project that made eBPF famous: Cilium.

Hands-On Lab

# 1. On any cluster with kube-proxy in iptables mode, see the walk.
kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --target-port=80
iptables -t nat -L KUBE-SERVICES -n     # one chain per service
iptables -t nat -L KUBE-SVC-XXXXXX -n   # one chain per endpoint

# 2. Measure the first-packet cost the chains create.
kubectl run client --image=nicolaka/netshoot -- sleep 3600
kubectl exec client -- sh -c 'time curl -s http://web >/dev/null'   # cold
kubectl exec client -- sh -c 'time curl -s http://web >/dev/null'   # warm

The cold-vs-warm gap is the chain walk; Chapter 11 replaces it with a map lookup whose cost does not grow with the number of services.

Summary

• The Kubernetes networking model is four promises: unique pod IPs, full pod-to-pod reachability without NAT, pod-to-service reachability, and a flat pod view of the network.
• CNI is the plugin contract that builds the pod’s network (veth, IP, routes) - routing-based or overlay-based, both running on the Chapter 1 packet path.
• Services are rules, not network objects: a virtual IP that some component must translate into per-packet decisions.
• kube-proxy has done that with userspace, iptables (linear chains - the classic tail-latency story), and IPVS - all userspace agents reconciling kernel state.
• The eBPF thesis: the same decisions as hash lookups in BPF maps, made in the kernel, before netfilter - the subject of Chapter 11.

Next: Chapter 11 is the main event of Part IV - Cilium, the eBPF data plane that replaced kube-proxy, chain by chain, map by map.