Chapter 11: Cilium - The eBPF Data Plane for Kubernetes

“Cilium is what happens when the chapters of this book get shipped: XDP and tc programs on every node, sockmap in the socket layer, cgroup hooks on every pod, and a control plane that compiles the cluster’s intent into BPF maps - no iptables, no kube-proxy, no userspace proxy on the hot path.”

Chapter 10 described the problem: a userspace agent translating the cluster’s logical model into kernel rules through iptables. This chapter describes the answer that made eBPF famous: Cilium, the CNI and network data plane built entirely on the mechanisms of Parts I-III. By the end you will be able to read a Cilium architecture diagram the way you read a packet path - as a set of BPF programs, maps, and hooks you already know - and you will understand why the project’s claim “no iptables, no kube-proxy” is a statement about where the decisions are made, not a marketing slogan.

11.1 The Architecture: Agent, Datapath, and the API Server

Cilium’s components map cleanly onto the two-plane discipline of this book:

• The agent (cilium-agent, a daemonset on every node) is the control plane: it watches the Kubernetes API server (services, endpoints, pods, policies), and compiles those objects into BPF programs and map entries. It loads the programs, attaches them to the node’s interfaces and cgroups, and updates the maps as the cluster changes. This is the Aya user in Chapter 3, scaled to a cluster.
• The datapath is the set of BPF programs the agent attaches: an XDP program for the node’s external interface, tc programs on the veth pairs of every pod, sockmap programs in the socket layer, cgroup hooks on every pod’s cgroup. These are the programs of Chapters 5, 6, 8 and 9, compiled from C (with Rust in the tooling increasingly) into the same ELF objects the Aya loader understands.
• The API server is the source of truth; the agent is the translator; the maps are the compiled output. When a Service is created, the agent does not add an iptables chain - it inserts an entry into the BPF service map, and the data path changes on the next packet.

The crucial difference from kube-proxy is the unit of work. kube-proxy programs netfilter, a state machine with its own view of the world; Cilium programs BPF maps, which are just data the kernel consults per packet. Updating a map is atomic, cheap, and immediately visible - there is no “wait for the iptables resync”, no half-applied chain, no per-node drift beyond the agent’s own watch latency.

11.2 The Service Data Path: From ClusterIP to Pod

Follow a connection to a Service under Cilium, using the hook ladder of Parts I-III:

1. Socket layer (new connections): a pod’s connect() to 10.96.0.10:80 hits the CgroupSockAddr program (Chapter 9), which looks up the service in a BPF map and rewrites the address to a backend pod. The kernel opens the real connection; no packet ever carries the ClusterIP, and no NAT runs on the data path.
2. Packet layer (everything else): traffic that arrives without a socket-level rewrite - forwarded traffic, traffic from outside, UDP - is handled by the tc programs on the veths and the XDP program on the node’s NIC: parse the tuple, hash it (consistent hashing for stickiness, Maglev-style for even spread), look up the backend in the service map, DNAT/encapsulate as needed, redirect.
3. Conntrack: Cilium maintains connection state in BPF maps - its own conntrack (CT), keyed by the 4-tuple, recording the mapping between the client’s view (ClusterIP) and the backend’s view (pod IP) so that reply packets are translated back without recomputation. This is Chapter 7’s netfilter conntrack, reimplemented in the map vocabulary of Chapter 4 - and it is why Cilium can claim per-node conntrack with no dependence on nf_conntrack.

The result is the Chapter 10 iptables walk reduced to one hash lookup, one map entry, one conntrack insert - before the packet reaches netfilter. The first-packet tail latency that iptables chains produced (Chapter 10.4) is replaced by the deterministic cost of a map lookup.

11.3 Why “No kube-proxy” Matters: The Numbers

The performance claims deserve the honest treatment of Chapter 13, but the shape of the argument is clear from the architecture alone:

• PPS capacity: the data path is XDP + tc + sockmap - the hooks this book measured as ~100 ns (drop), ~0.5-1 us (redirect), ~1-5 us (tc end-to-end). iptables’ linear chain walk on first packets is the difference between hundreds of thousands of new connections per second and millions, and between Mpps-scale and tens-of-Mpps-scale forwarding.
• First-packet latency: a map lookup vs a chain walk - the difference between microseconds and milliseconds on cold services, which is the classic “why is the first request slow” story in iptables clusters.
• CPU: no kube-proxy process on every node, no netfilter traversal on every packet, no conntrack table in the netfilter subsystem - the kernel does less per packet because the decision is a lookup, not a traversal.

The honest counterpoints (Cilium’s docs say them too): the agent is more complex than kube-proxy (more moving parts in the control plane), the map tables must be sized and evicted (Chapter 4’s LRU_HASH discipline), and the kernel must be recent enough for the features used (BTF, the modern hooks - the platform requirements of the foreword).

11.4 The Datapath Programs, Mapped to This Book

Cilium’s source tree (bpf/ directory) is the best real-world reading companion to Parts I-III. The mapping is direct:

Read any of them with the Chapter 5 parse-bounds-lookup pattern in mind and they stop being “the Cilium codebase” and become “a bigger version of the programs in this book”. That is the point of learning the primitives before the product.

11.5 Running Cilium: The Hands-On Minimum

Cilium is the one component of this book you should run before the capstone, because the capstone runs inside it. The minimal path:

# A kind cluster with Cilium as the CNI (the capstone's environment).
kind create cluster --name ebpf-book
helm repo add cilium https://helm.cilium.io
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set hostServices.enabled=true

# Verify the data path is eBPF, not iptables.
cilium status
kubectl -n kube-system get pods -l k8s-app=cilium

# The moment of recognition: watch the BPF programs the agent loaded.
bpftool prog list | grep -E 'xdp|sched_cls|sock'   # Chapter 14's tool

When bpftool prog list shows Cilium’s sched_cls programs attached to veth pairs, you are looking at Chapter 6’s hook with a daemonset in front of it. When cilium status reports KubeProxyReplacement: True, you are looking at Chapter 10’s problem solved by Chapter 4’s maps. The capstone (Chapter 16) builds a miniature of exactly this on the same cluster.

11.6 The Datapath Design Review: What to Read, and Why

The moment the model meets the code, use the source tree as a textbook. The bpf/ directory is organised exactly like Part II of this book, and reading it with the Chapter 5-9 patterns in mind turns “the Cilium codebase” into “a bigger version of the programs I have written”:

• bpf_xdp.c - the XDP program of Chapter 5 at node scale: parse the frame with data/data_end, check the tuple against a service map, XDP_REDIRECT. Find the bounds discipline you practised in Chapter 5, and watch what a production program adds: frag handling, IPv4/IPv6, and the encapsulation decisions.
• bpf_lxc.c - the tc program of Chapter 6 on every veth: the per-endpoint policy lookup (the identity pair of Chapter 12), the conntrack insert, the L4 check. This is Chapter 9’s cgroup filter generalised from “this pod” to “any endpoint, identified by a 24-bit identity”.
• bpf_sock.c - the cgroup hooks of Chapter 9: the socket-level service rewrite you studied in Section 11.2, in production form.
• bpf_sockmap.c - the sockmap of Chapter 8, with the same fail-open discipline around bpf_msg_redirect_map.

The review habit that pays off: for each program, answer the three questions this book has trained you to ask - which hook? which maps? what is the fail-open path? The answers are visible in the source within a few hundred lines, because Cilium is written with the same discipline the CODING_STANDARDS of this book enforce: bounds first, maps for state, actions as the only output.

11.7 ClusterMesh: The Multi-Cluster Extension

The last architectural note before Chapter 12: ClusterMesh extends the same model across clusters. Each cluster runs its own agent and datapath, and a control plane shares service metadata (identities, endpoints, policies) between them - so a service in cluster A can have backends in cluster B, with the routing still decided by the same BPF maps, just with tunnel endpoints for the inter-cluster leg. The design consequence is the one this book has repeated at every scale: the data plane never learns a new trick for multi-cluster; the control plane simply writes different map entries (a backend IP in cluster B instead of cluster A). Whatever the cluster looks like, the per-packet work stays “parse, look up, act” - and the capstone in Chapter 16 is a miniature of that statement.

Hands-On Lab

# 1. The cluster from the chapter.
kind create cluster --name ebpf-book
helm repo add cilium https://helm.cilium.io
helm install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true --set hostServices.enabled=true

# 2. The moment of recognition: the hooks of this book, with a daemonset.
cilium status                      # KubeProxyReplacement: True
kubectl -n kube-system get pods -l k8s-app=cilium
sudo bpftool prog list | grep -E 'xdp|sched_cls|sock'

# 3. Watch a Service become a map entry.
kubectl expose deployment web --port=80
sudo bpftool map dump | grep -A2 80   # the ClusterIP key appears

Every line of bpftool prog list output corresponds to a chapter in this book: xdp is Chapter 5, sched_cls Chapter 6, sock Chapters 8-9.

Summary

• Cilium is the agent (control plane) + BPF programs (data plane) architecture: the agent watches the API server and compiles cluster intent into BPF maps; the programs execute per packet.
• The service path uses the whole hook ladder: CgroupSockAddr for new connections, XDP/tc for packet traffic, sockmap for the mesh, BPF-map conntrack for replies - the iptables walk replaced by hash lookups.
• “No kube-proxy” means the decisions moved from netfilter chains to BPF maps: first-packet latency and PPS capacity change because the unit of work changed, not because of magic.
• The datapath programs map 1:1 to this book’s hooks - reading Cilium’s bpf/ is reading a larger Chapter 5-9.
• Run it: kind + helm install cilium, then bpftool prog list is the moment the model meets the machine.

Next: Chapter 12 layers the rest of the cloud story on the data path - the Cilium service mesh, network policies, mTLS, and Hubble observability.