Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Chapter 12: Cilium Service Mesh, Security & Hubble

“A service mesh is a decision about who is allowed to talk to whom, and how - executed without a proxy in the path. Cilium’s answer is: the policy is a map, the execution is the kernel, and the proof is in the flows.”

Chapter 11 built the data plane. This chapter builds the policy layer on top of it: how Cilium turns the cluster’s security intent into the mechanisms of Chapters 4-9 - network policies compiled into map entries and executed by the tc/sockmap programs, mTLS between pods without a userspace proxy for the data path, the L7 proxy that appears only when L7 rules demand it, and Hubble, the observability layer that turns the ring buffers of Chapter 4 into the cluster’s flow log. When you finish this chapter you will understand the full arc of the book: from one packet on one NIC, to the security policy of an entire cluster, through the same primitives.

12.1 Network Policy: Intent Compiled into Maps

A Kubernetes NetworkPolicy says “pods with label app=api may receive from pods with label app=frontend on port 443”. Cilium extends this with its own CiliumNetworkPolicy (L3-L7, CIDRs, FQDNs, TLS-aware). The architecture is the one from Chapter 11: the policy is data, not code. When a policy lands, the agent:

  1. Resolves the labels to identities. Every pod gets an identity (a 24-bit number) derived from its labels. Identities are the currency of the policy engine: the map key is an identity pair, not a pod pair, which is what makes policy survive pod churn - a new pod with the same labels inherits the policy without a new rule.
  2. Compiles the policy into map entries. The endpoint’s policy maps (LXC_ID -> allowed identity -> allowed L4/L7) are updated. The tc program on the pod’s veth (Chapter 6) consults them per packet.
  3. Enforces at both ends. Cilium enforces egress policy at the source pod and ingress at the destination pod - two independent checks, so a compromised node cannot bypass one end’s policy.

The per-packet cost is what Chapters 4-6 prepared you for: a tuple -> identity lookup, a policy map lookup, an L4 check - all O(1) map operations. There is no rule engine on the data path. This is the architectural statement that separates eBPF policy from iptables policy: iptables evaluates a list of rules per packet; Cilium evaluates a set of maps per packet. The worst case of the former grows with policy size; the worst case of the latter does not.

12.2 The L7 Story: Proxy Only When You Ask For It

L4 policy (IP, port, identity) is free with the maps above. L7 policy (HTTP paths, Kafka topics, gRPC methods) needs to see the payload, and payload parsing is not a verifier-friendly activity. Cilium’s answer is the honest hybrid this book has been teaching since Chapter 8:

  • The default data path stays in the kernel. L4 policy, routing, LB - all in BPF, no proxy.
  • An Envoy-based proxy (cilium-envoy) is deployed per node, and a packet whose flow matches an L7 rule is redirected to it - but only that flow, and only at the socket level (sockmap, Chapter 8), so the redirect is a map lookup, not a packet-level hairpin.
  • The policy engine decides which flows need L7 by inspecting the policy maps before the data path decides.

The design principle to internalise: the proxy is a fallback for the cases the kernel cannot cheaply handle, not the default path. L7 rules are the exception in most clusters; paying for them on every packet is what the older service meshes did, and it is precisely the cost Cilium removes by making the proxy opt-in per flow.

12.3 mTLS Without the Proxy Tax

Mutual TLS between pods is the security baseline of a mesh: every connection is authenticated and encrypted in both directions. The classic mesh does this in a sidecar proxy - which means every byte crosses the proxy, and the mesh’s latency and CPU story is dominated by the proxy. Cilium’s approach splits the problem the same way the data path does:

  • The handshake and certificate lifecycle are control plane: the agent manages certificates (via the cert-manager integration or a CA), and the first handshake of a connection can involve the agent/Envoy for the certificate exchange.
  • The steady-state encryption is kernel-level: the kernel’s IPsec (XFRM) is programmed with the negotiated keys, so the data path is encrypt/decrypt in the kernel’s crypto stack - no userspace proxy in the byte path, with the kernel’s crypto acceleration (AES-NI, etc.) applying.

The honest trade: kernel IPsec gives you per-packet performance and no proxy CPU, at the cost of features the proxies offer (protocol-level inspection, retries, traffic shaping) - which is exactly why the L7 proxy of Section 12.2 coexists with it, and why the policy decides which path a flow takes. The mesh is a policy-compiled routing decision, like everything else in this chapter.

12.4 Hubble: The Ring Buffer, Scaled to a Cluster

Hubble is Cilium’s observability layer, and for this book it is the satisfying payoff of Chapter 4’s ring buffer. The data path programs emit one event per significant flow transition - connection established, policy applied, packet dropped, reply seen - through the ring (or the older perf buffer) to the agent, which enriches the events with Kubernetes context (namespaces, labels, identities) and serves them through the Hubble API and UI. A flow entry looks like:

K8s Namespace: default   K8s Pod Name: frontend-7d8f5b
Source IP: 10.0.1.5      Destination IP: 10.0.2.9
Destination Service: api:443
Verdict: FORWARDED       Policy: allowed-by: {"l4":["443/TCP"]}

The design is the one you have used since Chapter 4: the kernel produces events, userspace enriches, humans query. The eBPF part is the cheap part (a ring write per event, amortised); the value is the enrichment, which is why Hubble’s API returns kubectl-shaped answers and not raw tracepoint dumps. When the capstone ships its own flow events through its own ring (Chapter 16), it is building a one-node Hubble.

12.5 The Policy You Can Write Today

The hands-on minimum that makes the chapter real:

# A policy that says: only pods labeled app=frontend may reach app=api on
# port 443, over mTLS, with HTTP path /healthz allowed at L7.
cat <<'EOF' | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-ingress
spec:
  endpointSelector:
    matchLabels: {app: api}
  ingress:
  - fromEndpoints:
    - matchLabels: {app: frontend}
    toPorts:
    - ports: [{port: "443", protocol: TCP}]
      rules:
        http:
        - method: GET
          path: "/healthz"
EOF

# Watch the policy become data: the endpoint's policy map is updated, and
# hubble shows the flows it allows.
cilium endpoint list
hubble observe --from-pod frontend --to-pod api

The two commands are the chapter in miniature: the policy is a YAML object until the agent compiles it; then it is a set of map entries, and the proof of enforcement is a flow log. That is the entire arc of this book, in two commands.

12.6 Where the Mesh Ends and Your Code Begins

The service mesh is the top of the stack this book climbs - but notice what it is not: it is not a separate system. It is the same hooks (Chapters 5-9), the same maps (Chapter 4), and the same two-plane discipline (control plane compiles, data plane executes) that you have been using since Chapter 3. When the capstone builds its mesh in Chapter 16, it will assemble: XDP LB (Chapter 5), sockmap redirect (Chapter 8), a policy map (Chapter 4), and a flow ring (Chapter 4) - and it will be a Cilium in miniature. The distance from “I wrote a packet filter” to “I understand a service mesh” is exactly the distance from Chapter 5 to this chapter - the primitives did not change; only the scale of the intent did.

Hands-On Lab

# 1. Apply the L4 policy from Section 12.5, then generate a matching flow.
kubectl apply -f api-ingress.yaml
kubectl run client --image=nicolaka/netshoot -- sleep 3600
kubectl exec client -- curl -s https://api:443/healthz

# 2. Watch the policy become data and the flow become a log.
cilium endpoint list               # policy enforcement state per pod
hubble observe --from-pod client   # the flow, verdict and allowed-by
kubectl exec client -- curl -s http://api:80/    # non-allowlisted: blocked
hubble observe --deny                             # the denial appears

The proof that policy is compiled, not interpreted: the per-packet cost does not change when you add the 10th or the 100th policy rule - it is a map lookup either way (Chapter 12.1).

Summary

  • Network policy is compiled into map entries: identity pairs, not pod pairs, key the policy maps; enforcement happens at both ends of every connection, per packet, O(1).
  • The L7 proxy is opt-in per flow: the default data path is BPF; only flows matching L7 rules are socket-redirected to per-node Envoy.
  • mTLS splits the problem: certificate lifecycle in the control plane, steady-state encryption in kernel IPsec - no proxy in the byte path.
  • Hubble is the ring buffer scaled to a cluster: kernel events, userspace enrichment, human queries - the observability pattern of Chapter 4, productionised.
  • The mesh is not a separate system: it is the same hooks, maps, and two-plane discipline as everything before - the capstone proves it.

Next: Part V turns from features to operations. Chapter 13 is the performance methodology - the per-packet budget of Chapter 1, measured and spent well.