Chapter 16: Capstone - A Service-Mesh Data Path in Rust with eBPF

“Every chapter taught you a program. This chapter teaches you a system: the pieces you built - the XDP load balancer, the sockmap redirect, the policy map, the flow ring - wired into one Rust workspace, deployed to a real cluster, and measured. This is the book, assembled.”

The capstone is the book’s promise made concrete: a service-mesh data path in Rust with eBPF - the same architecture as the Cilium stack of Chapters 11-12, built from the primitives of Chapters 3-9, running against a kind cluster. It is deliberately minimal so that every piece is legible, and deliberately complete so that every piece is real: an XDP program that load-balances traffic by a hash, sockmap programs that redirect connections between replicas in the kernel, a policy map that decides what may pass, and a ring buffer that ships flow events to a Hubble-style observer. By the end you will have written - and run - a one-node Cilium.

16.1 The Architecture: The Book, Assembled

The capstone is one workspace, three kernel programs, and one userspace control plane:

code/ch16_capstone/
├── Cargo.toml                 # workspace
├── common/                    # shared #[repr(C)] contract (Ch. 3)
│   └── src/lib.rs             #   FlowEvent, Backend, PolicyRule
├── kernel/                    # the three data-path programs
│   ├── src/lb.rs              #   XDP: hash -> backend (Ch. 5)
│   ├── src/mesh.rs            #   sockmap SK_MSG redirect (Ch. 8)
│   ├── src/flow.rs            #   tracepoint -> flow ring (Ch. 7, 4)
│   └── src/main.rs
└── userspace/                 # the control plane
    ├── src/main.rs            #   load, attach, fill maps, drain rings
    └── src/observer.rs        #   the Hubble-style flow reader

The mapping to the book is the point of the diagram:

16.2 The Kernel Side: Three Programs, One Contract

The common crate is the contract the three programs and the control plane share - the Chapter 3 discipline in action:

Source: code/ch16_capstone/common/src/lib.rs

#![allow(unused)]
fn main() {
// code/ch16_capstone/common/src/lib.rs
// The shared contract: every type that crosses the kernel/userspace
// boundary lives here, #[repr(C)], fixed-size, one source of truth (Ch.3).
#![no_std]

// One flow event, shipped through the ring (Ch.4).
#[repr(C)]
pub struct FlowEvent {
    pub src_ip: u32,      // network byte order, as parsed from the packet
    pub dst_ip: u32,
    pub src_port: u16,
    pub dst_port: u16,
    pub verdict: u8,      // 1 = allowed, 0 = denied (fail-open default)
    pub ts: u64,          // bpf_ktime_get_ns
}

// The load-balancer table entry (Ch.5).
#[repr(C)]
pub struct Backend {
    pub ifindex: u32,     // redirect target interface
    pub flags: u32,       // reserved
}

// The policy entry (Ch.4, 12): key = dst ip, value = allow(1)/deny(0).
pub type PolicyRule = u8;

// Compile-time layout check - the contract test of Ch.14, run at build.
pub const _: () = {
    assert!(core::mem::size_of::<FlowEvent>() == 24);
    assert!(core::mem::align_of::<FlowEvent>() == 8);
};
}

The load balancer is the Chapter 5 program with a hash: instead of looking up a fixed port, it hashes the flow tuple and picks a backend from a small array - the Katran pattern of Chapter 5.6:

Source: code/ch16_capstone/kernel/src/lb.rs

#![allow(unused)]
fn main() {
// code/ch16_capstone/kernel/src/lb.rs
// The XDP load balancer: hash the flow tuple, pick a backend, redirect.
#![no_std]

use aya_bpf::{
    macros::{map, xdp},
    maps::Array,
    programs::XdpContext,
};
use aya_bpf::bindings::xdp_action::*;

const MAX_BACKENDS: u32 = 8;
const ETH_IP: usize = 14 + 20;
const HASH_SEED: u32 = 0x9e37_79b9; // a large odd constant (Knuth's 2^32/golden ratio)

#[repr(C)]
pub struct Backend {
    pub ifindex: u32,
    pub flags: u32,
}
#[map]
static BACKENDS: Array<Backend> = Array::with_max_entries(MAX_BACKENDS, 0);

// Multiply-xor hash of the 4-tuple: enough spread for backend selection,
// no loops - the verifier needs no proof of termination (Ch.16.2).
fn flow_hash(src: u32, dst: u32, sport: u16, dport: u16) -> u32 {
    let mut h = src ^ dst;
    h = h.wrapping_mul(HASH_SEED);
    h ^= (u32::from(sport) << 16) | u32::from(dport);
    h.wrapping_mul(HASH_SEED)
}

// The 4-tuple, parsed with the bounds-checked slice idiom of Ch.5.
fn tuple(frame: &[u8]) -> Option<(u32, u32, u16, u16)> {
    // ethertype at offset 12: only IPv4 (0x0800) has a fixed TCP offset.
    if frame.get(12..14)? != b"" {
        return None;
    }
    let s = frame.get(14 + 12..14 + 16)?;
    let d = frame.get(14 + 16..14 + 20)?;
    let sp = frame.get(ETH_IP..ETH_IP + 2)?;
    let dp = frame.get(ETH_IP + 2..ETH_IP + 4)?;
    Some((
        u32::from_be_bytes([s[0], s[1], s[2], s[3]]),
        u32::from_be_bytes([d[0], d[1], d[2], d[3]]),
        u16::from_be_bytes([sp[0], sp[1]]),
        u16::from_be_bytes([dp[0], dp[1]]),
    ))
}

#[xdp]
pub fn lb(ctx: XdpContext) -> u32 {
    let frame = unsafe {
        core::slice::from_raw_parts(
            ctx.data() as *const u8,
            ctx.data_end() - ctx.data(),
        )
    };

    let Some((src, dst, sport, dport)) = tuple(frame) else {
        return XDP_PASS; // not IPv4 or truncated: fail open
    };

    // Array lookup cannot fail (Ch.4); an empty slot passes.
    let idx = (flow_hash(src, dst, sport, dport) % MAX_BACKENDS) as usize;
    let backend = unsafe { BACKENDS.get_ptr(idx) };
    match unsafe { backend.as_ref() } {
        Some(be) if be.ifindex != 0 => aya_bpf::helpers::bpf_redirect(be.ifindex, 0),
        _ => XDP_PASS,
    }
}
}

The sockmap program is Chapter 8’s SK_MSG verdict verbatim (the mesh half), and the flow program is Chapter 7’s tracepoint plus Chapter 4’s ring. Together they are the three faces of the book’s data path: the packet face (XDP), the socket face (sockmap), and the observability face (tracepoint + ring) - all three sharing the one common contract.

16.3 The Control Plane: Load, Attach, Fill, Drain

The userspace side is Chapter 3’s lifecycle, run for three programs, with the control-plane duties of Chapters 4-5:

Source: code/ch16_capstone/userspace/src/main.rs

// code/ch16_capstone/userspace/src/main.rs
// The control plane: load, attach all three programs, fill maps, drain
// the flow ring. Chapter 3's lifecycle, run three times (Ch.16.3).
mod observer;

use aya::{
    maps::{Array, RingBuf, SockHash},
    programs::{SockMap, TracePoint, Xdp},
    Ebpf,
};
use common::{Backend, FlowEvent};
use std::error::Error;
use std::time::Duration;

const KERNEL_ELF: &[u8] =
    include_bytes!("../../kernel/target/bpfel-unknown-none/release/kernel");
const DEFAULT_IFACE: &str = "eth0";
const BACKEND_IFINDEX: u32 = 2;
const POLL_INTERVAL: Duration = Duration::from_millis(100);

fn main() -> Result<(), Box<dyn Error>> {
    let iface = std::env::args().nth(1).unwrap_or_else(|| DEFAULT_IFACE.to_string());
    let mut ebpf = Ebpf::load(KERNEL_ELF)?;

    // --- Chapter 5: the XDP load balancer ---
    let lb: &mut Xdp = ebpf.program_mut("lb")?.try_into()?;
    lb.load()?;
    lb.attach(&iface, aya::programs::XdpFlags::default())?;

    // --- Chapter 8: the sockmap mesh ---
    let mesh: &mut SockMap = ebpf.program_mut("msg_redirect")?.try_into()?;
    mesh.load()?;
    mesh.attach(&aya::programs::SockMapAttachType::MsgVerdict)?;

    // --- Chapter 7: the flow tracepoint ---
    let flow: &mut TracePoint = ebpf.program_mut("sched_process_exec")?.try_into()?;
    flow.load()?;
    flow.attach("sched", "sched_process_exec")?;

    // --- Chapter 4: fill the maps (the control-plane half) ---
    let mut backends: Array<_, Backend> = ebpf.map_mut("BACKENDS")?.try_into()?;
    backends.set(0, Backend { ifindex: BACKEND_IFINDEX, flags: 0 }, 0)?;
    println!("lb: backend[0] = ifindex {BACKEND_IFINDEX}");

    let _conns: SockHash<_, u32> = ebpf.map_mut("CONNS")?.try_into()?;
    // (In production the mesh accepts a connection, dials the backend,
    // and inserts both sockets - Chapter 8's control-plane pattern.)

    // --- Chapter 4/12: drain the flow ring like a one-node Hubble ---
    let mut flows: RingBuf = ebpf.map_mut("FLOWS")?.try_into()?;
    println!("capstone running: lb + mesh + flow observer on {iface}; Ctrl-C to exit");

    loop {
        std::thread::sleep(POLL_INTERVAL);
        for item in flows.iterator() {
            let ev: FlowEvent = item.read()?;
            observer::log(&ev);
        }
    }
}

Every line maps to a chapter; that is the capstone’s design goal. The control plane is deliberately unoptimised (a 100 ms poll) because the data path is the point - and the separation between the two is the book’s central discipline, now visible in one file.

16.4 Running It: kind, Cilium, and the Two Viewpoints

The capstone runs against the cluster you built in Chapter 11.5. The procedure, with the verification each step demands:

# 1. Build the kernel side and the userspace side.
cd code/ch16_capstone/kernel && cargo build --release --target bpfel-unknown-none
cd ../userspace && cargo build --release

# 2. The cluster: kind + Cilium (from Chapter 11), so the mesh has
#    pods and policies to route between.
kind create cluster --name ebpf-book
helm install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true

# 3. Run the capstone on the node. (In production this is a daemonset
#    with the loader's capabilities - Chapter 15.)
sudo ./target/release/userspace eth0

# 4. Generate traffic between two pods; watch BOTH viewpoints.
kubectl run client --image=nicolaka/netshoot -- sleep 3600
kubectl exec client -- curl -s http://api:443/healthz

# Viewpoint A - the kernel (Chapter 14): the XDP program, live.
bpftool prog list | grep xdp
bpftool map dump name BACKENDS

# Viewpoint B - the ring: the flow observer's stream, plus Hubble's.
# (the capstone's stdout shows FlowEvent lines; hubble shows the same
#  flows with pod names - Chapter 12's enrichment.)
hubble observe --from-pod client

The verification loop is the book’s loop: the same traffic visible from the kernel (bpftool) and from the ring (the observer), with Hubble as the reference implementation. When the three agree, the capstone is correct; when they disagree, you have a debugging session straight out of Chapter 14.

16.5 Measuring It: The Chapter 13 Budget, Applied

The capstone ships with the benchmark of Chapter 13: generate load with pktgen (the kernel’s packet generator) or a curl loop, and measure the data path from both sides - perf on the XDP program, the ring’s event rate as the observer’s throughput, and the bpftool prog show run-time counters. The honest claims you will be able to make, each with its method attached:

• The XDP LB runs at Mpps scale on a single core (a hash, an array lookup, a redirect - the Chapter 13 cost table says so, the benchmark proves it).
• The sockmap mesh moves connections without userspace round trips - the proxy-vs-sockmap latency comparison of Chapter 8, measured with the same load.
• The flow ring ships events at the tracepoint rate with bounded CPU - the observability cost of Chapter 13’s table, amortised by batching.

16.6 Where the Capstone Goes From Here

The capstone is complete, not finished - the epilogue’s list, applied:

1. Make the LB consistent-hashing (Maglev-style): replace the modulo with a backend-selection map so removing a backend does not reshuffle every flow.
2. Add the cgroup hook (Chapter 9): attach the policy program to the client pod’s cgroup so the mesh’s allow/deny decisions are per-pod.
3. Wire the observer to Hubble: replace the 100 ms poll with Cilium’s ring-reading pattern and push enriched flows to the Hubble API
   • the capstone becomes a plugin, not a demo.
4. Ship it: the daemonset, the capability model of Chapter 15, the CI of Chapter 14 (load test on a real kernel), the benchmark gate of Chapter 13.

Then, the book’s final instruction - the one every chapter was practicing: “Let me show you the code and the cluster.”

Hands-On Lab

# 1. Build everything.
cd code/ch16_capstone/kernel && cargo build --release --target bpfel-unknown-none
cd ../userspace && cargo build --release

# 2. The cluster (from Chapter 11).
kind create cluster --name ebpf-book
helm install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true

# 3. Run the capstone and generate traffic between two pods.
sudo ./target/release/userspace eth0 &
kubectl run client --image=nicolaka/netshoot -- sleep 3600
kubectl exec client -- curl -s http://api:443/healthz

# 4. The two viewpoints, one truth.
sudo bpftool prog list | grep -E 'xdp|sock|tracepoint'   # kernel side
# (the userspace stdout shows FlowEvent lines; hubble shows the same
#  flows enriched with pod names)
hubble observe --from-pod client

When the kernel viewpoint, the ring, and Hubble agree, the capstone is correct - and you have built a one-node Cilium (Chapters 11-12, assembled from Chapters 3-9).

Summary

• The capstone is one workspace, three kernel programs, one control plane: XDP LB (Ch. 5), sockmap mesh (Ch. 8), flow tracepoint (Ch. 7), all sharing one #[repr(C)] contract (Ch. 3).
• The common crate is the contract: one source of truth for the structs both sides read, with the compile-time layout check of Ch. 14.
• The control plane is Chapter 3’s lifecycle run three times: load, attach, fill maps, drain rings - the book’s two-plane discipline in one file.
• Run it against kind + Cilium, verify from two viewpoints - bpftool (the kernel) and the flow ring (userspace) - with Hubble as the reference.
• Measure with Chapter 13’s method: kernel, NIC, method, per-packet cost printed with every claim.
• The extensions are the book’s remaining chapters: consistent hashing, cgroup policy, Hubble integration, and the production ship of Ch. 14-15.