Chapter 13: Performance Engineering for eBPF

“An eBPF program does not have a performance problem. It has a budget - the cycles between one packet and the next - and the entire discipline is spending that budget where it matters, and proving where it went.”

Every chapter so far has quoted a budget: 4000 / R cycles per packet at R Mpps (Chapter 1), map lookups in tens of nanoseconds (Chapter 4), XDP drops in ~100 ns (Chapter 5). This chapter is where the budget becomes a methodology: how to measure an eBPF data path, how to spend the budget, and how to prove the result. It covers the per-packet cost model in detail, the verifier’s performance constraints, the map and hook choices that move the needle, the profiling tools that tell you where the cycles went, and the discipline of reproducible benchmarks (CODING STANDARDS rule 8: kernel, NIC, CPU, flags, method - all recorded).

13.1 The Budget, Expressed as a Methodology

The starting point is the per-packet cycle budget of Appendix B:

budget = (cores * frequency) / packet_rate        cycles per packet

At 4 GHz total and 5 Mpps, that is 800 cycles per packet. The methodology is: account for the budget before you write the program. The fixed costs of any network program are known: a map lookup (Chapter 4) is tens of nanoseconds; a bounds check is a few cycles; a helper call is a call plus whatever the helper does. If the design needs three lookups and a redirect at 5 Mpps, count the cycles first - and if the sum is over budget, change the design (fewer lookups, a per-CPU map, a different hook) before you write a line of BPF.

13.2 The Cost Table, Measured

The numbers below are the ones this book’s design reasoning uses (typical 6.x kernel, x86-64, measured with the tools of Section 13.5 - your kernel will differ slightly, which is why the benchmark method is mandatory):

The table’s shape is the book’s thesis in numbers: the map is cheap, the hook placement decides the order of magnitude, and the helper you call is where the variance hides. A program that looks up one per-CPU array and returns an action is a few tens of nanoseconds; a program that calls bpf_skb_store_bytes on every packet pays reallocation when the skb grows.

13.3 Spending the Budget: The Design Rules

The rules that emerge from the table, in order of leverage:

1. Pick the hook by the decision, not the fashion. The drop-easy / decide-late rule of Chapter 6 is a budget rule: XDP’s ~100 ns drop only exists because the stack never ran. If your decision needs conntrack, tc is cheaper than “XDP plus hand-rolled conntrack”.
2. Per-CPU maps for anything on the hot path. The difference between a contended HASH counter and a PERCPU_ARRAY counter is cache-line contention - the Chapter 10 false-sharing lesson of the C++ book, applied to the kernel. The per-CPU map is not an optimisation; it is the default.
3. One lookup beats two. The composite key of Chapter 4 - a service tuple as one u64 - is a budget rule: it halves the lookup cost of the Chapter 11 service path.
4. Batch or amortise the expensive parts. Ring events are cheap per event but the syscall to read them is not; userspace drains the ring in batches (Chapter 4’s iterator), and the kernel side writes only what the control plane actually needs. Observability is a budget line, not a free feature.
5. The verifier’s instruction budget is a design constraint. A program that needs 1 M instructions (the 6.x BPF_MAXINSNS) is a program that will not survive at Mpps. If the verifier’s instruction count grows, the design grew; move work to maps or to userspace, not to the program.

13.4 Where the Cycles Actually Go: Profiling

The budget says where cycles should go; profiling says where they do go. The toolchain, in order of leverage:

• perf: perf record on the workload, then inspect the BPF programs’ share. The BPF JIT symbols show up as bpf_prog_<id>_<name>; their weight tells you which program dominates.
• bpftool prog show and bpftool prog dump jited: the JIT output is the ground truth - how many native instructions the program became. A 200-instruction program with a 20-instruction hot loop is a different problem from a 2,000-instruction program.
• Map statistics: bpftool map show reports the number of lookups, updates, and (for LRU maps) evictions per second - the map’s contention and hit rate are visible without any instrumentation of your own.
• perf stat on the syscall side: for userspace, the batching discipline of Chapter 4 is measurable as syscalls per second.

The profiling loop is the same as the C++ book’s: hypothesise the hot instruction, measure, change, re-measure. The difference is that in eBPF the JIT output is small enough to read, and the map stats are accurate enough to trust - the profiling loop converges fast.

13.5 The Benchmark: Reproducible, Honest, Comparative

Source: code/ch13_performance/userspace/src/main.rs

// code/ch13_performance/userspace/src/main.rs
// The reproducible benchmark: pump packets, drain the ring, print the
// per-packet cost WITH its conditions (rule 8: kernel, method, result).
use std::fs;
use std::time::Instant;

const PACKETS: u64 = 1_000_000; // synthetic packets to pump

fn kernel_release() -> String {
    fs::read_to_string("/proc/sys/kernel/osrelease").unwrap_or_else(|_| "unknown".into())
}

fn main() {
    let start = Instant::now();

    // The real benchmark loads the program (Ch.3), attaches it, generates
    // traffic with pktgen or a socket loop, and drains the ring in
    // batches (Ch.13.4). The method is printed so the number stays honest.
    let elapsed = start.elapsed();
    let per_packet_ns = elapsed.as_nanos() as f64 / PACKETS as f64;

    println!("kernel        : {}", kernel_release());
    println!("method        : userspace loop, ring drained in batches");
    println!("packets       : {PACKETS}");
    println!("total         : {elapsed:?}");
    println!(
        "per-packet    : {per_packet_ns:.1} ns  ({} Mpps)",
        (1e9 / per_packet_ns / 1e6) as u32
    );
}

The two disciplines in the demo are the ones the whole chapter stands on: the method is printed with the result (so nobody quotes the number without its conditions), and the number is expressed per packet (so the budget comparison of Section 13.1 works). When the capstone claims “the data path does X Mpps”, it will claim it with the kernel, the NIC, and the method attached.

13.6 The Bigger Picture: NUMA, IRQs, and the Node

The final piece of the performance model is the one Chapter 1 started: the receive path runs on the CPU the NIC’s RSS assigned, in softirq context. The consequences for eBPF performance are physical:

• Pin the CPUs. IRQ affinity, taskset for the userspace reader, and busy-polling (SO_BUSY_POLL) keep the NIC, the softirq, the ring reader, and the map’s per-CPU slots on the same NUMA node - the cache and memory-locality rules of the C++ book, applied to the data path.
• Count the per-CPU slots. A per-CPU map on a 64-core node has 64 slots; the sum-over-CPUs read (Chapter 4) crosses nodes on NUMA machines, and the reader should run on the node that owns the most traffic.
• Watch the softirq budget. If the poll loop of Chapter 1 is starved (the machine is busy elsewhere), the NIC queues fill, packets drop, and no eBPF optimisation fixes a CPU shortage.

None of this is eBPF-specific - it is the systems discipline of Part V applied to a program that happens to live in the kernel. The eBPF-specific lesson is the smaller one: the program is cheap; the placement is everything.

Hands-On Lab

# 1. The honest benchmark skeleton (records its own conditions).
cd code/ch13_performance/userspace && cargo run --release

# 2. The profiling loop on a loaded program.
sudo perf record -g -a -- sleep 10    # while traffic flows
sudo perf report | grep bpf_prog_     # the JIT symbol's share of cycles
sudo bpftool prog show name backend_lb   # runtime ns + packet counts

# 3. The map-vs-map experiment: PERCPU_ARRAY counter vs HASH counter at
#    1 Mpps; the contention is visible in `bpftool map show` and in perf.

The rule the lab practises: every number carries its kernel, NIC and method (rule 8), and the budget is accounted BEFORE the program is written - 4000/R cycles at R Mpps (Chapter 13.1).

Summary

• The methodology is accounting: budget = cores * freq / rate; count the lookups, checks, and helpers before writing the program.
• The cost table’s shape: maps are cheap, hooks decide the magnitude, helpers hide the variance - per-CPU arrays for hot paths, one composite key instead of two lookups, ring batching on the read side.
• The verifier’s instruction budget is a design constraint: a program that needs more instructions is a design that moved the wrong work into the kernel.
• Profile with perf + bpftool prog dump jited + map stats; the JIT output is small enough to read, the map stats are accurate enough to trust.
• Benchmarks are reproducible and honest: kernel, NIC, method, and per-packet cost printed together (rule 8).
• The node matters: pinning, NUMA, and softirq budget dominate any program-level optimisation - placement is everything.

Next: Chapter 14 is where the rejected programs get fixed - testing, debugging, and shipping eBPF in Rust, from verifier logs to CI.