Chapter 14: Testing, Debugging & Shipping eBPF in Rust

“Your eBPF program will be rejected. The question is whether the rejection is a compiler error or an incident. The difference is the discipline of this chapter: test the program before the kernel does, read the verifier like a compiler, and ship the diagnosis with the code.”

The hardest part of eBPF development is not writing programs; it is the moment the verifier disagrees with you - and the moments after, when a program that loaded silently does the wrong thing. This chapter is the complete debugging and testing playbook: reading verifier logs, using bpftool and bpftrace to inspect a running system, writing tests that catch rejections and logic bugs before they reach the kernel, testing the userspace control plane, and the CI setup that makes all of it run on every push. By the end, the failure modes of every earlier chapter - bounds violations, map contract mismatches, attach failures, silent drops - have a name and a recipe.

14.1 Reading the Verifier Like a Compiler

The verifier rejection of Chapter 2 deserves a full decode. The canonical log line:

12: (71) r1 = *(u8 *)(r2 + 20)     ; R2=pkt(off=14,r=14,imm=0)
invalid access to packet, off=20 size=1, R2 has only 14 bytes of readable data

The vocabulary: r2 is a packet pointer whose known-readable region is 14 bytes (the Ethernet header, r=14); the program tried to read byte 20. The verifier’s log is positional - it names the instruction, the register, and the property it could not prove. The debugging loop:

1. Read the register states, not just the message. R2=pkt(off=14,r=14) tells you what the verifier knows - usually “the bounds check you wrote covered 14 bytes, and then you read past it”.
2. Find the missing check. The fix is almost always a bounds comparison before the access: if data + 34 > data_end { return PASS }.
3. Check the obvious suspects first: helper calls that invalidate packet pointers (Chapter 5), structs without #[repr(C)], usize in a shared struct (Chapter 3’s contract rules).

The discipline that prevents most rejections: write the check before the access, every time, and review the pair together. The code review rule for eBPF code is “every read of packet memory must be preceded, within five lines, by the bounds check that justifies it.”

14.2 bpftool: The Runtime Truth

bpftool is the swiss army knife of the BPF subsystem, and the debugging session for any mystery is a sequence of its commands:

# What is attached, where, and how big is it?
bpftool prog list                        # all loaded programs
bpftool prog show id 123                 # one program's details
bpftool prog dump jited id 123           # the native code (Ch. 13)
bpftool prog dump xlated id 123          # the verified BPF instructions

# The maps: sizes, usage, and the live counters.
bpftool map list
bpftool map dump name ALLOW              # see the actual key/value entries

# The BTF type info that CO-RE relocations used (Ch. 2).
bpftool btf dump file /sys/kernel/btf/vmlinux | head

# Live tracing of every load/attach on the system.
bpftrace -e 'tracepoint:syscalls:sys_enter_bpf { printf("bpf() from %s\n", comm); }'

The key habit: when a program misbehaves, look at the maps first. A program that “does nothing” is usually a program whose map lookup misses - bpftool map dump shows you the keys that exist, and the miss becomes obvious (wrong key layout, wrong endianness, wrong composite key construction - the Chapter 4 contract bugs). The map is the program’s visible state, and it is fully inspectable while the program runs.

14.3 bpftrace: Confirm the Model Before You Debug It

bpftrace is the one-liner layer on top of eBPF, and it is the fastest way to confirm or refute a hypothesis about the kernel, not your program. Before debugging your tc classifier, check the premise with a one-liner:

# Hypothesis: retransmits spike when the backend pool shrinks.
bpftrace -e 'k:tcp_retransmit_skb { @[nsecs/1e6] = count(); }'

# Hypothesis: packets are dropped in the stack, not by my XDP program.
bpftrace -e 'tracepoint:skb:kfree_skb { @[args->reason] = count(); }'

The habit is the one the C++ book calls “verify the premise”: if your program is not doing what you expect, first establish what the kernel is actually doing - bpftrace answers that in one line, and the answer often redirects the whole investigation from your code to the model underneath.

14.4 Testing the Kernel Side: Rejection Tests and Logic Tests

The kernel side of an Aya project is hard to unit test directly (it needs the kernel), but the logic is testable in two ways:

Source: code/ch14_testing_debugging/kernel/src/main.rs

#![allow(unused)]
fn main() {
// code/ch14_testing_debugging/kernel/src/main.rs
// The kernel program: a thin shell over the host-testable parse module.
#![no_std]
#![no_main]

mod parse;

use aya_bpf::{macros::xdp, programs::XdpContext};
use aya_bpf::bindings::xdp_action::*;

#[xdp]
pub fn xdp_parse(ctx: XdpContext) -> u32 {
    let frame = unsafe { parse::frame(ctx.data(), ctx.data_end()) };

    // One pure function call: None means non-TCP or truncated - pass.
    match parse::dst_port(frame) {
        Some(port) if port == parse::SERVICE_PORT => XDP_DROP, // demo rule
        _ => XDP_PASS,
    }
}

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    unsafe { core::hint::unreachable_unchecked() }
}
}

The pure function itself lives in kernel/src/parse.rs - no panics, no BPF, no no_main attributes, just the bounds logic:

Source: code/ch14_testing_debugging/kernel/src/parse.rs

#![allow(unused)]
fn main() {
// code/ch14_testing_debugging/kernel/src/parse.rs
// Pure, total packet parsing: no unsafe, no kernel calls, no allocation.
// The program turns (data, data_end) into a slice ONCE, then all parsing
// is ordinary slice logic - bounds-checked, idiomatic, host-testable.
// The same module is exercised by `cargo test` on the host (Ch.14).
use core::mem::size_of;

// Build a slice over the DMA'd frame. The verifier accepts this because
// the length is derived from data_end - data (Ch.2's bounds discipline).
#[inline(always)]
pub unsafe fn frame<'a>(start: usize, end: usize) -> &'a [u8] {
    debug_assert!(end >= start, "data_end must not precede data");
    core::slice::from_raw_parts(start as *const u8, end - start)
}

#[inline(always)]
pub fn u8_at(f: &[u8], off: usize) -> Option<u8> {
    f.get(off).copied()
}

#[inline(always)]
pub fn u16_be(f: &[u8], off: usize) -> Option<u16> {
    let s = f.get(off..off + size_of::<u16>())?;
    Some(u16::from_be_bytes([s[0], s[1]]))
}

#[inline(always)]
pub fn u32_be(f: &[u8], off: usize) -> Option<u32> {
    let s = f.get(off..off + size_of::<u32>())?;
    Some(u32::from_be_bytes([s[0], s[1], s[2], s[3]]))
}

// ---- Chapter 14's testable core: the parse logic, pure and total --------

pub const ETH_IP: usize = 14 + 20;
pub const SERVICE_PORT: u16 = 8080;

/// Returns the dst port for an IPv4/TCP frame, None otherwise.
pub fn dst_port(frame: &[u8]) -> Option<u16> {
    if u16_be(frame, 12)? != 0x0800 || u8_at(frame, 14 + 9)? != 6 {
        return None;
    }
    u16_be(frame, ETH_IP + 2)
}
}

The testable design: the packet logic is a pure function over a slice, which is exactly the slice the program hands it. The same function can be compiled into the kernel program and into a native test binary (the #[cfg(test)] module in the same crate, or a tests/ integration test against the common crate). The rejection tests are then ordinary Rust:

Source: code/ch14_testing_debugging/kernel/tests/parse.rs

#![allow(unused)]
fn main() {
// code/ch14_testing_debugging/kernel/tests/parse.rs
// Host-run tests against the EXACT module the kernel program uses.
// No kernel, no BPF - the verifier's bounds discipline, checked by the
// Rust test harness first (Ch.14.4).
#[path = "../src/parse.rs"]
mod parse;

fn tcp_frame(dport: u16) -> Vec<u8> {
    let mut f = vec![0u8; parse::ETH_IP + 8];
    f[12..14].copy_from_slice(&0x0800u16.to_be_bytes()); // IPv4
    f[14 + 9] = 6;                                      // TCP
    f[parse::ETH_IP + 2..parse::ETH_IP + 4].copy_from_slice(&dport.to_be_bytes());
    f
}

#[test]
fn rejects_truncated_frames() {
    let short = tcp_frame(8080);
    assert_eq!(parse::dst_port(&short[..parse::ETH_IP + 1]), None);
}

#[test]
fn rejects_non_tcp() {
    let mut f = tcp_frame(8080);
    f[14 + 9] = 17; // UDP
    assert_eq!(parse::dst_port(&f), None);
}

#[test]
fn finds_tcp_port() {
    assert_eq!(parse::dst_port(&tcp_frame(8080)), Some(8080));
}
}

The pattern is the one every serious eBPF project uses: host-run tests for the parse logic, kernel-run tests for the load/attach, and the same function in both places. The rejection that would have taken an hour to read from a verifier log becomes a test failure that names the bug.

14.5 Testing the Userspace Side: The Control Plane Is a Program

The userspace side has no excuse - it is ordinary Rust, and it gets ordinary tests: the map contract (insert what the kernel expects, read what it wrote), the ring draining (batch reads, handle overflow), and the attach errors (wrong interface, missing privileges, kernel config). The tests that matter most are the contract tests: a common struct that crosses the boundary must produce exactly the byte layout both sides expect, and a #[test] that checks size_of and offsets catches the #[repr(C)] mistakes of Chapter 3 before they become map garbage.

Source: code/ch14_testing_debugging/userspace/tests/contract.rs

#![allow(unused)]
fn main() {
// code/ch14_testing_debugging/userspace/tests/contract.rs
// The contract test: the shared struct's layout is the agreement between
// kernel and userspace. If this fails, every map read is garbage (Ch.14.5).
#[test]
fn contract_layout_is_stable() {
    use core::mem::{align_of, size_of};

    // The ring event of Chapter 9: fixed size, C layout, no surprises.
    #[repr(C)]
    struct DenyEvent {
        dst_ip: u32,
        proto: u8,
        ts: u64,
    }
    assert_eq!(size_of::<DenyEvent>(), 16);
    assert_eq!(align_of::<DenyEvent>(), 8);
}
}

14.6 Shipping: CI, Privileges, and the Load Test

The CI setup for an eBPF repo is the point where the discipline becomes a pipeline:

1. Lint and build everything: cargo fmt --check, cargo clippy on the userspace and common crates, cargo build --target bpfel-unknown-none on the kernel crate - every push.
2. Run the host tests: the parse tests and contract tests of Sections 14.4-14.5 - no kernel needed, fast, and they catch 80% of bugs.
3. Run the kernel tests on a real kernel: a CI job on an ubuntu-latest runner (which has BTF enabled by default on modern versions) that loads the programs against a veth pair, attaches them, generates traffic, and asserts the events arrive. This is the test that the local machine (macOS, Windows, or a locked-down Linux) cannot run - which is why the book’s CONTRIBUTING.md asks contributors to state their platform, and why the workflow runs the kernel job regardless.
4. The load test as a gate: the benchmark of Chapter 13, run in CI, failing on a regression beyond a threshold - the honest, repeatable performance number, enforced.

The final shipping concern is privileges: loading eBPF needs CAP_BPF (+ CAP_PERFMON for tracing, CAP_SYS_ADMIN on older kernels), which is why production deployments run the loader in a privileged init container and the runtime (after attach) needs nothing - the maps are updated by the control plane, and the data path runs unattended. That separation is the security posture of Chapter 15, made operational.

Hands-On Lab

# 1. The tests that need no kernel: run them anywhere.
cd code/ch14_testing_debugging/kernel
cargo test -- --nocapture        # parse tests against the pure module

# 2. Break a bound on purpose and watch the test catch it.
#    (change `ETH_IP + 2` to `ETH_IP + 4` in parse.rs, re-run: the
#     truncation test fails - that is a verifier rejection, caught first)

# 3. The runtime truth, when the program is loaded.
sudo bpftool prog list
sudo bpftool map dump name BACKENDS
sudo bpftrace -e 'k:tcp_retransmit_skb { @ = count(); }'

The CI ladder from Section 14.6 in practice: fmt/clippy, host tests, kernel load tests on a real runner, benchmark gate - every push.

Summary

• Read the verifier log as a compiler error: the register states name the missing proof; the fix is the bounds check before the access.
• bpftool is the runtime truth: program list, JIT dump, and - most often - the map dump that shows the contract bug.
• bpftrace confirms the premise: establish what the kernel is doing before debugging what your program is not.
• The testable design: packet logic as a pure slice function, tested on the host; the same function compiled into the kernel program.
• The userspace side gets ordinary tests: the layout contract test catches the #[repr(C)] bugs before they reach a map.
• CI runs the ladder: fmt/clippy/build, host tests, kernel load tests, benchmark gate - and production separates privileged loader from unprivileged runtime.

Next: Chapter 15 closes Part V with the security model itself - what eBPF can do, what it cannot, and where the attack surface really is.