Chapter 1: The Linux Networking Stack from First Principles

“A packet is born in the NIC and dies in your socket. Everything in between - every ring, every queue, every copy, every syscall - is a decision someone made about speed versus complexity.”

Every eBPF program in this book does one of two things: it observes the packet path or it changes it. You cannot do either well without a precise model of that path. This chapter builds the model from the wire to the socket: the NIC’s DMA rings, the interrupt-versus-poll trade, the softirq where the stack lives, the protocol dispatch, and the socket receive queue where your recvfrom finds its data. Along the way we mark the exact points where eBPF can attach - because every hook in this book is a point on this path, and the cost of a hook is the cost of the path it sits on.

1.1 The NIC and the Ring

A modern NIC does not push packets at the CPU. It pulls - or rather, the driver and the NIC cooperate on a shared structure called a ring buffer (also known as a descriptor ring, or in the modern API, a page pool). The driver allocates a region of memory, fills it with empty packet buffers (pages), and tells the NIC, “here is where you may write incoming frames”. The NIC DMAs a frame into the next free buffer and advances its head pointer. The driver’s job is to keep ahead of the NIC: reclaim consumed buffers and replenish empty ones before the ring underruns.

Two facts follow immediately, and both shape everything else in this book:

1. There is no syscall on the receive path. The bytes travel from the wire into kernel memory by DMA. The kernel does not ask for them; the NIC deposits them. This is why eBPF can run at line rate: the packets are already in memory the kernel owns.
2. The ring is a producer-consumer queue in memory. The same mental model - head, tail, slots, one side writes, one side reads - appears again as the BPF ring buffer (Chapter 4) and as sockmap’s socket queues (Chapter 8). Learning to think in rings is the single most transferable skill in this book.

1.2 Interrupts vs. NAPI: Why the Kernel Polls

When a frame arrives, the NIC raises a hardware interrupt to get the CPU’s attention. Interrupts are how the NIC says “work is available”; they are also expensive - each one costs thousands of cycles of register save, pipeline drain, and return. Under heavy load, interrupt storms used to melt the machine: the CPU spent all its time servicing interrupts and none processing packets (the classic receive livelock).

The fix is NAPI (New API), the polled receive path introduced in Linux 2.4 and universal since. The driver first receives packets in interrupt mode, but on the first interrupt it does two things: it disables further interrupts and it schedules a softirq poll on a per-CPU napi_struct. The poll loop then drains the ring in batches - dozens or hundreds of frames per poll - and only re-enables interrupts when the ring is empty. The result: at low load, interrupts are rare and cheap; at high load, the CPU polls in efficient batches and the interrupt rate drops to near zero.

This is the first cost you must internalise: the CPU that runs the poll loop is the CPU that runs the whole stack for that queue. Which CPU is determined by RSS (Receive Side Scaling): the NIC hashes each frame’s headers and steers it to one of several queues, each with its own IRQ and its own CPU. Hash the (src ip, dst ip, src port, dst port) tuple and all packets of one flow land on the same CPU - which preserves ordering and spreads flows across cores. Chapter 7 returns to RSS with the details that matter for eBPF.

1.3 softirq: Where the Stack Actually Runs

The poll loop runs in softirq context - a deferred interrupt context that preempts whatever the CPU was doing. The stack, from netif_receive_skb onward, runs inside this softirq, not in the context of any process. Three consequences matter for everything we build:

• The receive path is not your process. When a packet arrives for your socket, the kernel does the protocol work right then, in softirq context, on the RSS-assigned CPU - not when you happen to call recv.
• Latency is budget. Every step between the ring and your socket costs cycles inside a context you do not control. The kernel’s engineering objective is to make those steps cheap: merge packets (GRO), delay allocation, batch work.
• eBPF hooks in the path inherit that budget. An XDP program (Chapter 5) runs in the driver’s poll context before the skb is even allocated. A tc ingress program runs after GRO, on the skb. Both have the same implicit deadline: finish before the next poll batch, or you become the bottleneck.

1.4 GRO and the Cost of Bigger Packets

The stack’s most effective optimisation is not cleverer parsing; it is fewer, bigger packets. GRO (Generic Receive Offload) merges consecutive TCP segments that arrive back-to-back for the same flow into one larger skb before the stack processes them. Where the NIC would hand you ten 1448-byte segments, the stack hands the TCP layer one ~14 KB super-packet (up to the MTU-limited gro_max_size). One skb means one round of header parsing, one trip through netfilter and the protocol layers, one entry into the socket queue instead of ten. The bytes are identical; the work per byte collapses.

GRO exists because of its transmit twin, GSO (Generic Segmentation Offload): the kernel hands the NIC one large skb, and the NIC segments it into wire-sized frames. GRO reverses that at receive. Whenever you read a Cilium or kernel BPF program that seems to assume “one skb per TCP segment”, remember GRO: by the time tc sees the packet, it may represent a run of segments. Chapter 7 covers the precise implications.

1.5 Protocol Dispatch and the Path to the Socket

After GRO, netif_receive_skb delivers the skb to protocol handlers: ptype_all (for taps like tcpdump and AF_PACKET sockets) and ptype_base (the IP, ARP, and other dispatch tables keyed by ethertype). The IP layer runs, netfilter’s hooks fire (PREROUTING, conntrack, FORWARD/INPUT), and for a locally destined packet the transport layer takes over. TCP validates the segment, updates the receive window, and queues the data into the socket’s receive buffer - an sk_buff_head protected by the socket lock - then wakes a blocked reader if one exists.

The receive queue is where the kernel’s job ends and your program’s begins:

recvfrom(fd, buf, n, 0);   // syscall
  -> copy data from socket receive queue into buf
  -> release the skb back to the page pool

That copy - from kernel buffer to userspace buffer - is the last copy in the receive path, and for most workloads it is the only one that touches your program. This is the default kernel path: interrupt, DMA, poll, GRO, stack, queue, syscall, copy. Roughly 2-10 us end-to-end with jitter, and a full syscall round trip per recvfrom. Chapter 12 of the C++ companion book (“Low-Latency I/O”) showed how mmap, io_uring, and kernel bypass shave this path; this book’s eBPF chapters show how to inhabit it.

1.6 Where eBPF Hooks In: The Map of This Book

The path you just built has a hook at almost every stage, and each hook is a chapter:

The ordering principle: the earlier the hook, the fewer kernel allocations and copies it has already paid for, and the fewer protocol guarantees it can rely on. XDP sees a raw Ethernet frame with no skb, no routing decision, no connection tracking - and can drop it in ~100 ns. A socket filter sees the fully validated, queued data of an established connection - and cannot avoid the stack’s cost, because the stack already ran.

1.7 The Cost Model You Will Quote

For the rest of the book, these are the numbers to reason with (details in Appendix B):

• Per-packet budget: at R Mpps on a 4 GHz core, you have 4000 / R cycles per packet. At 1 Mpps you have 4000 cycles - plenty for a hash lookup and a redirect. At 10 Mpps you have 400 - enough for a well-shaped XDP program, nothing more. Every eBPF design question in this book is, at bottom, a question about this budget.
• The two taxes: copies and syscalls. The default path pays both (one memcpy, one syscall per recv). eBPF’s value proposition is that it can run before either tax applies - or eliminate them entirely (sockmap, Chapter 8).
• The hook placement rule: a program attached later in the path sees more (validated, merged, routed) data and costs more to skip. Choose the hook that sees just enough to make your decision.

Source: code/ch01_packet_path/parse_frame.rs

// code/ch01_packet_path/parse_frame.rs
// Chapter 1 demo - walk an Ethernet frame with bounds-checked slice
// helpers. Portable: compiles and runs anywhere (no kernel, no BPF).
// Build: rustc parse_frame.rs && ./parse_frame
use std::net::Ipv4Addr;

const ETH_HDR_LEN: usize = 14;  // dst MAC 6 + src MAC 6 + ethertype 2
const IP_HDR_LEN: usize = 20;   // IPv4 without options
const ETH_P_IP: u16 = 0x0800;   // IPv4 ethertype (big-endian on the wire)
const PROTO_UDP: u8 = 17;

// The same bounds-checked readers the kernel-side programs use (net.rs).
fn u16_be(f: &[u8], off: usize) -> Option<u16> {
    let s = f.get(off..off + 2)?;
    Some(u16::from_be_bytes([s[0], s[1]]))
}
fn u32_be(f: &[u8], off: usize) -> Option<u32> {
    let s = f.get(off..off + 4)?;
    Some(u32::from_be_bytes([s[0], s[1], s[2], s[3]]))
}

/// Parse an IPv4 frame into (src, dst, protocol), or None if it is not
/// an IPv4 frame or is truncated. Pure and total - never panics.
fn parse_ipv4(frame: &[u8]) -> Option<(Ipv4Addr, Ipv4Addr, u8)> {
    // ethertype at offset 12; anything that is not IPv4 is not ours.
    if u16_be(frame, 12)? != ETH_P_IP {
        return None;
    }
    let proto = frame.get(ETH_HDR_LEN + 9)?;
    let src = Ipv4Addr::from(u32_be(frame, ETH_HDR_LEN + 12)?);
    let dst = Ipv4Addr::from(u32_be(frame, ETH_HDR_LEN + 16)?);
    Some((src, dst, *proto))
}

fn main() {
    // Build a synthetic 60-byte frame: eth header + IPv4 header + payload.
    let mut frame = vec![0u8; ETH_HDR_LEN + IP_HDR_LEN + 20];
    frame[12..14].copy_from_slice(&ETH_P_IP.to_be_bytes());
    frame[ETH_HDR_LEN + 0] = 0x45;                  // IPv4, IHL = 5 words
    frame[ETH_HDR_LEN + 8] = 64;                    // TTL
    frame[ETH_HDR_LEN + 9] = PROTO_UDP;
    frame[ETH_HDR_LEN + 12..ETH_HDR_LEN + 16].copy_from_slice(&[192, 168, 1, 10]);
    frame[ETH_HDR_LEN + 16..ETH_HDR_LEN + 20].copy_from_slice(&[10, 0, 0, 1]);

    match parse_ipv4(&frame) {
        Some((src, dst, proto)) => {
            println!("ethertype = 0x{ETH_P_IP:04x} (IPv4)");
            println!("protocol = {proto}, src = {src}, dst = {dst}");
            println!("packet parse complete; no syscall, no copy, just a header walk");
        }
        None => println!("not an IPv4 frame (or truncated)"),
    }
}

The demo is deliberately boring: it is a header walk with explicit bounds checks, which is exactly the shape of a verifier-accepted XDP program, minus the kernel. If you can explain why every assert! here is mandatory, you already understand the single most important constraint in this book: eBPF programs never trust the packet; they check every access against data_end.

Hands-On Lab

# 1. Build and run the portable header-walk demo.
rustc code/ch01_packet_path/parse_frame.rs && ./parse_frame

# 2. See the real thing: capture a frame and compare the offsets you
#    parsed with what the wire actually carries.
sudo tcpdump -i any -c 1 -nn -XX    # the -XX shows raw bytes, hex + ascii

Walk the hexdump: bytes 0-5 dst MAC, 6-11 src MAC, 12-13 ethertype (08 00 = IPv4), 14-33 IPv4 header, and find the protocol byte at 14+9. If you can point at each field, you already know how an XDP program parses (Chapter 5 does the same walk in the kernel).

Summary

• The receive path is DMA into a ring, then a poll: the NIC writes frames into kernel memory; NAPI disables interrupts under load and polls the ring in batches inside softirq context.
• RSS decides which CPU owns a queue by hashing the flow tuple; the CPU that polls is the CPU that runs the whole stack for that flow.
• GRO merges back-to-back TCP segments into one skb; fewer, bigger packets mean fewer rounds of parsing and queueing.
• The path ends with a socket receive queue: the kernel queues data, then recvfrom copies it out. One syscall, one copy - the two taxes.
• eBPF hooks exist at almost every stage: earlier hooks (XDP) see raw frames before skb allocation; later hooks (socket, sockmap, cgroup) see validated connection state. Earlier is cheaper and less informed.
• The whole book is a budget question: 4000 / R cycles per packet at R Mpps on a 4 GHz core. Design programs inside that budget.

Next: Chapter 2 opens the box - the eBPF virtual machine, the verifier that makes in-kernel programs safe, and the JIT that makes them fast.