Chapter 7: TCP/IP, Sockets & the Receive Path

“TCP is not a protocol for moving bytes. It is a protocol for agreeing, byte by byte, on what has been received - and the receive path is where that agreement is enforced, at line rate, in softirq context.”

Chapters 5-6 put eBPF on the packet path. This chapter puts the protocol under a microscope: what the kernel actually does with a TCP segment from the moment GRO hands it over until recvfrom returns your bytes. This is the layer where sockets, conntrack, timers, retransmits, and backpressure live - and it is the layer eBPF observes through kprobes and tracepoints and modifies through sockmap (Chapter 8) and cgroup hooks (Chapter 9). A systems engineer who cannot trace a segment through this path is debugging blind.

7.1 The Socket: The Kernel’s File of the Network

A socket is a pair of queues - receive and send - wrapped in state, owned by a file descriptor, and created by the socket() syscall. The receive path of Chapter 1 ends here: the stack queues the incoming data into the socket’s receive buffer (sk_rcvbuf), and your recvfrom copies it out. The crucial fact for everything that follows: the socket is where the kernel’s protocol processing meets the process’s syscalls, and every eBPF program that touches sockets - sockmap, socket filters, cgroup hooks - is attached at this meeting point.

A TCP socket’s state machine matters less for this book than its queues: sk_receive_queue (data waiting for recv), sk_write_queue (data waiting for the network), and the out-of-order queue. The receive queue is an sk_buff_head, and the queue discipline is the lock discipline: the socket lock protects it, and the softirq that delivers the segment takes that lock. Contention on the socket lock - one flow per socket, one CPU per flow via RSS - is why TCP throughput is fundamentally single-queue per socket, and why sockmap’s redirection (Chapter 8) is such a big deal: it moves the socket-level decision into the kernel without touching the queue.

7.2 The Receive Path, From GRO to recvfrom

Follow one segment through the path of Chapter 1, now with protocol detail:

1. GRO (Chapter 1) merges back-to-back segments of one flow into a larger skb. The TCP layer usually sees runs, not segments.
2. Protocol dispatch routes the skb by ethertype to the IP layer; netfilter hooks fire (PREROUTING, conntrack, INPUT). Conntrack creates or updates the connection entry - the table Cilium replaces with BPF maps in Chapter 11.
3. TCP processing (tcp_v4_rcv) looks up the socket by the 4-tuple (src ip, src port, dst ip, dst port) in the per-netns listen/established hash tables. The segment is validated: sequence numbers, checksum, window. ACKs are handled immediately - the receive window slides, timers are updated - and then the data is queued.
4. The socket queue: the data lands in sk_receive_queue, and if a process is blocked in recvfrom, it is woken. The kernel copies the data out under the socket lock and releases the skb.
5. Your syscall: recvfrom returns; the bytes are in your buffer. The default path cost - from NIC to your buffer - is the 2-10 us of Chapter 1.

The two things eBPF observes here are function entry points (kprobes) and tracepoints - kernel-instrumented points with stable, documented argument blocks. The most useful for network debugging: tcp:tcp_rcv_space_adjust (receive buffer behaviour), tcp:tcp_retransmit_skb (the retransmit event, i.e. the network is losing), sock:inet_sock_set_state (every socket state transition - the backbone of Cilium’s Hubble flow logging), and skb:kfree_skb (every dropped packet, with the drop reason).

7.3 A Tracing Program: Watching Retransmits with Aya

Source: code/ch07_tcp_receive_path/kernel/src/main.rs

#![allow(unused)]
fn main() {
// code/ch07_tcp_receive_path/kernel/src/main.rs
// Observe the receive path: count TCP retransmits per destination, and
// ship one event per retransmit through the ring buffer.
#![no_std]
#![no_main]

use aya_bpf::{
    macros::{map, tracepoint},
    maps::{HashMap, RingBuf},
    programs::TracePointContext,
};

const MAX_DESTS: u32 = 4096;
// struct sock: skc_daddr sits at offset 32 in the common fields on the
// kernels this book targets; CO-RE resolves it in production code (Ch.2).
const SK_DADDR_OFF: usize = 32;

// Per-destination retransmit counter, keyed by dst ip (host byte order).
#[map]
static RETRANS: HashMap<u32, u32> = HashMap::with_max_entries(MAX_DESTS, 0);

#[repr(C)]
pub struct RetransEvent {
    pub dst_ip: u32,
    pub ts: u64,
}
#[map]
static EVENTS: RingBuf = RingBuf::with_max_entries(4096, 0);

// tcp:tcp_retransmit_skb fires on every retransmission, in softirq
// context, with a stable argument block: (sk, skb).
#[tracepoint]
pub fn tcp_retransmit_skb(ctx: TracePointContext) -> i32 {
    let Ok(dst_ip) = try_retransmit(&ctx) else {
        return 0; // truncated tracepoint block: nothing to record
    };

    // Count, then emit one event. Both map ops are best-effort.
    let mut n = RETRANS.get(&dst_ip).unwrap_or_default() + 1;
    let _ = RETRANS.insert(&dst_ip, &mut n, 0);

    let Ok(entry) = EVENTS.reserve::<RetransEvent>(0) else {
        return 0; // ring full: drop the event, keep counting
    };
    unsafe {
        entry.write(RetransEvent {
            dst_ip,
            ts: aya_bpf::helpers::bpf_ktime_get_ns(),
        });
    }
    entry.submit(0);
    0
}

fn try_retransmit(ctx: &TracePointContext) -> Result<u32, i32> {
    let sk: u64 = unsafe { ctx.read_at(0)? };
    // Reach the socket's destination field as an unaligned u32 read; the
    // offset is a kernel-version detail CO-RE resolves in production.
    let p = (sk as *const u8).add(SK_DADDR_OFF) as *const u32;
    Ok(u32::from_be(unsafe { p.read_unaligned() }))
}

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    unsafe { core::hint::unreachable_unchecked() }
}
}

This is the observability face of eBPF: the program does not change the path, it reports on it. The retransmit event is the signal that the network is losing packets - the single most important metric for any TCP-dependent system, and the reason Cilium’s Hubble and every serious network observability stack are built on exactly these tracepoints.

7.4 GRO, GSO and the Segment Illusion

Chapter 1 introduced GRO/GSO as a cost optimisation; here is the protocol consequence. Because GRO merges segments before TCP sees them, the TCP layer’s receive window accounting and ACK generation work on merged runs; because GSO re-segments on transmit, the wire sees standard MTU frames again. The practical rule for eBPF writers: an skb at tc ingress may represent several wire segments, so per-segment assumptions (one ACK per skb, one header per skb) are wrong. Read the lengths and count segments from the headers, never assume.

7.5 Backpressure: What Happens When You Are Slow

The receive path has a built-in negotiation between the stack and your process, and it is the last piece of the model. If your process does not read fast enough, the socket’s receive buffer fills; the stack stops accepting data for that socket (the window closes, the sender throttles) and eventually drops. This is not a failure; it is the protocol working. TCP is a flow-control protocol; the receive buffer is the flow-control mechanism. When you see recvfrom latency climbing, the question is never “why is the kernel slow” but “where is the queue filling” - and the answer is either your process’s read loop, the socket buffer size, or the NIC queue of Chapter 1. eBPF sees all three: the socket queues via sockmap (Chapter 8), the cgroup’s socket usage via cgroup hooks (Chapter 9), and the NIC queues via XDP/tc counters.

Hands-On Lab

# 1. Load the retransmit counter on the host.
cd code/ch07_tcp_receive_path/kernel && cargo build --release --target bpfel-unknown-none

# 2. Induce loss deliberately, then watch the events arrive.
sudo tc qdisc add dev eth0 root netem loss 10%   # drop 10% of packets
# (run a download through eth0 - the retransmits fire)
sudo bpftool map dump name RETRANS               # per-destination counts

# 3. Cross-check with bpftrace - the same tracepoint, one line.
sudo bpftrace -e 'k:tcp_retransmit_skb { @ = count(); }'
sudo tc qdisc del dev eth0 root netem            # clean up

The moment to notice: the tracepoint fires in softirq context, on the RSS-assigned CPU, for a connection that may belong to another process. That is why observability programs are attached from outside the process (Chapters 7, 15).

Summary

• A socket is two queues plus state, owned by an fd - the meeting point of kernel protocol processing and process syscalls.
• The receive path is GRO -> IP/netfilter/conntrack -> TCP lookup and queue -> socket lock -> syscall copy, and eBPF observes each stage through tracepoints (tcp_retransmit_skb, inet_sock_set_state, kfree_skb).
• kprobes/tracepoints are the observability hooks: they report, they do not modify - and they are what Hubble and every serious network observability stack are built on.
• GRO merges segments; GSO re-segments. An skb is not a segment; read lengths from headers, never assume.
• Backpressure is the protocol working: the receive buffer is the flow-control mechanism, and slow readers throttle the sender by design.

Next: Chapter 8 changes the path instead of reporting on it - sockmap and SK_MSG, eBPF’s in-kernel socket redirect, the mechanism behind Cilium’s service mesh.