Chapter 8: Sockmap & SK_MSG - eBPF in the Socket Layer

“A proxy is a program that moves bytes between sockets. sockmap is a program that moves bytes between sockets without leaving the kernel - no copy to userspace, no syscall, no proxy process at all.”

Chapter 7 ended at the socket. This chapter enters it. sockmap is a BPF map type whose values are sockets, and SK_MSG / SK_SKB are the program types attached to it: they run in the socket layer, on data being sent to or received from a socket in the map, and they can redirect that data to another socket in the map - in the kernel, without a round trip through userspace. This is the mechanism behind Cilium’s socket-level load balancing and its service mesh data path: when a pod talks to a service, the kernel itself rewrites the destination socket, and no proxy process ever touches the bytes. This chapter is the complete practical treatment: the map, the program types, the redirect helpers, and the semantics that make it correct.

8.1 The Problem: Why Proxies Are Slow

A classic service proxy (Envoy, HAProxy, an L4 load balancer) does the following per connection: accept on the front socket, read into a userspace buffer, write to the back socket. Every byte crosses the kernel/userspace boundary twice per direction - two read/write pairs, four context switches in the worst case, two copies - and the proxy process consumes a core per some thousands of connections. The kernel, by contrast, already has both sockets. The question sockmap answers is: why should the bytes visit userspace at all when the kernel can move them directly?

8.2 The sockmap: A Map Whose Values Are Sockets

A sockmap (or its hash twin, sockhash) is a BPF map keyed by anything you like - typically the service tuple or a connection id - whose values are sockets. The socket is inserted from userspace by passing the file descriptor; the kernel stores the real socket object. Two things make this special:

1. Programs attach to the map, not to an interface. A SK_MSG program is attached to a sockmap with a direction (BPF_SK_MSG_VERDICT), and it runs on every send on a socket in that map. A SK_SKB program runs on received data instead.
2. Redirect is map-mediated. The program calls bpf_msg_redirect_map (SK_MSG) or bpf_sk_redirect_map (SK_SKB) with a key; the kernel looks up the target socket in the same map and delivers the data to it - queuing it on the target’s receive queue, with the flow-control, ordering, and wake-up semantics of a normal receive.

The mental model: the sockmap is a routing table for sockets. The program is the routing logic. The data plane of a sockmap proxy is entirely inside the kernel.

8.3 SK_MSG: The Send-Side Program

The SK_MSG program runs in the send path of a socket in the map, before the data is handed to the protocol layer. Its context is the message being sent; its decision is a verdict: SK_MSG_PASS (let the send proceed normally) or SK_MSG_REDIRECT (send these bytes to another socket in the map, via bpf_msg_redirect_map).

Source: code/ch08_sockmap/kernel/src/main.rs

#![allow(unused)]
fn main() {
// code/ch08_sockmap/kernel/src/main.rs
// SK_MSG redirect: every send on a socket in the map is redirected to a
// peer socket looked up by a per-connection key in the same map.
#![no_std]
#![no_main]

use aya_bpf::{
    macros::{map, sk_msg},
    maps::SockHash,
    programs::SkMsgContext,
};
use aya_bpf::bindings::bpf_msg_action::*;

const MAX_CONNS: u32 = 65536;
const CB_CONN_ID: usize = 0; // control plane stamps this at setup (Ch.8.3)

// Key = connection id, value = the peer socket for that direction.
#[map]
static CONNS: SockHash<u32> = SockHash::with_max_entries(MAX_CONNS, 0);

#[sk_msg]
pub fn msg_redirect(ctx: SkMsgContext) -> i32 {
    let conn_id = unsafe { ctx.cb(CB_CONN_ID) as u32 };

    // One map-mediated handoff, in the kernel. A lookup miss must never
    // eat data: any error verdict degrades to PASS (fail-open).
    let rc = unsafe {
        aya_bpf::helpers::bpf_msg_redirect_hash(
            &mut ctx as *mut SkMsgContext as *mut _,
            &mut CONNS as *mut SockHash<u32> as *mut _,
            &conn_id as *const u32 as *const _,
            0,
        )
    };
    if rc == SK_MSG_PASS || rc == SK_MSG_REDIRECT {
        rc as i32
    } else {
        SK_MSG_PASS
    }
}

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    unsafe { core::hint::unreachable_unchecked() }
}
}

Two details carry the whole design. The connection id in the cb: the skb control block (32 bytes of per-packet scratch, Chapter 6) is how the kernel side remembers which connection this send belongs to without a per-send map lookup on the tuple - the control plane writes it at connection setup, the program reads it per send. The fail-open fallback: bpf_msg_redirect_hash returns SK_MSG_PASS (accept) or SK_MSG_REDIRECT (done); anything else is an error, and the program passes the data through rather than dropping it. A sockmap misroute must never become a silent data loss.

8.4 The Userspace Side: Building the Socket Table

The userspace side of sockmap is where the sockets come from - and where the correctness lives. The control plane must own both sockets: the front (client-facing) and the back (upstream) socket of every proxied connection. In the classic pattern, a userspace program accepts the incoming connection and dials the backend - then inserts both sockets into the sockhash and hands the data path to the kernel:

Source: code/ch08_sockmap/userspace/src/main.rs

// code/ch08_sockmap/userspace/src/main.rs
// The control plane: accept a connection, dial the backend, insert both
// sockets into the sockhash, and let the kernel move the bytes (Ch.8.4).
use aya::{maps::SockHash, programs::SockMap, Ebpf};
use std::error::Error;
use std::net::TcpListener;
use std::os::fd::AsRawFd;

const KERNEL_ELF: &[u8] =
    include_bytes!("../../kernel/target/bpfel-unknown-none/release/kernel");
const FRONT_ADDR: &str = "127.0.0.1:8080";
const BACK_ADDR: &str = "127.0.0.1:9090";

fn main() -> Result<(), Box<dyn Error>> {
    let mut ebpf = Ebpf::load(KERNEL_ELF)?;

    // Attach the SK_MSG program to the sockhash: it runs on every send
    // on a socket inserted into this map.
    let prog: &mut SockMap = ebpf.program_mut("msg_redirect")?.try_into()?;
    prog.load()?;
    prog.attach(&aya::programs::SockMapAttachType::MsgVerdict)?;

    let mut conns: SockHash<_, u32> = ebpf.map_mut("CONNS")?.try_into()?;
    let front = TcpListener::bind(FRONT_ADDR)?;
    let back = TcpListener::bind(BACK_ADDR)?;
    println!("sockmap control plane ready on {FRONT_ADDR}");

    for (i, (front_sock, _)) in front.incoming().enumerate() {
        let (back_sock, _) = back.accept()?;
        let conn_id = i as u32;

        // Both sockets under the same key; redirects stay in the kernel.
        conns.insert(conn_id, front_sock.as_raw_fd(), 0)?;
        conns.insert(conn_id ^ 1, back_sock.as_raw_fd(), 0)?;
        println!("connection {conn_id}: kernel now moves the bytes");
    }
    Ok(())
}

The pattern is a hybrid: userspace for the connection establishment (where the complexity lives: TLS, auth, backend discovery), kernel for the steady-state data movement (where the speed matters). This is precisely the architecture Cilium’s service mesh uses - and it is why the mesh can claim “the data path never leaves the kernel” while still supporting userspace features: the control plane is userspace, the hot path is not.

8.5 SK_SKB: The Receive Side and the Full Proxy

The receive-side twin, SK_SKB, runs on data received by a socket in the map and can redirect that data to a peer socket (bpf_sk_redirect_map). Put SK_MSG and SK_SKB together and you have a complete in-kernel proxy: sends on socket A redirect to B; receives on B redirect (or pass) to A’s userspace if L7 handling is needed. The verdict semantics mirror SK_MSG: SK_SKB_PASS / SK_SKB_REDIRECT.

The direction flag is where the subtlety lives. BPF_F_INGRESS makes the redirected data land on the receive path of the target socket (so the target’s recv sees it); without it, the data goes to the target’s send path (for chaining). Getting the flag wrong is the classic sockmap bug: the bytes are redirected, the packet disappears into a queue nobody reads, and the connection hangs. Chapter 14’s testing chapter returns to exactly this class of bug with a reproduction recipe.

8.6 What sockmap Buys and What It Costs

The honest balance sheet, because every chapter in this book ends with one:

• Buys: no userspace round trip per message, no copy into userspace buffers, no proxy process on the hot path, kernel-managed flow control between the two sockets (a slow reader still throttles the writer - the receive queue semantics of Chapter 7 apply unchanged).
• Costs: the control plane must manage socket lifetime (a socket in a map is pinned by the map - closing the fd does not close it until it is removed), the map lookup per message, and the loss of userspace processing: any L7 logic (parsing, rewriting, auth) must happen either in the BPF program or back in userspace, which means the SK_MSG pass / redirect decision is a policy decision the control plane must set up correctly in advance.

The numbers that matter (measured in Chapter 13): an in-kernel sockmap redirect saves the two syscalls and the copy of a proxy round trip - roughly a 2-10x reduction in per-message latency for small messages, and a corresponding reduction in the CPU cost per connection. That is why Cilium uses sockmap for service routing and the mesh data path, and why the capstone builds its mesh on exactly this mechanism.

Hands-On Lab

# 1. Build the kernel side and run the control plane.
cd code/ch08_sockmap/kernel && cargo build --release --target bpfel-unknown-none
cd ../userspace && sudo cargo run --release

# 2. Connect a client; the kernel moves the bytes from now on.
nc 127.0.0.1 8080 &

# 3. Prove no userspace proxy is in the path: the control plane process
#    shows zero read()/write() syscalls per message.
sudo strace -c -p <pid-of-userspace>     # after the handshake: quiet
sudo bpftool map dump name CONNS         # both sockets under one key

The comparison experiment: run the same load through a userspace proxy (a plain nc | nc pipe) and watch strace count a read+write per byte for both directions. That syscall+copy cost is what sockmap removes.

Summary

• sockmap/sockhash are BPF maps whose values are sockets; programs attach to the map and run on data sent to (SK_MSG) or received from (SK_SKB) sockets in it.
• The program’s verdict is PASS or REDIRECT, delivered by bpf_msg_redirect_hash / bpf_sk_redirect_map - a map-mediated handoff between sockets, entirely in the kernel.
• The control plane owns the sockets: it accepts, dials, and inserts both sides of a connection under keys; the kernel then moves the bytes.
• BPF_F_INGRESS chooses receive-path delivery on the target; the wrong flag produces a silent hang - the classic sockmap bug.
• The balance: no round trip and no copy against control-plane socket management and a per-message lookup - the trade that makes sockmap the data path of Cilium’s mesh.

Next: Chapter 9 moves the hook from the socket to the cgroup - where entire groups of sockets are filtered, counted, and rate-limited at once.