Chapter 9: cgroup eBPF - Socket Filtering & Resource Control

“The cgroup is where the kernel learns which process is which. Put an eBPF program there and it learns which group is which - and can govern every socket they open, connect, or use.”

Sockmap (Chapter 8) governs sockets one connection at a time. This chapter governs them by the container: the cgroup hooks, attached to a cgroup (v2, in modern kernels) and running on every socket operation made by any process in that group. Socket filters that see every packet of every socket in the group; connect/accept hooks that can rewrite addresses before the connection exists; sockopt hooks that police setsockopt; and the cgroup bandwidth controller that rate-limits the group’s traffic. This is the hook family that makes per-pod network policy possible - and the last piece of the socket-layer story before Part IV turns to Kubernetes, where all of it is applied.

9.1 The cgroup: The Kernel’s Grouping Primitive

A cgroup (control group) is the kernel’s mechanism for grouping processes and governing them as a unit: CPU shares, memory limits, and - since cgroup v2 and BPF - network behaviour. Every process belongs to exactly one cgroup in the hierarchy; in Kubernetes, the container runtime places each pod in its own cgroup (that is how kubectl knows what to kill, and how the OOM killer knows what to blame). For eBPF, the cgroup is a scope: attach a program to the cgroup’s socket hooks and it runs for every socket created or used by any process in that group.

The attachment mechanism is the same link-based one as everything else (Chapter 3): the userspace side opens the cgroup by path (/sys/fs/cgroup/...), attaches the program with a type (BPF_CGROUP_INET_INGRESS, etc.), and the kernel invokes the program at the matching socket operation.

9.2 The Hook Family, By Socket Lifecycle

A socket lives through stages, and cgroup eBPF has a hook at almost every stage:

For this book’s purposes - and for Cilium’s - the two that matter most are CgroupSkb (the per-group packet filter, which is how pod network policy is enforced on the socket layer) and CgroupSockAddr (which is how Cilium’s socket-level load balancing rewrites service addresses at connect() time).

9.3 CgroupSkb: The Group’s Packet Filter

The CgroupSkb program is the cgroup analogue of the socket filter of Chapter 6, but at group scope: one program, every packet of every socket in the group, ingress and egress. The verdict is the familiar one - return 1 to allow, 0 to drop - and the program receives the packet plus the socket’s identity through the context.

Source: code/ch09_cgroup_hooks/kernel/src/main.rs

#![allow(unused)]
fn main() {
// code/ch09_cgroup_hooks/kernel/src/main.rs
// Pod-level egress filtering, cgroup style: policy map, per-CPU counter,
// ring events for denials. Fail-open: a missing entry allows (Ch.9.3).
#![no_std]
#![no_main]

mod parse;

use aya_bpf::{
    macros::{cgroup_skb, map},
    maps::{HashMap, PerCpuArray, RingBuf},
    programs::CgroupSkbContext,
};

const MAX_POLICY: u32 = 1024;
const XDP_VERDICT_ALLOW: i32 = 1;

// The policy: key = dst ip, value = allow (1) / deny (0).
#[map]
static POLICY: HashMap<u32, u8> = HashMap::with_max_entries(MAX_POLICY, 0);

// Hot counter: per-CPU slots, no contention.
#[map]
static PKTS: PerCpuArray<u64> = PerCpuArray::with_max_entries(8, 0);

#[repr(C)]
pub struct DenyEvent {
    pub dst_ip: u32,
    pub ts: u64,
}
#[map]
static DENIES: RingBuf = RingBuf::with_max_entries(2048, 0);

#[cgroup_skb]
pub fn egress_filter(ctx: CgroupSkbContext) -> i32 {
    let Some(start) = ctx.data() else {
        return XDP_VERDICT_ALLOW; // no frame: allow, the stack decides
    };
    let frame = unsafe { parse::frame(start, ctx.data_end()) };

    if let Some(slot) = PKTS.get_ptr_mut(0) {
        unsafe { *slot += 1 };
    }

    // Fail-open: allow anything not explicitly denied.
    let Some(dst_ip) = parse::dst_ip(frame) else {
        return XDP_VERDICT_ALLOW;
    };
    match POLICY.get(&dst_ip) {
        Some(parse::ALLOW) => XDP_VERDICT_ALLOW,
        Some(_) => {
            emit_deny(dst_ip);
            0 // drop
        }
        None => XDP_VERDICT_ALLOW,
    }
}

fn emit_deny(dst_ip: u32) {
    let Ok(entry) = DENIES.reserve::<DenyEvent>(0) else {
        return; // ring full: the drop already happened, skip the event
    };
    unsafe {
        entry.write(DenyEvent {
            dst_ip,
            ts: aya_bpf::helpers::bpf_ktime_get_ns(),
        });
    }
    entry.submit(0);
}

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    unsafe { core::hint::unreachable_unchecked() }
}
}

The design choices are the book’s standing rules applied at group scope: bounds-checked parsing, per-CPU counters on the hot path, the ring buffer for events, and fail-open policy (a missing entry allows, never blocks). The one new idea is the scope: the same program, attached to a pod’s cgroup, governs every socket that pod owns - which is exactly the semantic Kubernetes network policy needs (Chapter 12 shows Cilium’s implementation).

9.4 CgroupSockAddr: Rewriting Connections at connect()

The CgroupSockAddr hook runs inside connect() (and bind()), before the connection exists, and it can rewrite the address the kernel will use. This is how socket-level load balancing works: the program sees “connecting to service 10.96.0.10:80” and rewrites it to “connecting to backend 10.1.2.3:8080” - the kernel then opens the real connection, and no packet-level NAT or proxy is needed at all. The rewrites are recorded in a map so that replies and subsequent packets can be correlated.

The difference from the tc/NAT approach (Chapter 6) is fundamental: packet-level NAT rewrites packets that already exist; socket-level redirection rewrites the connection before it exists. The former runs on the data path and must see both directions; the latter runs once, at connect, and the kernel’s own socket plumbing does the rest. Cilium uses both - CgroupSockAddr for new connections, packet-level eBPF for the rest - and Part IV returns to the split.

9.5 Bandwidth: The Group as a Rate-Limited Entity

The final cgroup capability in this book is bandwidth control: eBPF programs attached to the cgroup that implement token-bucket shaping for the group’s traffic, and the kernel’s max/high rate limits exposed through the cgroup filesystem. The token bucket is the classic shape - a bucket that refills at the configured rate, drains per packet, and drops (egress) or delays packets when empty - implemented in a BPF program with a per-cgroup map for the bucket state. The design lesson is the one that repeats through this book: the state (tokens, timestamps) lives in maps, the decision (allow/delay/drop) is per-packet, and the policy (rates) comes from userspace.

9.6 Attaching to a Pod: The Kubernetes Connection

In Kubernetes, each pod’s processes live in a cgroup, so the attach is: find the pod’s cgroup path (via the container runtime or /sys/fs/cgroup), attach the program with the right type, and the program now governs that pod. Aya’s CgroupSkb and friends attach to an open cgroup fd; the userspace side resolves the path, attaches, and fills the policy maps:

Source: code/ch09_cgroup_hooks/userspace/src/main.rs

// code/ch09_cgroup_hooks/userspace/src/main.rs
// Attach the egress filter to a pod's cgroup and fill its policy.
use aya::{maps::HashMap, programs::CgroupSkb, Ebpf};
use std::error::Error;
use std::fs::File;

const KERNEL_ELF: &[u8] =
    include_bytes!("../../kernel/target/bpfel-unknown-none/release/kernel");
const DEFAULT_CGROUP: &str = "/sys/fs/cgroup/kubepods/pod-demo";
const ALLOWED_DST: u32 = 0x0a00_0001; // 10.0.0.1
const ALLOW: u8 = 1;

fn main() -> Result<(), Box<dyn Error>> {
    let cgroup_path = std::env::args()
        .nth(1)
        .unwrap_or_else(|| DEFAULT_CGROUP.to_string());
    let cgroup = File::open(&cgroup_path)?;

    let mut ebpf = Ebpf::load(KERNEL_ELF)?;
    let prog: &mut CgroupSkb = ebpf.program_mut("egress_filter")?.try_into()?;
    prog.load()?;
    prog.attach(cgroup.try_clone()?, aya::programs::CgroupSkbAttachType::Egress)?;
    println!("egress filter attached to {cgroup_path}");

    let mut policy: HashMap<_, u32, u8> = ebpf.map_mut("POLICY")?.try_into()?;
    policy.insert(ALLOWED_DST, ALLOW, 0)?;
    println!("policy: 10.0.0.1 allowed, everything else passes (fail-open)");
    Ok(())
}

That is the complete per-pod data path: the cgroup scopes the program, the map carries the policy, the ring reports the denials. Chapter 12 shows Cilium scaling this from one pod to a whole cluster - but the primitive is exactly the one you just attached.

Hands-On Lab

# 1. Create a scratch cgroup and run a shell inside it.
sudo mkdir -p /sys/fs/cgroup/demo && echo $$ | sudo tee /sys/fs/cgroup/demo/cgroup.procs

# 2. Attach the egress filter to it and fill the policy.
cd code/ch09_cgroup_hooks/kernel && cargo build --release --target bpfel-unknown-none
cd ../userspace && sudo cargo run --release -- /sys/fs/cgroup/demo

# 3. Prove the scope: traffic from the cgroup is filtered, traffic
#    outside it is not.
curl -s http://10.0.0.1/ >/dev/null    # inside the cgroup: allowed
# add a deny rule for 10.0.0.2, then curl it from inside: dropped + event
sudo bpftool map dump name DENIES

The fail-open property is worth testing on purpose: delete the policy map entry and confirm the pod’s traffic still flows. A policy engine that black-holes on an empty table is a network incident waiting to happen.

Summary

• cgroups are the kernel’s grouping primitive and the scope for a family of socket hooks: one program governs every socket of every process in the group.
• The hooks map to the socket lifecycle: CgroupSkb filters packets, CgroupSockAddr rewrites connects, CgroupSockopt polices options.
• CgroupSkb is the per-group packet filter: bounds-checked parse, policy map, per-CPU counters, ring events, fail-open.
• CgroupSockAddr rewrites the connection before it exists - the socket-level LB mechanism, distinct from packet-level NAT.
• Bandwidth control is a token bucket in BPF maps, with rates from userspace.
• Attaching to a pod is attaching to its cgroup: the primitive behind per-pod network policy, ready to be scaled by Part IV.

Next: Part IV steps out of the single host and into the cluster - Chapter 10 is the Kubernetes networking model: CNI, kube-proxy, and the service abstraction eBPF was built to replace.