Control Groups

A control group, or cgroup, is a set of processes that the kernel accounts for and limits as a unit. Where a namespace controls what a process can see, a control group controls how much it can use: how many processes it may contain, how much memory, how much CPU time. Control groups form a tree, and a limit on a group applies to everything below it.

In this chapter, we will learn how processes are placed in a group and how the kernel enforces a limit. We will first create a group and measure 2 limits with cgfacts. Then we will write cgmodel, a model of hierarchical accounting, and follow a fork and an allocation that are stopped by a limit set on a parent group.

The interface

Control groups have no system calls of their own. The kernel presents them as a file system, mounted at /sys/fs/cgroup. A directory is a group. Creating a directory creates a group, and the kernel fills it with control files.

Every process is in exactly 1 group, shown in /proc/<pid>/cgroup. A child process starts in the group of its parent, so a process that has been moved into a group cannot leave the group’s accounting by forking.

A controller is the part of the kernel that accounts for 1 kind of resource: pids, memory, cpu, io. A group has the control files of a controller only when its parent lists that controller in cgroup.subtree_control. On this machine the user’s session is given the memory and pids controllers, which is why the chapter measures those 2.

The group of the shell this chapter was written in, and the controllers available to it:

$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-1f7da0d1-a682-46b3-8151-d6603c0e21ae.scope
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
memory pids

The line in /proc/self/cgroup is a path below /sys/fs/cgroup. Each component is a group, created by the service manager systemd: a group for all users, a group for user 1000, a group for that user’s session manager, and so on down to a group for a single terminal window. The root group has 6 controllers, and the group of user 1000 has received 2 of them.

The counters of a group can be read at any time. For the group of user 1000:

$ cd /sys/fs/cgroup/user.slice/user-1000.slice
$ cat pids.current pids.max
1347
40594
$ cat memory.current memory.max
12315291648
max
$ grep -E '^(anon|file) ' memory.stat
anon 5923209216
file 5730648064

All processes of the user together are 1347 tasks, of 40594 allowed, and use 12.3 GB, with no limit. memory.stat divides the usage by kind: 5.9 GB is anonymous memory of programs, and 5.7 GB is page cache, which is charged to the group whose process first read the file (chapter 8).

The kernel’s structures

A group is a struct cgroup. A task does not point to its group directly. It points to a struct css_set, which holds, for each controller, the state of the group the task belongs to. Tasks in the same groups share a css_set, so fork only has to copy a pointer. These are fields from include/linux/cgroup-defs.h and include/linux/page_counter.h of Linux 5.15.

struct css_set {
        struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT]; /* state per controller */
        struct list_head tasks;                /* the tasks that use this set */
        struct cgroup *dfl_cgrp;               /* the group, in the unified tree */
        /* ... */
};

struct cgroup {
        struct cgroup_subsys_state self;       /* links to the parent group */
        int level;                             /* depth in the tree */
        struct kernfs_node *kn;                /* its directory under /sys/fs/cgroup */
        u16 subtree_control;                   /* controllers passed to children */
        struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT];
        /* ... */
};

struct cgroup_subsys_state {
        struct cgroup *cgroup;                 /* the group this state belongs to */
        struct cgroup_subsys_state *parent;    /* the same controller, parent group */
        /* ... */
};

struct page_counter {
        atomic_long_t usage;                   /* pages charged: memory.current */
        unsigned long max;                     /* the limit: memory.max */
        struct page_counter *parent;           /* the counter of the parent group */
        /* ... */
};

Measuring limits

cgfacts needs only the standard libraries. It reads and writes files and starts shell commands.

$ cargo new cgfacts

A group is a path. set and get write and read a control file, and event reads a counter from a file such as pids.events.

Listing 12.1 The imports of cgfacts, in Go. code/go/cgfacts/main.go

import (
	"bufio"
	"errors"
	"fmt"
	"log"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"syscall"
	"time"
)

Listing 12.2 The imports of cgfacts, in Rust. code/rust/cgfacts/src/main.rs

use std::fs;
use std::io;
use std::os::unix::process::ExitStatusExt;
use std::path::{Path, PathBuf};
use std::process::{self, Command};
use std::thread;
use std::time::Duration;

Listing 12.3 Reading and writing control files, in Go. code/go/cgfacts/main.go

// A group is the directory of a control group under /sys/fs/cgroup.
type group string

// set writes a value to one of the group's control files.
func (g group) set(file, value string) error {
	return os.WriteFile(filepath.Join(string(g), file), []byte(value), 0)
}

// get reads one of the group's control files.
func (g group) get(file string) (string, error) {
	text, err := os.ReadFile(filepath.Join(string(g), file))

	return strings.TrimSpace(string(text)), err
}

// event returns the counter with the given key from a file of "key value"
// lines, such as pids.events and memory.events.
func (g group) event(file, key string) (string, error) {
	text, err := g.get(file)

	if err != nil {
		return "", err
	}

	for lines := bufio.NewScanner(strings.NewReader(text)); lines.Scan(); {
		if fields := strings.Fields(lines.Text()); len(fields) == 2 && fields[0] == key {
			return fields[1], nil
		}
	}

	return "", fmt.Errorf("%s: no line %q", file, key)
}

Listing 12.4 Reading and writing control files, in Rust. code/rust/cgfacts/src/main.rs

/// The directory of a control group under `/sys/fs/cgroup`.
struct Group(PathBuf);

impl Group {
    /// Writes a value to one of the group's control files.
    fn set(&self, file: &str, value: &str) -> io::Result<()> {
        fs::write(self.0.join(file), value)
    }

    /// Reads one of the group's control files.
    fn get(&self, file: &str) -> io::Result<String> {
        Ok(fs::read_to_string(self.0.join(file))?.trim().to_owned())
    }

    /// Returns the counter with the given key from a file of "key value"
    /// lines, such as `pids.events` and `memory.events`.
    fn event(&self, file: &str, key: &str) -> io::Result<String> {
        self.get(file)?
            .lines()
            .filter_map(|line| line.split_once(' '))
            .find(|(name, _)| *name == key)
            .map(|(_, value)| value.to_owned())
            .ok_or_else(|| io::ErrorKind::NotFound.into())
    }
}

create finds a place for a new group. It walks up from the group of the program to the nearest group whose cgroup.subtree_control contains memory and pids, and makes a directory there. An ordinary user can do this because the system’s service manager has given the user ownership of that part of the tree.

command builds a shell command that first writes its own pid to the cgroup.procs of the new group. remove deletes the group. A group can be removed only when it is empty, so remove first writes to cgroup.kill, which ends every process in the group.

Listing 12.5 Creating a group, entering it, and removing it, in Go. code/go/cgfacts/main.go

// create makes a new group. It goes up from the group of this process to the
// nearest group that distributes the memory and pids controllers to its
// children, and creates a directory there.
func create() (group, error) {
	line, err := os.ReadFile("/proc/self/cgroup")

	if err != nil {
		return "", err
	}

	_, path, _ := strings.Cut(strings.TrimSpace(string(line)), "::")

	for dir := filepath.Join("/sys/fs/cgroup", path); dir != "/sys/fs/cgroup"; dir = filepath.Dir(dir) {
		enabled, err := group(dir).get("cgroup.subtree_control")

		if err != nil || !strings.Contains(enabled, "memory") || !strings.Contains(enabled, "pids") {
			continue
		}

		created := filepath.Join(dir, fmt.Sprintf("cgfacts-%d", os.Getpid()))

		return group(created), os.Mkdir(created, 0o755)
	}

	return "", errors.New("no parent group delegates the memory and pids controllers")
}

// command returns a shell script that will run in the group. The shell
// first moves itself into the group by writing its own process number to
// cgroup.procs, and every process it starts afterwards is created there.
func (g group) command(script string) *exec.Cmd {
	return exec.Command("sh", "-c", `echo $$ > "$0/cgroup.procs" && `+script, string(g))
}

// remove ends every process left in the group, waits until the group is
// empty, and deletes it.
func (g group) remove() error {
	if err := g.set("cgroup.kill", "1"); err != nil {
		return err
	}

	for {
		current, err := g.get("pids.current")

		if err != nil {
			return err
		}

		if current == "0" {
			return os.Remove(string(g))
		}

		time.Sleep(time.Millisecond)
	}
}

Listing 12.6 Creating a group, entering it, and removing it, in Rust. code/rust/cgfacts/src/main.rs

impl Group {
    /// Makes a new group. It goes up from the group of this process to the
    /// nearest group that distributes the memory and pids controllers to its
    /// children, and creates a directory there.
    fn create() -> io::Result<Group> {
        let line = fs::read_to_string("/proc/self/cgroup")?;
        let path = line.trim().split_once("::").map_or("", |(_, path)| path);
        let root = Path::new("/sys/fs/cgroup");
        let own = root.join(path.trim_start_matches('/'));

        for dir in own.ancestors().take_while(|dir| *dir != root) {
            let Ok(enabled) = fs::read_to_string(dir.join("cgroup.subtree_control")) else {
                continue;
            };

            if enabled.contains("memory") && enabled.contains("pids") {
                let created = dir.join(format!("cgfacts-{}", process::id()));

                fs::create_dir(&created)?;

                return Ok(Group(created));
            }
        }

        Err(io::Error::other(
            "no parent group delegates the memory and pids controllers",
        ))
    }

    /// Returns a shell script that will run in the group. The shell first
    /// moves itself into the group by writing its own process number to
    /// `cgroup.procs`, and every process it starts afterwards is created
    /// there.
    fn command(&self, script: &str) -> Command {
        let mut command = Command::new("sh");

        command
            .arg("-c")
            .arg(format!(r#"echo $$ > "$0/cgroup.procs" && {script}"#))
            .arg(&self.0);

        command
    }

    /// Ends every process left in the group, waits until the group is empty,
    /// and deletes it.
    fn remove(&self) -> io::Result<()> {
        self.set("cgroup.kill", "1")?;

        while self.get("pids.current")? != "0" {
            thread::sleep(Duration::from_millis(1));
        }

        fs::remove_dir(&self.0)
    }
}

count starts a process in the group and reads pids.current.

Listing 12.7 The kernel counts the processes of a group, in Go. code/go/cgfacts/main.go

// count starts a process in the group and reads the group's process count.
func count(g group) error {
	child := g.command("exec sleep 30")

	if err := child.Start(); err != nil {
		return err
	}

	current, err := g.get("pids.current")

	for err == nil && current != "1" {
		time.Sleep(time.Millisecond)
		current, err = g.get("pids.current")
	}

	child.Process.Kill()
	child.Wait()

	if err != nil {
		return err
	}

	fmt.Printf("a process moved into the new group: pids.current %s\n", current)

	return nil
}

Listing 12.8 The kernel counts the processes of a group, in Rust. code/rust/cgfacts/src/main.rs

/// Starts a process in the group and reads the group's process count.
fn count(group: &Group) -> io::Result<()> {
    let mut child = group.command("exec sleep 30").spawn()?;

    while group.get("pids.current")? != "1" {
        thread::sleep(Duration::from_millis(1));
    }

    child.kill()?;
    child.wait()?;

    println!("a process moved into the new group: pids.current 1");

    Ok(())
}

limitPids sets pids.max to 4. The shell, which is in the group, then tries to start 6 background commands and prints a line for each success.

Listing 12.9 A limit on the number of processes, in Go. code/go/cgfacts/main.go

// limitPids sets pids.max to 4 and runs a shell that tries to start 6
// background commands and prints a line for each one it started.
func limitPids(g group) error {
	if err := g.set("pids.max", "4"); err != nil {
		return err
	}

	started, _ := g.command("for i in 1 2 3 4 5 6; do sleep 1 & echo started; done 2>/dev/null").Output()
	refused, err := g.event("pids.events", "max")

	if err != nil {
		return err
	}

	fmt.Printf("pids.max 4, a shell starts background commands: started %d, forks refused %s\n",
		strings.Count(string(started), "started"), refused)

	return g.set("pids.max", "max")
}

Listing 12.10 A limit on the number of processes, in Rust. code/rust/cgfacts/src/main.rs

/// Sets `pids.max` to 4 and runs a shell that tries to start 6 background
/// commands and prints a line for each one it started.
fn limit_pids(group: &Group) -> io::Result<()> {
    group.set("pids.max", "4")?;

    let script = "for i in 1 2 3 4 5 6; do sleep 1 & echo started; done 2>/dev/null";
    let output = group.command(script).output()?;
    let started = String::from_utf8_lossy(&output.stdout)
        .matches("started")
        .count();

    println!(
        "pids.max 4, a shell starts background commands: started {started}, forks refused {}",
        group.event("pids.events", "max")?
    );

    group.set("pids.max", "max")
}

limitMemory sets memory.max to 32 MiB and memory.swap.max to 0, so that the group cannot move its memory to swap space. It then runs tail /dev/zero. tail keeps the last lines of its input in memory, and /dev/zero supplies bytes without a line end, so the memory of the process grows without bound.

Listing 12.11 A limit on memory, in Go. code/go/cgfacts/main.go

// limitMemory sets memory.max to 32 MiB with no swap and runs a command
// whose memory grows without bound.
func limitMemory(g group) error {
	if err := g.set("memory.max", "33554432"); err != nil {
		return err
	}

	if err := g.set("memory.swap.max", "0"); err != nil {
		return err
	}

	child := g.command("exec tail /dev/zero")
	child.Run()

	kills, err := g.event("memory.events", "oom_kill")

	if err != nil {
		return err
	}

	if status, ok := child.ProcessState.Sys().(syscall.WaitStatus); ok && status.Signaled() {
		fmt.Printf("memory.max 32 MiB, a command that grows without bound: killed by signal %d, oom_kill %s\n",
			status.Signal(), kills)
	}

	return nil
}

Listing 12.12 A limit on memory, in Rust. code/rust/cgfacts/src/main.rs

/// Sets `memory.max` to 32 MiB with no swap and runs a command whose memory
/// grows without bound.
fn limit_memory(group: &Group) -> io::Result<()> {
    group.set("memory.max", "33554432")?;
    group.set("memory.swap.max", "0")?;

    let status = group.command("exec tail /dev/zero").status()?;
    let kills = group.event("memory.events", "oom_kill")?;

    if let Some(signal) = status.signal() {
        println!(
            "memory.max 32 MiB, a command that grows without bound: killed by signal {signal}, oom_kill {kills}"
        );
    }

    Ok(())
}

a process moved into the new group: pids.current 1
pids.max 4, a shell starts background commands: started 3, forks refused 1
memory.max 32 MiB, a command that grows without bound: killed by signal 9, oom_kill 1

The 2 controllers react differently because the resources differ. A fork can be refused with an error, and the program can handle it. An ordinary memory access has no way to return an error. When a group reaches its memory limit, the kernel first tries to free memory inside the group by discarding clean page cache pages and writing out dirty ones. When that is not enough, the OOM killer selects a process in the group and ends it.

Hierarchical accounting

The counters of a group include all of its descendants. A process in a child group is counted in the child, in the parent, and in every group up to the root, and a limit at any of those levels can stop it.

The kernel enforces a limit at the moment a resource is requested, by walking up the tree. For the pids controller the moment is fork.

For the memory controller the moment is the allocation of a frame, usually inside a page fault (chapter 7). A page fault cannot return an error to the program, so the kernel has to make room or end a process.

The events are counted hierarchically as well. memory.events of the group of user 1000 on this machine shows oom_kill 16, and those 16 kills include every run of cgfacts made while this chapter was written, 4 levels further down the tree.

This is what makes delegation safe. An administrator sets a limit on a group and hands the subtree below it to a user or to a container. The owner of the subtree can create groups and distribute the resource among them, and cannot exceed the limit set above.

Modelling the hierarchy

Listing 12.13 The imports of cgmodel, in Go. code/go/cgmodel/main.go

import (
	"errors"
	"fmt"
	"math"
)

Listing 12.14 The imports of cgmodel, in Rust. code/rust/cgmodel/src/main.rs

use std::fmt;

Listing 12.15 Groups and processes, in Go. code/go/cgmodel/main.go

// unlimited is the value of a limit that is written "max" in a control file.
const unlimited = math.MaxInt

var errAgain = errors.New("EAGAIN")

// A group is one node of the hierarchy. pids and memory are the usage of the
// group and of every group below it. refused and oomKills are the counters
// the kernel publishes in pids.events and memory.events.
type group struct {
	name      string
	parent    *group
	children  []*group
	processes []*process

	pidsMax, memoryMax int
	pids, memory       int
	refused, oomKills  int
}

// A process belongs to one group and holds an amount of memory.
type process struct {
	name   string
	home   *group
	memory int
}

// child adds a group below g, with no limits of its own.
func (g *group) child(name string) *group {
	c := &group{name: name, parent: g, pidsMax: unlimited, memoryMax: unlimited}
	g.children = append(g.children, c)

	return c
}

Listing 12.16 Groups and processes, in Rust. code/rust/cgmodel/src/main.rs

/// The value of a limit that is written "max" in a control file.
const UNLIMITED: usize = usize::MAX;

#[derive(Debug)]
struct Again;

impl fmt::Display for Again {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("EAGAIN")
    }
}

/// One node of the hierarchy. `pids` and `memory` are the usage of the group
/// and of every group below it. `refused` and `oom_kills` are the counters
/// the kernel publishes in `pids.events` and `memory.events`.
struct Group {
    parent: Option<usize>,
    pids_max: usize,
    memory_max: usize,
    pids: usize,
    memory: usize,
    refused: usize,
    oom_kills: usize,
}

/// A process belongs to one group and holds an amount of memory.
struct Process {
    name: &'static str,
    home: usize,
    memory: usize,
    alive: bool,
}

/// The hierarchy: groups and processes, referred to by index.
#[derive(Default)]
struct Hierarchy {
    groups: Vec<Group>,
    processes: Vec<Process>,
}

impl Hierarchy {
    /// Adds a group below `parent` and returns its index.
    fn group(&mut self, parent: Option<usize>, pids_max: usize, memory_max: usize) -> usize {
        self.groups.push(Group {
            parent,
            pids_max,
            memory_max,
            pids: 0,
            memory: 0,
            refused: 0,
            oom_kills: 0,
        });

        self.groups.len() - 1
    }

    /// The group itself and every group above it, nearest first.
    fn upward(&self, group: usize) -> Vec<usize> {
        let mut path = vec![group];

        while let Some(parent) = self.groups[path[path.len() - 1]].parent {
            path.push(parent);
        }

        path
    }
}

spawn checks every level before it changes any counter. The refusal is recorded in the group whose limit was reached, which is where the kernel increments pids.events.

Listing 12.17 Starting a process under a hierarchy of limits, in Go. code/go/cgmodel/main.go

// spawn starts a process in g. The process is counted in g and in every
// group above it, so it is refused when any of them is at its limit.
func (g *group) spawn(name string) (*process, error) {
	for up := g; up != nil; up = up.parent {
		if up.pids == up.pidsMax {
			up.refused++

			return nil, errAgain
		}
	}

	for up := g; up != nil; up = up.parent {
		up.pids++
	}

	p := &process{name: name, home: g}
	g.processes = append(g.processes, p)

	return p, nil
}

Listing 12.18 Starting a process under a hierarchy of limits, in Rust. code/rust/cgmodel/src/main.rs

impl Hierarchy {
    /// Starts a process in a group and returns its index. The process is
    /// counted in the group and in every group above it, so it is refused
    /// when any of them is at its limit.
    fn spawn(&mut self, group: usize, name: &'static str) -> Result<usize, Again> {
        let path = self.upward(group);

        for &up in &path {
            if self.groups[up].pids == self.groups[up].pids_max {
                self.groups[up].refused += 1;

                return Err(Again);
            }
        }

        for &up in &path {
            self.groups[up].pids += 1;
        }

        self.processes.push(Process {
            name,
            home: group,
            memory: 0,
            alive: true,
        });

        Ok(self.processes.len() - 1)
    }
}

allocate charges first and then enforces. For each level that is over its limit, it selects the process with the most memory in the subtree of that level. The victim is chosen from the whole subtree, so it can be a process in a different child group from the one that made the allocation.

Listing 12.19 Charging memory, and ending the largest process, in Go. code/go/cgmodel/main.go

// charge adds an amount, which may be negative, to the memory count of a
// group and of every group above it.
func (g *group) charge(amount int) {
	for up := g; up != nil; up = up.parent {
		up.memory += amount
	}
}

// allocate gives a process more memory. While a group above the process is
// over its limit, the largest process in that group's subtree is killed. It
// returns the names of the processes killed.
func (p *process) allocate(amount int) []string {
	p.memory += amount
	p.home.charge(amount)

	var killed []string

	for up := p.home; up != nil; up = up.parent {
		for up.memory > up.memoryMax {
			victim := up.largest()
			up.oomKills++
			killed = append(killed, victim.name)
			victim.kill()
		}
	}

	return killed
}

// largest returns the process with the most memory in the subtree of g.
func (g *group) largest() *process {
	var found *process

	for _, p := range g.processes {
		if found == nil || p.memory > found.memory {
			found = p
		}
	}

	for _, c := range g.children {
		if p := c.largest(); p != nil && (found == nil || p.memory > found.memory) {
			found = p
		}
	}

	return found
}

// kill removes a process from its group and returns its memory and its
// place in the process count.
func (p *process) kill() {
	p.home.charge(-p.memory)

	for up := p.home; up != nil; up = up.parent {
		up.pids--
	}

	for i, other := range p.home.processes {
		if other == p {
			p.home.processes = append(p.home.processes[:i], p.home.processes[i+1:]...)

			break
		}
	}
}

Listing 12.20 Charging memory, and ending the largest process, in Rust. code/rust/cgmodel/src/main.rs

impl Hierarchy {
    /// Gives a process more memory. While a group above the process is over
    /// its limit, the largest process in that group's subtree is killed.
    /// Returns the names of the processes killed.
    fn allocate(&mut self, process: usize, amount: usize) -> Vec<&'static str> {
        let path = self.upward(self.processes[process].home);

        self.processes[process].memory += amount;

        for &up in &path {
            self.groups[up].memory += amount;
        }

        let mut killed = Vec::new();

        for &up in &path {
            while self.groups[up].memory > self.groups[up].memory_max {
                let Some(victim) = self.largest(up) else {
                    break;
                };

                self.groups[up].oom_kills += 1;
                killed.push(self.processes[victim].name);
                self.kill(victim);
            }
        }

        killed
    }

    /// Returns the living process with the most memory in the subtree of a
    /// group.
    fn largest(&self, group: usize) -> Option<usize> {
        (0..self.processes.len())
            .filter(|&p| self.processes[p].alive)
            .filter(|&p| self.upward(self.processes[p].home).contains(&group))
            .max_by_key(|&p| self.processes[p].memory)
    }

    /// Ends a process and returns its memory and its place in the process
    /// count.
    fn kill(&mut self, process: usize) {
        let memory = self.processes[process].memory;

        for up in self.upward(self.processes[process].home) {
            self.groups[up].memory -= memory;
            self.groups[up].pids -= 1;
        }

        self.processes[process].alive = false;
    }
}

Listing 12.21 The trace, in Go. code/go/cgmodel/main.go

func main() {
	team := &group{name: "team", pidsMax: 4, memoryMax: 100}
	jobA, jobB := team.child("job-a"), team.child("job-b")

	var started []*process

	for _, start := range []struct {
		in   *group
		name string
	}{{jobA, "a1"}, {jobA, "a2"}, {jobA, "a3"}, {jobB, "b1"}} {
		p, err := start.in.spawn(start.name)

		if err != nil {
			fmt.Println(err)

			return
		}

		started = append(started, p)
	}

	fmt.Printf("3 processes in job-a and 1 in job-b: pids.current job-a %d, job-b %d, team %d\n",
		jobA.pids, jobB.pids, team.pids)

	_, err := jobB.spawn("b2")
	fmt.Printf("another process in job-b, team at pids.max 4: %v, pids.events max of team %d\n", err, team.refused)

	a1, b1 := started[0], started[3]

	a1.allocate(60)
	fmt.Printf("a1 allocates 60: memory.current job-a %d, job-b %d, team %d\n", jobA.memory, jobB.memory, team.memory)

	killed := b1.allocate(50)
	fmt.Printf("b1 allocates 50, team at memory.max 100: killed %v, oom_kill of team %d\n", killed, team.oomKills)
	fmt.Printf("afterwards: memory.current job-a %d, job-b %d, team %d, pids.current team %d\n",
		jobA.memory, jobB.memory, team.memory, team.pids)
}

Listing 12.22 The trace, in Rust. code/rust/cgmodel/src/main.rs

fn main() -> Result<(), Again> {
    let mut tree = Hierarchy::default();

    let team = tree.group(None, 4, 100);
    let job_a = tree.group(Some(team), UNLIMITED, UNLIMITED);
    let job_b = tree.group(Some(team), UNLIMITED, UNLIMITED);

    let a1 = tree.spawn(job_a, "a1")?;
    tree.spawn(job_a, "a2")?;
    tree.spawn(job_a, "a3")?;
    let b1 = tree.spawn(job_b, "b1")?;

    println!(
        "3 processes in job-a and 1 in job-b: pids.current job-a {}, job-b {}, team {}",
        tree.groups[job_a].pids, tree.groups[job_b].pids, tree.groups[team].pids
    );

    if let Err(error) = tree.spawn(job_b, "b2") {
        println!(
            "another process in job-b, team at pids.max 4: {error}, pids.events max of team {}",
            tree.groups[team].refused
        );
    }

    tree.allocate(a1, 60);
    println!(
        "a1 allocates 60: memory.current job-a {}, job-b {}, team {}",
        tree.groups[job_a].memory, tree.groups[job_b].memory, tree.groups[team].memory
    );

    let killed = tree.allocate(b1, 50);
    println!(
        "b1 allocates 50, team at memory.max 100: killed [{}], oom_kill of team {}",
        killed.join(" "),
        tree.groups[team].oom_kills
    );
    println!(
        "afterwards: memory.current job-a {}, job-b {}, team {}, pids.current team {}",
        tree.groups[job_a].memory,
        tree.groups[job_b].memory,
        tree.groups[team].memory,
        tree.groups[team].pids
    );

    Ok(())
}

3 processes in job-a and 1 in job-b: pids.current job-a 3, job-b 1, team 4
another process in job-b, team at pids.max 4: EAGAIN, pids.events max of team 1
a1 allocates 60: memory.current job-a 60, job-b 0, team 60
b1 allocates 50, team at memory.max 100: killed [a1], oom_kill of team 1
afterwards: memory.current job-a 0, job-b 50, team 50, pids.current team 3

The other controllers

The session of an ordinary user on this machine has only memory and pids. A container runtime that runs with privilege, or a service manager, also uses these.

cpu.max and memory.max are what the options --cpus and --memory of a container runtime set, and the resource limits of a Kubernetes pod are written to these files.

This chapter describes cgroup v2, in which all controllers share a single tree. The older cgroup v1 gave each controller a tree of its own, mounted separately, and it is still found on older systems. stat -fc %T /sys/fs/cgroup prints cgroup2fs on a system that uses v2.

Inspecting control groups on a live system

A process that disappears without an error message has often been ended by the OOM killer of its group. memory.events of the group, and the kernel log shown by dmesg, record it.

The complete programs

Listing 12.23 The complete cgfacts, in Go. code/go/cgfacts/main.go

// Command cgfacts measures control groups. It creates a group below the one
// it runs in, starts shell commands inside the new group, and observes the
// kernel counting their processes, refusing a fork beyond pids.max, and
// ending a process that exceeds memory.max.
//
// Run it from code/go with:
//
//	go run ./cgfacts
package main

import (
	"bufio"
	"errors"
	"fmt"
	"log"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"syscall"
	"time"
)


// A group is the directory of a control group under /sys/fs/cgroup.
type group string

// set writes a value to one of the group's control files.
func (g group) set(file, value string) error {
	return os.WriteFile(filepath.Join(string(g), file), []byte(value), 0)
}

// get reads one of the group's control files.
func (g group) get(file string) (string, error) {
	text, err := os.ReadFile(filepath.Join(string(g), file))

	return strings.TrimSpace(string(text)), err
}

// event returns the counter with the given key from a file of "key value"
// lines, such as pids.events and memory.events.
func (g group) event(file, key string) (string, error) {
	text, err := g.get(file)

	if err != nil {
		return "", err
	}

	for lines := bufio.NewScanner(strings.NewReader(text)); lines.Scan(); {
		if fields := strings.Fields(lines.Text()); len(fields) == 2 && fields[0] == key {
			return fields[1], nil
		}
	}

	return "", fmt.Errorf("%s: no line %q", file, key)
}


// create makes a new group. It goes up from the group of this process to the
// nearest group that distributes the memory and pids controllers to its
// children, and creates a directory there.
func create() (group, error) {
	line, err := os.ReadFile("/proc/self/cgroup")

	if err != nil {
		return "", err
	}

	_, path, _ := strings.Cut(strings.TrimSpace(string(line)), "::")

	for dir := filepath.Join("/sys/fs/cgroup", path); dir != "/sys/fs/cgroup"; dir = filepath.Dir(dir) {
		enabled, err := group(dir).get("cgroup.subtree_control")

		if err != nil || !strings.Contains(enabled, "memory") || !strings.Contains(enabled, "pids") {
			continue
		}

		created := filepath.Join(dir, fmt.Sprintf("cgfacts-%d", os.Getpid()))

		return group(created), os.Mkdir(created, 0o755)
	}

	return "", errors.New("no parent group delegates the memory and pids controllers")
}

// command returns a shell script that will run in the group. The shell
// first moves itself into the group by writing its own process number to
// cgroup.procs, and every process it starts afterwards is created there.
func (g group) command(script string) *exec.Cmd {
	return exec.Command("sh", "-c", `echo $$ > "$0/cgroup.procs" && `+script, string(g))
}

// remove ends every process left in the group, waits until the group is
// empty, and deletes it.
func (g group) remove() error {
	if err := g.set("cgroup.kill", "1"); err != nil {
		return err
	}

	for {
		current, err := g.get("pids.current")

		if err != nil {
			return err
		}

		if current == "0" {
			return os.Remove(string(g))
		}

		time.Sleep(time.Millisecond)
	}
}


// count starts a process in the group and reads the group's process count.
func count(g group) error {
	child := g.command("exec sleep 30")

	if err := child.Start(); err != nil {
		return err
	}

	current, err := g.get("pids.current")

	for err == nil && current != "1" {
		time.Sleep(time.Millisecond)
		current, err = g.get("pids.current")
	}

	child.Process.Kill()
	child.Wait()

	if err != nil {
		return err
	}

	fmt.Printf("a process moved into the new group: pids.current %s\n", current)

	return nil
}


// limitPids sets pids.max to 4 and runs a shell that tries to start 6
// background commands and prints a line for each one it started.
func limitPids(g group) error {
	if err := g.set("pids.max", "4"); err != nil {
		return err
	}

	started, _ := g.command("for i in 1 2 3 4 5 6; do sleep 1 & echo started; done 2>/dev/null").Output()
	refused, err := g.event("pids.events", "max")

	if err != nil {
		return err
	}

	fmt.Printf("pids.max 4, a shell starts background commands: started %d, forks refused %s\n",
		strings.Count(string(started), "started"), refused)

	return g.set("pids.max", "max")
}


// limitMemory sets memory.max to 32 MiB with no swap and runs a command
// whose memory grows without bound.
func limitMemory(g group) error {
	if err := g.set("memory.max", "33554432"); err != nil {
		return err
	}

	if err := g.set("memory.swap.max", "0"); err != nil {
		return err
	}

	child := g.command("exec tail /dev/zero")
	child.Run()

	kills, err := g.event("memory.events", "oom_kill")

	if err != nil {
		return err
	}

	if status, ok := child.ProcessState.Sys().(syscall.WaitStatus); ok && status.Signaled() {
		fmt.Printf("memory.max 32 MiB, a command that grows without bound: killed by signal %d, oom_kill %s\n",
			status.Signal(), kills)
	}

	return nil
}


func run() error {
	g, err := create()

	if err != nil {
		return err
	}

	for _, experiment := range []func(group) error{count, limitPids, limitMemory} {
		if err := experiment(g); err != nil {
			g.remove()

			return err
		}
	}

	return g.remove()
}

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

Listing 12.24 The complete cgfacts, in Rust. code/rust/cgfacts/src/main.rs

//! Measures control groups. It creates a group below the one it runs in,
//! starts shell commands inside the new group, and observes the kernel
//! counting their processes, refusing a fork beyond `pids.max`, and ending a
//! process that exceeds `memory.max`.
//!
//! Run it from code/rust with:
//!
//! ```text
//! cargo run -p cgfacts
//! ```

use std::fs;
use std::io;
use std::os::unix::process::ExitStatusExt;
use std::path::{Path, PathBuf};
use std::process::{self, Command};
use std::thread;
use std::time::Duration;

/// The directory of a control group under `/sys/fs/cgroup`.
struct Group(PathBuf);

impl Group {
    /// Writes a value to one of the group's control files.
    fn set(&self, file: &str, value: &str) -> io::Result<()> {
        fs::write(self.0.join(file), value)
    }

    /// Reads one of the group's control files.
    fn get(&self, file: &str) -> io::Result<String> {
        Ok(fs::read_to_string(self.0.join(file))?.trim().to_owned())
    }

    /// Returns the counter with the given key from a file of "key value"
    /// lines, such as `pids.events` and `memory.events`.
    fn event(&self, file: &str, key: &str) -> io::Result<String> {
        self.get(file)?
            .lines()
            .filter_map(|line| line.split_once(' '))
            .find(|(name, _)| *name == key)
            .map(|(_, value)| value.to_owned())
            .ok_or_else(|| io::ErrorKind::NotFound.into())
    }
}

impl Group {
    /// Makes a new group. It goes up from the group of this process to the
    /// nearest group that distributes the memory and pids controllers to its
    /// children, and creates a directory there.
    fn create() -> io::Result<Group> {
        let line = fs::read_to_string("/proc/self/cgroup")?;
        let path = line.trim().split_once("::").map_or("", |(_, path)| path);
        let root = Path::new("/sys/fs/cgroup");
        let own = root.join(path.trim_start_matches('/'));

        for dir in own.ancestors().take_while(|dir| *dir != root) {
            let Ok(enabled) = fs::read_to_string(dir.join("cgroup.subtree_control")) else {
                continue;
            };

            if enabled.contains("memory") && enabled.contains("pids") {
                let created = dir.join(format!("cgfacts-{}", process::id()));

                fs::create_dir(&created)?;

                return Ok(Group(created));
            }
        }

        Err(io::Error::other(
            "no parent group delegates the memory and pids controllers",
        ))
    }

    /// Returns a shell script that will run in the group. The shell first
    /// moves itself into the group by writing its own process number to
    /// `cgroup.procs`, and every process it starts afterwards is created
    /// there.
    fn command(&self, script: &str) -> Command {
        let mut command = Command::new("sh");

        command
            .arg("-c")
            .arg(format!(r#"echo $$ > "$0/cgroup.procs" && {script}"#))
            .arg(&self.0);

        command
    }

    /// Ends every process left in the group, waits until the group is empty,
    /// and deletes it.
    fn remove(&self) -> io::Result<()> {
        self.set("cgroup.kill", "1")?;

        while self.get("pids.current")? != "0" {
            thread::sleep(Duration::from_millis(1));
        }

        fs::remove_dir(&self.0)
    }
}

/// Starts a process in the group and reads the group's process count.
fn count(group: &Group) -> io::Result<()> {
    let mut child = group.command("exec sleep 30").spawn()?;

    while group.get("pids.current")? != "1" {
        thread::sleep(Duration::from_millis(1));
    }

    child.kill()?;
    child.wait()?;

    println!("a process moved into the new group: pids.current 1");

    Ok(())
}

/// Sets `pids.max` to 4 and runs a shell that tries to start 6 background
/// commands and prints a line for each one it started.
fn limit_pids(group: &Group) -> io::Result<()> {
    group.set("pids.max", "4")?;

    let script = "for i in 1 2 3 4 5 6; do sleep 1 & echo started; done 2>/dev/null";
    let output = group.command(script).output()?;
    let started = String::from_utf8_lossy(&output.stdout)
        .matches("started")
        .count();

    println!(
        "pids.max 4, a shell starts background commands: started {started}, forks refused {}",
        group.event("pids.events", "max")?
    );

    group.set("pids.max", "max")
}

/// Sets `memory.max` to 32 MiB with no swap and runs a command whose memory
/// grows without bound.
fn limit_memory(group: &Group) -> io::Result<()> {
    group.set("memory.max", "33554432")?;
    group.set("memory.swap.max", "0")?;

    let status = group.command("exec tail /dev/zero").status()?;
    let kills = group.event("memory.events", "oom_kill")?;

    if let Some(signal) = status.signal() {
        println!(
            "memory.max 32 MiB, a command that grows without bound: killed by signal {signal}, oom_kill {kills}"
        );
    }

    Ok(())
}

fn main() -> io::Result<()> {
    let group = Group::create()?;

    let result = count(&group)
        .and_then(|()| limit_pids(&group))
        .and_then(|()| limit_memory(&group));

    group.remove()?;

    result
}

Listing 12.25 The complete cgmodel, in Go. code/go/cgmodel/main.go

// Command cgmodel models the control group hierarchy: every group counts the
// processes and the memory of its whole subtree, a fork is refused when any
// group above it is at its limit, and a group that exceeds its memory limit
// loses the largest process in its subtree.
//
// Run it from code/go with:
//
//	go run ./cgmodel
package main

import (
	"errors"
	"fmt"
	"math"
)


// unlimited is the value of a limit that is written "max" in a control file.
const unlimited = math.MaxInt

var errAgain = errors.New("EAGAIN")

// A group is one node of the hierarchy. pids and memory are the usage of the
// group and of every group below it. refused and oomKills are the counters
// the kernel publishes in pids.events and memory.events.
type group struct {
	name      string
	parent    *group
	children  []*group
	processes []*process

	pidsMax, memoryMax int
	pids, memory       int
	refused, oomKills  int
}

// A process belongs to one group and holds an amount of memory.
type process struct {
	name   string
	home   *group
	memory int
}

// child adds a group below g, with no limits of its own.
func (g *group) child(name string) *group {
	c := &group{name: name, parent: g, pidsMax: unlimited, memoryMax: unlimited}
	g.children = append(g.children, c)

	return c
}


// spawn starts a process in g. The process is counted in g and in every
// group above it, so it is refused when any of them is at its limit.
func (g *group) spawn(name string) (*process, error) {
	for up := g; up != nil; up = up.parent {
		if up.pids == up.pidsMax {
			up.refused++

			return nil, errAgain
		}
	}

	for up := g; up != nil; up = up.parent {
		up.pids++
	}

	p := &process{name: name, home: g}
	g.processes = append(g.processes, p)

	return p, nil
}


// charge adds an amount, which may be negative, to the memory count of a
// group and of every group above it.
func (g *group) charge(amount int) {
	for up := g; up != nil; up = up.parent {
		up.memory += amount
	}
}

// allocate gives a process more memory. While a group above the process is
// over its limit, the largest process in that group's subtree is killed. It
// returns the names of the processes killed.
func (p *process) allocate(amount int) []string {
	p.memory += amount
	p.home.charge(amount)

	var killed []string

	for up := p.home; up != nil; up = up.parent {
		for up.memory > up.memoryMax {
			victim := up.largest()
			up.oomKills++
			killed = append(killed, victim.name)
			victim.kill()
		}
	}

	return killed
}

// largest returns the process with the most memory in the subtree of g.
func (g *group) largest() *process {
	var found *process

	for _, p := range g.processes {
		if found == nil || p.memory > found.memory {
			found = p
		}
	}

	for _, c := range g.children {
		if p := c.largest(); p != nil && (found == nil || p.memory > found.memory) {
			found = p
		}
	}

	return found
}

// kill removes a process from its group and returns its memory and its
// place in the process count.
func (p *process) kill() {
	p.home.charge(-p.memory)

	for up := p.home; up != nil; up = up.parent {
		up.pids--
	}

	for i, other := range p.home.processes {
		if other == p {
			p.home.processes = append(p.home.processes[:i], p.home.processes[i+1:]...)

			break
		}
	}
}


func main() {
	team := &group{name: "team", pidsMax: 4, memoryMax: 100}
	jobA, jobB := team.child("job-a"), team.child("job-b")

	var started []*process

	for _, start := range []struct {
		in   *group
		name string
	}{{jobA, "a1"}, {jobA, "a2"}, {jobA, "a3"}, {jobB, "b1"}} {
		p, err := start.in.spawn(start.name)

		if err != nil {
			fmt.Println(err)

			return
		}

		started = append(started, p)
	}

	fmt.Printf("3 processes in job-a and 1 in job-b: pids.current job-a %d, job-b %d, team %d\n",
		jobA.pids, jobB.pids, team.pids)

	_, err := jobB.spawn("b2")
	fmt.Printf("another process in job-b, team at pids.max 4: %v, pids.events max of team %d\n", err, team.refused)

	a1, b1 := started[0], started[3]

	a1.allocate(60)
	fmt.Printf("a1 allocates 60: memory.current job-a %d, job-b %d, team %d\n", jobA.memory, jobB.memory, team.memory)

	killed := b1.allocate(50)
	fmt.Printf("b1 allocates 50, team at memory.max 100: killed %v, oom_kill of team %d\n", killed, team.oomKills)
	fmt.Printf("afterwards: memory.current job-a %d, job-b %d, team %d, pids.current team %d\n",
		jobA.memory, jobB.memory, team.memory, team.pids)
}

Listing 12.26 The complete cgmodel, in Rust. code/rust/cgmodel/src/main.rs

//! Models the control group hierarchy: every group counts the processes and
//! the memory of its whole subtree, a fork is refused when any group above
//! it is at its limit, and a group that exceeds its memory limit loses the
//! largest process in its subtree.
//!
//! Run it from code/rust with:
//!
//! ```text
//! cargo run -p cgmodel
//! ```

use std::fmt;

/// The value of a limit that is written "max" in a control file.
const UNLIMITED: usize = usize::MAX;

#[derive(Debug)]
struct Again;

impl fmt::Display for Again {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("EAGAIN")
    }
}

/// One node of the hierarchy. `pids` and `memory` are the usage of the group
/// and of every group below it. `refused` and `oom_kills` are the counters
/// the kernel publishes in `pids.events` and `memory.events`.
struct Group {
    parent: Option<usize>,
    pids_max: usize,
    memory_max: usize,
    pids: usize,
    memory: usize,
    refused: usize,
    oom_kills: usize,
}

/// A process belongs to one group and holds an amount of memory.
struct Process {
    name: &'static str,
    home: usize,
    memory: usize,
    alive: bool,
}

/// The hierarchy: groups and processes, referred to by index.
#[derive(Default)]
struct Hierarchy {
    groups: Vec<Group>,
    processes: Vec<Process>,
}

impl Hierarchy {
    /// Adds a group below `parent` and returns its index.
    fn group(&mut self, parent: Option<usize>, pids_max: usize, memory_max: usize) -> usize {
        self.groups.push(Group {
            parent,
            pids_max,
            memory_max,
            pids: 0,
            memory: 0,
            refused: 0,
            oom_kills: 0,
        });

        self.groups.len() - 1
    }

    /// The group itself and every group above it, nearest first.
    fn upward(&self, group: usize) -> Vec<usize> {
        let mut path = vec![group];

        while let Some(parent) = self.groups[path[path.len() - 1]].parent {
            path.push(parent);
        }

        path
    }
}

impl Hierarchy {
    /// Starts a process in a group and returns its index. The process is
    /// counted in the group and in every group above it, so it is refused
    /// when any of them is at its limit.
    fn spawn(&mut self, group: usize, name: &'static str) -> Result<usize, Again> {
        let path = self.upward(group);

        for &up in &path {
            if self.groups[up].pids == self.groups[up].pids_max {
                self.groups[up].refused += 1;

                return Err(Again);
            }
        }

        for &up in &path {
            self.groups[up].pids += 1;
        }

        self.processes.push(Process {
            name,
            home: group,
            memory: 0,
            alive: true,
        });

        Ok(self.processes.len() - 1)
    }
}

impl Hierarchy {
    /// Gives a process more memory. While a group above the process is over
    /// its limit, the largest process in that group's subtree is killed.
    /// Returns the names of the processes killed.
    fn allocate(&mut self, process: usize, amount: usize) -> Vec<&'static str> {
        let path = self.upward(self.processes[process].home);

        self.processes[process].memory += amount;

        for &up in &path {
            self.groups[up].memory += amount;
        }

        let mut killed = Vec::new();

        for &up in &path {
            while self.groups[up].memory > self.groups[up].memory_max {
                let Some(victim) = self.largest(up) else {
                    break;
                };

                self.groups[up].oom_kills += 1;
                killed.push(self.processes[victim].name);
                self.kill(victim);
            }
        }

        killed
    }

    /// Returns the living process with the most memory in the subtree of a
    /// group.
    fn largest(&self, group: usize) -> Option<usize> {
        (0..self.processes.len())
            .filter(|&p| self.processes[p].alive)
            .filter(|&p| self.upward(self.processes[p].home).contains(&group))
            .max_by_key(|&p| self.processes[p].memory)
    }

    /// Ends a process and returns its memory and its place in the process
    /// count.
    fn kill(&mut self, process: usize) {
        let memory = self.processes[process].memory;

        for up in self.upward(self.processes[process].home) {
            self.groups[up].memory -= memory;
            self.groups[up].pids -= 1;
        }

        self.processes[process].alive = false;
    }
}

fn main() -> Result<(), Again> {
    let mut tree = Hierarchy::default();

    let team = tree.group(None, 4, 100);
    let job_a = tree.group(Some(team), UNLIMITED, UNLIMITED);
    let job_b = tree.group(Some(team), UNLIMITED, UNLIMITED);

    let a1 = tree.spawn(job_a, "a1")?;
    tree.spawn(job_a, "a2")?;
    tree.spawn(job_a, "a3")?;
    let b1 = tree.spawn(job_b, "b1")?;

    println!(
        "3 processes in job-a and 1 in job-b: pids.current job-a {}, job-b {}, team {}",
        tree.groups[job_a].pids, tree.groups[job_b].pids, tree.groups[team].pids
    );

    if let Err(error) = tree.spawn(job_b, "b2") {
        println!(
            "another process in job-b, team at pids.max 4: {error}, pids.events max of team {}",
            tree.groups[team].refused
        );
    }

    tree.allocate(a1, 60);
    println!(
        "a1 allocates 60: memory.current job-a {}, job-b {}, team {}",
        tree.groups[job_a].memory, tree.groups[job_b].memory, tree.groups[team].memory
    );

    let killed = tree.allocate(b1, 50);
    println!(
        "b1 allocates 50, team at memory.max 100: killed [{}], oom_kill of team {}",
        killed.join(" "),
        tree.groups[team].oom_kills
    );
    println!(
        "afterwards: memory.current job-a {}, job-b {}, team {}, pids.current team {}",
        tree.groups[job_a].memory,
        tree.groups[job_b].memory,
        tree.groups[team].memory,
        tree.groups[team].pids
    );

    Ok(())
}

References

• cgroups(7), Linux manual page: https://man7.org/linux/man-pages/man7/cgroups.7.html
• Control Group v2, Linux kernel documentation: https://www.kernel.org/doc/html/v5.15/admin-guide/cgroup-v2.html
• kernel/cgroup/pids.c and mm/memcontrol.c, Linux 5.15: https://elixir.bootlin.com/linux/v5.15/source/kernel/cgroup/pids.c