Building a Container

A container is an ordinary process that runs with a restricted view of the system and with limits on what it may use. The kernel has no object called a container. A container runtime builds one from the mechanisms of the previous chapters: namespaces for the view, a root file system for the files, and a control group for the limits.

In this chapter, we will write box, a program that starts a shell in a container. It needs no privileges and no container software. We will build it in stages: the control group, the creation of the first process in new namespaces, the root file system, and the final exec.

The parts

A mount namespace holds the list of mounted file systems that its processes see. A process in a new mount namespace starts with a copy of its parent’s list and can then mount and unmount without any effect outside. box uses that freedom to assemble a directory tree for the container and to make it the root.

A real container image supplies a complete root file system, with its own programs and libraries. box has no image. It borrows the program directories of the host, which keeps the example self-contained, and exposes nothing else: the home directories, /etc and /tmp of the host do not exist inside.

Mounts and the root directory

A mount attaches a file system at a directory, the mount point. The files of the file system then appear below that directory. What a process sees at / is the mount at the top of its mount namespace, and a path such as /usr/bin/ls is resolved by walking down from there, crossing into another file system at each mount point.

box needs 3 kinds of mount, all made with the mount system call.

The root directory is changed with pivot_root, in 3 steps.

An older call, chroot, also changes what a process sees as /. It changes only the starting point of path lookup and leaves the host’s file system mounted, and a process with privilege can get out of it. pivot_root followed by the unmount removes the host’s tree from the mount namespace altogether, which is why container runtimes use it.

Setting up

box uses golang.org/x/sys/unix in Go and the nix crate with 5 features in Rust.

$ cargo new box
$ cargo add -p box nix --features fs,hostname,mount,sched,user

The control group

createGroup is the code of chapter 12: it finds the nearest group that passes on the pids controller and creates a directory below it. The group is created before the container, so that the first process of the container can start inside it.

Listing 13.1 The imports of box, in Go. code/go/box/main.go

import (
	"bufio"
	"errors"
	"fmt"
	"log"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"syscall"
	"time"

	"golang.org/x/sys/unix"
)

Listing 13.2 The imports of box, in Rust. code/rust/box/src/main.rs

use std::env;
use std::fs;
use std::io;
use std::os::unix::fs::symlink;
use std::os::unix::process::CommandExt;
use std::path::{Path, PathBuf};
use std::process::{self, Command};
use std::thread;
use std::time::Duration;

use nix::mount::{MntFlags, MsFlags, mount, umount2};
use nix::sched::{CloneFlags, unshare};
use nix::unistd::{chdir, getgid, gethostname, getuid, pivot_root, sethostname};

Listing 13.3 Creating the control group of the container, in Go. code/go/box/main.go

// createGroup makes a control group for the container below the nearest
// group that passes the pids controller to its children, and sets pids.max.
func createGroup(pidsMax int) (string, error) {
	line, err := os.ReadFile("/proc/self/cgroup")

	if err != nil {
		return "", err
	}

	_, path, _ := strings.Cut(strings.TrimSpace(string(line)), "::")

	for dir := filepath.Join("/sys/fs/cgroup", path); dir != "/sys/fs/cgroup"; dir = filepath.Dir(dir) {
		enabled, err := os.ReadFile(filepath.Join(dir, "cgroup.subtree_control"))

		if err != nil || !strings.Contains(string(enabled), "pids") {
			continue
		}

		group := filepath.Join(dir, fmt.Sprintf("box-%d", os.Getpid()))

		if err := os.Mkdir(group, 0o755); err != nil {
			return "", err
		}

		return group, os.WriteFile(filepath.Join(group, "pids.max"), fmt.Append(nil, pidsMax), 0)
	}

	return "", errors.New("no parent group passes on the pids controller")
}

// refused returns the number of forks the group has refused.
func refused(group string) (string, error) {
	events, err := os.Open(filepath.Join(group, "pids.events"))

	if err != nil {
		return "", err
	}

	defer events.Close()

	for lines := bufio.NewScanner(events); lines.Scan(); {
		if count, ok := strings.CutPrefix(lines.Text(), "max "); ok {
			return count, nil
		}
	}

	return "", errors.New("pids.events has no line max")
}

// removeGroup waits until the group is empty and deletes it.
func removeGroup(group string) error {
	for {
		current, err := os.ReadFile(filepath.Join(group, "pids.current"))

		if err != nil {
			return err
		}

		if strings.TrimSpace(string(current)) == "0" {
			return os.Remove(group)
		}

		time.Sleep(time.Millisecond)
	}
}

Listing 13.4 Creating the control group of the container, in Rust. code/rust/box/src/main.rs

/// Makes a control group for the container below the nearest group that
/// passes the pids controller to its children, and sets `pids.max`.
fn create_group(pids_max: u32) -> io::Result<PathBuf> {
    let line = fs::read_to_string("/proc/self/cgroup")?;
    let path = line.trim().split_once("::").map_or("", |(_, path)| path);

    let root = Path::new("/sys/fs/cgroup");
    let own = root.join(path.trim_start_matches('/'));

    for dir in own.ancestors().take_while(|dir| *dir != root) {
        let Ok(enabled) = fs::read_to_string(dir.join("cgroup.subtree_control")) else {
            continue;
        };

        if enabled.contains("pids") {
            let group = dir.join(format!("box-{}", process::id()));

            fs::create_dir(&group)?;
            fs::write(group.join("pids.max"), pids_max.to_string())?;

            return Ok(group);
        }
    }

    Err(io::Error::other(
        "no parent group passes on the pids controller",
    ))
}

/// Returns the number of forks the group has refused.
fn refused(group: &Path) -> io::Result<String> {
    fs::read_to_string(group.join("pids.events"))?
        .lines()
        .find_map(|line| line.strip_prefix("max "))
        .map(str::to_owned)
        .ok_or_else(|| io::Error::other("pids.events has no line max"))
}

/// Waits until the group is empty and deletes it.
fn remove_group(group: &Path) -> io::Result<()> {
    while fs::read_to_string(group.join("pids.current"))?.trim() != "0" {
        thread::sleep(Duration::from_millis(1));
    }

    fs::remove_dir(group)
}

The first process

box starts itself again with the argument init, in new user, mount, UTS, PID, IPC and network namespaces. As in chapter 11, the languages differ in how the namespaces are created.

The Go version passes everything to the call that creates the child. Cloneflags names the namespaces, the mappings make the current user user 0 inside, and CgroupFD asks the kernel to create the child directly in the control group, so there is no interval in which the child runs outside it.

The Rust version starts a helper process that calls unshare and writes the mappings, and the helper starts the first process. That process then moves itself into the group by writing 0, which stands for the writing process, to cgroup.procs.

Listing 13.5 Starting the first process of the container, in Go. code/go/box/main.go

// launch starts this program again as the first process of a container. The
// clone flags create the namespaces, the mappings make the current user
// user 0 inside, and CgroupFD makes the kernel create the child inside the
// control group.
func launch(group, root, script string) (*exec.Cmd, error) {
	directory, err := os.Open(group)

	if err != nil {
		return nil, err
	}

	defer directory.Close()

	child := exec.Command("/proc/self/exe", "init", root, script)
	child.Stdout, child.Stderr = os.Stdout, os.Stderr

	child.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWUSER | syscall.CLONE_NEWNS | syscall.CLONE_NEWUTS |
			syscall.CLONE_NEWPID | syscall.CLONE_NEWIPC | syscall.CLONE_NEWNET,
		UidMappings: []syscall.SysProcIDMap{{ContainerID: 0, HostID: os.Getuid(), Size: 1}},
		GidMappings: []syscall.SysProcIDMap{{ContainerID: 0, HostID: os.Getgid(), Size: 1}},
		UseCgroupFD: true,
		CgroupFD:    int(directory.Fd()),
	}

	return child, child.Start()
}

Listing 13.6 Starting the first process of the container, in Rust. code/rust/box/src/main.rs

/// Runs in a helper process. It moves itself into new namespaces, maps the
/// current user to user 0 inside, and starts the first process of the
/// container. The new PID namespace takes effect for that child.
fn enter(group: &str, root: &str, script: &str) -> io::Result<()> {
    let (uid, gid) = (getuid(), getgid());

    unshare(
        CloneFlags::CLONE_NEWUSER
            | CloneFlags::CLONE_NEWNS
            | CloneFlags::CLONE_NEWUTS
            | CloneFlags::CLONE_NEWPID
            | CloneFlags::CLONE_NEWIPC
            | CloneFlags::CLONE_NEWNET,
    )?;

    fs::write("/proc/self/setgroups", "deny")?;
    fs::write("/proc/self/uid_map", format!("0 {uid} 1"))?;
    fs::write("/proc/self/gid_map", format!("0 {gid} 1"))?;

    let status = Command::new("/proc/self/exe")
        .args(["init", group, root, script])
        .status()?;

    let Some(code) = status.code() else {
        process::exit(1);
    };

    process::exit(code);
}

The root file system

buildRoot runs inside the new mount namespace. Its first call sets the propagation of every mount to private. Without it, mounts made here could be copied back to the host’s namespace, depending on how the host is configured.

It then mounts a tmpfs, a file system held in memory, on an empty directory. This is the future root. It makes the host’s program directories visible below it with bind mounts: a bind mount shows an existing directory at a second place. The container also receives the device /dev/null, which shell scripts expect.

The last mount is a new /proc. The /proc of the host describes the host’s PID namespace. A proc file system mounted from inside the new PID namespace describes that namespace, and shows the container only its own processes.

enterRoot calls pivot_root, which makes the new directory the root of the mount namespace and attaches the previous root at a directory inside it. Unmounting that directory removes the last path from the container to the host’s files.

Listing 13.7 Building the root file system, and entering it, in Go. code/go/box/main.go

// shared are the directories of the host that the container can see.
var shared = []string{"bin", "sbin", "lib", "usr"}

// buildRoot assembles the root file system of the container in an empty
// directory: a tmpfs, the host's program directories, the null device, and
// a new /proc.
func buildRoot(root string) error {
	// Mount changes made from here on stay in this mount namespace.
	if err := unix.Mount("", "/", "", unix.MS_REC|unix.MS_PRIVATE, ""); err != nil {
		return fmt.Errorf("make mounts private: %w", err)
	}

	if err := unix.Mount("tmpfs", root, "tmpfs", 0, ""); err != nil {
		return fmt.Errorf("mount tmpfs: %w", err)
	}

	for _, name := range shared {
		source, target := "/"+name, filepath.Join(root, name)

		// On this system /bin, /sbin and /lib are symbolic links into /usr.
		if link, err := os.Readlink(source); err == nil {
			if err := os.Symlink(link, target); err != nil {
				return err
			}

			continue
		}

		if err := os.Mkdir(target, 0o755); err != nil {
			return err
		}

		if err := unix.Mount(source, target, "", unix.MS_BIND|unix.MS_REC, ""); err != nil {
			return fmt.Errorf("bind %s: %w", source, err)
		}
	}

	// The container gets a /dev with the null device and nothing else.
	null := filepath.Join(root, "dev", "null")

	if err := os.Mkdir(filepath.Dir(null), 0o755); err != nil {
		return err
	}

	if err := os.WriteFile(null, nil, 0o666); err != nil {
		return err
	}

	if err := unix.Mount("/dev/null", null, "", unix.MS_BIND, ""); err != nil {
		return fmt.Errorf("bind /dev/null: %w", err)
	}

	proc := filepath.Join(root, "proc")

	if err := os.Mkdir(proc, 0o555); err != nil {
		return err
	}

	if err := unix.Mount("proc", proc, "proc", 0, ""); err != nil {
		return fmt.Errorf("mount proc: %w", err)
	}

	return nil
}

// enterRoot makes root the root directory of the mount namespace and
// detaches the host's file system from it.
func enterRoot(root string) error {
	old := filepath.Join(root, "old")

	if err := os.Mkdir(old, 0o700); err != nil {
		return err
	}

	if err := unix.PivotRoot(root, old); err != nil {
		return fmt.Errorf("pivot_root: %w", err)
	}

	if err := os.Chdir("/"); err != nil {
		return err
	}

	if err := unix.Unmount("/old", unix.MNT_DETACH); err != nil {
		return fmt.Errorf("unmount the old root: %w", err)
	}

	return os.Remove("/old")
}

Listing 13.8 Building the root file system, and entering it, in Rust. code/rust/box/src/main.rs

/// The directories of the host that the container can see.
const SHARED: [&str; 4] = ["bin", "sbin", "lib", "usr"];

/// No source, file system type or options, for the calls to `mount` that
/// need none.
const NONE: Option<&str> = None;

/// Assembles the root file system of the container in an empty directory: a
/// tmpfs, the host's program directories, the null device, and a new /proc.
fn build_root(root: &Path) -> io::Result<()> {
    // Mount changes made from here on stay in this mount namespace.
    mount(NONE, "/", NONE, MsFlags::MS_REC | MsFlags::MS_PRIVATE, NONE)?;
    mount(Some("tmpfs"), root, Some("tmpfs"), MsFlags::empty(), NONE)?;

    for name in SHARED {
        let (source, target) = (Path::new("/").join(name), root.join(name));

        // On this system /bin, /sbin and /lib are symbolic links into /usr.
        if let Ok(link) = fs::read_link(&source) {
            symlink(link, target)?;

            continue;
        }

        fs::create_dir(&target)?;
        mount(
            Some(&source),
            &target,
            NONE,
            MsFlags::MS_BIND | MsFlags::MS_REC,
            NONE,
        )?;
    }

    // The container gets a /dev with the null device and nothing else.
    let null = root.join("dev/null");

    fs::create_dir(root.join("dev"))?;
    fs::write(&null, "")?;
    mount(Some("/dev/null"), &null, NONE, MsFlags::MS_BIND, NONE)?;

    let proc = root.join("proc");

    fs::create_dir(&proc)?;
    mount(Some("proc"), &proc, Some("proc"), MsFlags::empty(), NONE)?;

    Ok(())
}

/// Makes `root` the root directory of the mount namespace and detaches the
/// host's file system from it.
fn enter_root(root: &Path) -> io::Result<()> {
    let old = root.join("old");

    fs::create_dir(&old)?;
    pivot_root(root, &old)?;
    chdir("/")?;
    umount2("/old", MntFlags::MNT_DETACH)?;

    fs::remove_dir("/old")
}

Becoming the program

initialize runs the steps in order and ends with exec. The process that set up the container is replaced by the shell, which is therefore process 1 of the container.

Listing 13.9 The first process: set up, then exec, in Go. code/go/box/main.go

// initialize runs as process 1 of the container. It builds and enters the
// root file system, sets the host name, and replaces itself with a shell
// that runs the script.
func initialize(root, script string) error {
	if err := buildRoot(root); err != nil {
		return err
	}

	if err := enterRoot(root); err != nil {
		return err
	}

	if err := unix.Sethostname([]byte("box")); err != nil {
		return err
	}

	environment := []string{"PATH=/usr/sbin:/usr/bin:/sbin:/bin"}

	return unix.Exec("/bin/sh", []string{"sh", "-c", script}, environment)
}

Listing 13.10 The first process: set up, then exec, in Rust. code/rust/box/src/main.rs

/// Runs as process 1 of the container. It moves itself into the control
/// group, where writing 0 to `cgroup.procs` means "the writing process".
/// Then it builds and enters the root file system, sets the host name, and
/// replaces itself with a shell that runs the script.
fn initialize(group: &Path, root: &Path, script: &str) -> io::Result<()> {
    fs::write(group.join("cgroup.procs"), "0")?;

    build_root(root)?;
    enter_root(root)?;
    sethostname("box")?;

    let error = Command::new("/bin/sh")
        .args(["-c", script])
        .env_clear()
        .env("PATH", "/usr/sbin:/usr/bin:/sbin:/bin")
        .exec();

    Err(error)
}

run is the parent’s side. The script reports what the shell sees and then tries to start 6 background commands under a pids.max of 5.

Listing 13.11 Running the container, and reporting on it, in Go. code/go/box/main.go

// script is what the container runs.
const script = `echo "host name: $(hostname)"
echo "process number of the shell: $$"
echo "user number: $(id -u)"
echo "processes under /proc:" /proc/[0-9]*
echo "root directory:" /*
while read device directory type rest; do echo "mounted: $directory ($type)"; done < /proc/mounts
echo "network interfaces:" $(tail -n +3 /proc/net/dev | cut -d: -f1)
echo "effective capabilities:" $(grep CapEff /proc/self/status | cut -f2)
for i in 1 2 3 4 5 6; do sleep 1 & echo "started a background command"; done 2>/dev/null`

func run() error {
	group, err := createGroup(5)

	if err != nil {
		return err
	}

	defer removeGroup(group)

	root, err := os.MkdirTemp("", "box-root")

	if err != nil {
		return err
	}

	defer os.Remove(root)

	before, err := os.Hostname()

	if err != nil {
		return err
	}

	child, err := launch(group, root, script)

	if err != nil {
		return err
	}

	child.Wait()

	count, err := refused(group)

	if err != nil {
		return err
	}

	after, err := os.Hostname()

	if err != nil {
		return err
	}

	fmt.Printf("the container ended with status %d; forks refused by its group: %s\n", child.ProcessState.ExitCode(), count)
	fmt.Printf("host name outside unchanged: %v\n", before == after)

	return nil
}

func main() {
	var err error

	if len(os.Args) == 4 && os.Args[1] == "init" {
		err = initialize(os.Args[2], os.Args[3])
	} else {
		err = run()
	}

	if err != nil {
		log.Fatal(err)
	}
}

Listing 13.12 Running the container, and reporting on it, in Rust. code/rust/box/src/main.rs

/// What the container runs.
const SCRIPT: &str = r#"echo "host name: $(hostname)"
echo "process number of the shell: $$"
echo "user number: $(id -u)"
echo "processes under /proc:" /proc/[0-9]*
echo "root directory:" /*
while read device directory type rest; do echo "mounted: $directory ($type)"; done < /proc/mounts
echo "network interfaces:" $(tail -n +3 /proc/net/dev | cut -d: -f1)
echo "effective capabilities:" $(grep CapEff /proc/self/status | cut -f2)
for i in 1 2 3 4 5 6; do sleep 1 & echo "started a background command"; done 2>/dev/null"#;

fn hostname() -> io::Result<String> {
    Ok(gethostname()?.to_string_lossy().into_owned())
}

fn run() -> io::Result<()> {
    let group = create_group(5)?;
    let root = env::temp_dir().join(format!("box-root-{}", process::id()));

    fs::create_dir(&root)?;

    let before = hostname()?;

    let status = Command::new("/proc/self/exe")
        .arg("enter")
        .args([&group, &root])
        .arg(SCRIPT)
        .status()?;

    let Some(code) = status.code() else {
        return Err(io::Error::other("the container was ended by a signal"));
    };

    let count = refused(&group)?;

    remove_group(&group)?;
    fs::remove_dir(&root)?;

    println!("the container ended with status {code}; forks refused by its group: {count}");
    println!("host name outside unchanged: {}", before == hostname()?);

    Ok(())
}

fn main() -> io::Result<()> {
    let args: Vec<String> = env::args().collect();
    let args: Vec<&str> = args.iter().map(String::as_str).collect();

    match args.as_slice() {
        [_, "enter", group, root, script] => enter(group, root, script),
        [_, "init", group, root, script] => initialize(Path::new(group), Path::new(root), script),
        _ => run(),
    }
}

host name: box
process number of the shell: 1
user number: 0
processes under /proc: /proc/1
root directory: /bin /dev /lib /proc /sbin /usr
mounted: / (tmpfs)
mounted: /usr (ext4)
mounted: /dev/null (devtmpfs)
mounted: /proc (proc)
network interfaces: lo
effective capabilities: 000001ffffffffff
started a background command
started a background command
started a background command
started a background command
the container ended with status 2; forks refused by its group: 1
host name outside unchanged: true

The table states the central fact about containers. A container is a view. The process inside is an ordinary process of the host, scheduled by the same scheduler and using the same memory, and the kernel answers its questions from a restricted set of objects.

When the shell exits, 4 sleep commands are still running. The shell was process 1 of its PID namespace, and when the first process of a PID namespace ends, the kernel ends every other process in it. No process of the container survives its first process, which is why box can remove the control group as soon as its count reaches 0.

Images and runtimes

box borrows /usr from the host. A real container brings its own files, as an image: an archive of a root file system, built in layers.

The layers are combined by an overlay file system, overlayfs. It is mounted with a list of read-only directories, the lower layers, and a single writable directory, the upper layer, and it shows their union. A file is looked up in the upper layer first and then in each lower layer. A write to a file from a lower layer first copies the file to the upper layer. The image is not modified, so many containers share its layers, and each container’s changes are confined to its own upper layer. In terms of this chapter, a runtime replaces the tmpfs and the bind mounts of buildRoot with a single overlay mount, and everything after that is the same.

The software that people call “Docker” is several programs in a chain, and only the last makes the system calls.

The interface between the engine and the runtime is standardised by the Open Container Initiative. The engine unpacks the image and writes a file, config.json, that lists the namespaces to create, the mounts to make, the limits to set, and the command to run. The runtime carries it out. box is a runtime with its configuration fixed in the source.

Inspecting a container

A container is a set of ordinary processes, so every tool of the earlier chapters applies to it from the host.

The complete program

Listing 13.13 The complete box, in Go. code/go/box/main.go

// Command box runs a shell script in a container built by hand: new user,
// mount, UTS, PID, IPC and network namespaces, a root file system of its own, and a
// control group with a limit on the number of processes.
//
// Run it from code/go with:
//
//	go run ./box
package main

import (
	"bufio"
	"errors"
	"fmt"
	"log"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"syscall"
	"time"

	"golang.org/x/sys/unix"
)


// createGroup makes a control group for the container below the nearest
// group that passes the pids controller to its children, and sets pids.max.
func createGroup(pidsMax int) (string, error) {
	line, err := os.ReadFile("/proc/self/cgroup")

	if err != nil {
		return "", err
	}

	_, path, _ := strings.Cut(strings.TrimSpace(string(line)), "::")

	for dir := filepath.Join("/sys/fs/cgroup", path); dir != "/sys/fs/cgroup"; dir = filepath.Dir(dir) {
		enabled, err := os.ReadFile(filepath.Join(dir, "cgroup.subtree_control"))

		if err != nil || !strings.Contains(string(enabled), "pids") {
			continue
		}

		group := filepath.Join(dir, fmt.Sprintf("box-%d", os.Getpid()))

		if err := os.Mkdir(group, 0o755); err != nil {
			return "", err
		}

		return group, os.WriteFile(filepath.Join(group, "pids.max"), fmt.Append(nil, pidsMax), 0)
	}

	return "", errors.New("no parent group passes on the pids controller")
}

// refused returns the number of forks the group has refused.
func refused(group string) (string, error) {
	events, err := os.Open(filepath.Join(group, "pids.events"))

	if err != nil {
		return "", err
	}

	defer events.Close()

	for lines := bufio.NewScanner(events); lines.Scan(); {
		if count, ok := strings.CutPrefix(lines.Text(), "max "); ok {
			return count, nil
		}
	}

	return "", errors.New("pids.events has no line max")
}

// removeGroup waits until the group is empty and deletes it.
func removeGroup(group string) error {
	for {
		current, err := os.ReadFile(filepath.Join(group, "pids.current"))

		if err != nil {
			return err
		}

		if strings.TrimSpace(string(current)) == "0" {
			return os.Remove(group)
		}

		time.Sleep(time.Millisecond)
	}
}


// launch starts this program again as the first process of a container. The
// clone flags create the namespaces, the mappings make the current user
// user 0 inside, and CgroupFD makes the kernel create the child inside the
// control group.
func launch(group, root, script string) (*exec.Cmd, error) {
	directory, err := os.Open(group)

	if err != nil {
		return nil, err
	}

	defer directory.Close()

	child := exec.Command("/proc/self/exe", "init", root, script)
	child.Stdout, child.Stderr = os.Stdout, os.Stderr

	child.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWUSER | syscall.CLONE_NEWNS | syscall.CLONE_NEWUTS |
			syscall.CLONE_NEWPID | syscall.CLONE_NEWIPC | syscall.CLONE_NEWNET,
		UidMappings: []syscall.SysProcIDMap{{ContainerID: 0, HostID: os.Getuid(), Size: 1}},
		GidMappings: []syscall.SysProcIDMap{{ContainerID: 0, HostID: os.Getgid(), Size: 1}},
		UseCgroupFD: true,
		CgroupFD:    int(directory.Fd()),
	}

	return child, child.Start()
}


// shared are the directories of the host that the container can see.
var shared = []string{"bin", "sbin", "lib", "usr"}

// buildRoot assembles the root file system of the container in an empty
// directory: a tmpfs, the host's program directories, the null device, and
// a new /proc.
func buildRoot(root string) error {
	// Mount changes made from here on stay in this mount namespace.
	if err := unix.Mount("", "/", "", unix.MS_REC|unix.MS_PRIVATE, ""); err != nil {
		return fmt.Errorf("make mounts private: %w", err)
	}

	if err := unix.Mount("tmpfs", root, "tmpfs", 0, ""); err != nil {
		return fmt.Errorf("mount tmpfs: %w", err)
	}

	for _, name := range shared {
		source, target := "/"+name, filepath.Join(root, name)

		// On this system /bin, /sbin and /lib are symbolic links into /usr.
		if link, err := os.Readlink(source); err == nil {
			if err := os.Symlink(link, target); err != nil {
				return err
			}

			continue
		}

		if err := os.Mkdir(target, 0o755); err != nil {
			return err
		}

		if err := unix.Mount(source, target, "", unix.MS_BIND|unix.MS_REC, ""); err != nil {
			return fmt.Errorf("bind %s: %w", source, err)
		}
	}

	// The container gets a /dev with the null device and nothing else.
	null := filepath.Join(root, "dev", "null")

	if err := os.Mkdir(filepath.Dir(null), 0o755); err != nil {
		return err
	}

	if err := os.WriteFile(null, nil, 0o666); err != nil {
		return err
	}

	if err := unix.Mount("/dev/null", null, "", unix.MS_BIND, ""); err != nil {
		return fmt.Errorf("bind /dev/null: %w", err)
	}

	proc := filepath.Join(root, "proc")

	if err := os.Mkdir(proc, 0o555); err != nil {
		return err
	}

	if err := unix.Mount("proc", proc, "proc", 0, ""); err != nil {
		return fmt.Errorf("mount proc: %w", err)
	}

	return nil
}

// enterRoot makes root the root directory of the mount namespace and
// detaches the host's file system from it.
func enterRoot(root string) error {
	old := filepath.Join(root, "old")

	if err := os.Mkdir(old, 0o700); err != nil {
		return err
	}

	if err := unix.PivotRoot(root, old); err != nil {
		return fmt.Errorf("pivot_root: %w", err)
	}

	if err := os.Chdir("/"); err != nil {
		return err
	}

	if err := unix.Unmount("/old", unix.MNT_DETACH); err != nil {
		return fmt.Errorf("unmount the old root: %w", err)
	}

	return os.Remove("/old")
}


// initialize runs as process 1 of the container. It builds and enters the
// root file system, sets the host name, and replaces itself with a shell
// that runs the script.
func initialize(root, script string) error {
	if err := buildRoot(root); err != nil {
		return err
	}

	if err := enterRoot(root); err != nil {
		return err
	}

	if err := unix.Sethostname([]byte("box")); err != nil {
		return err
	}

	environment := []string{"PATH=/usr/sbin:/usr/bin:/sbin:/bin"}

	return unix.Exec("/bin/sh", []string{"sh", "-c", script}, environment)
}


// script is what the container runs.
const script = `echo "host name: $(hostname)"
echo "process number of the shell: $$"
echo "user number: $(id -u)"
echo "processes under /proc:" /proc/[0-9]*
echo "root directory:" /*
while read device directory type rest; do echo "mounted: $directory ($type)"; done < /proc/mounts
echo "network interfaces:" $(tail -n +3 /proc/net/dev | cut -d: -f1)
echo "effective capabilities:" $(grep CapEff /proc/self/status | cut -f2)
for i in 1 2 3 4 5 6; do sleep 1 & echo "started a background command"; done 2>/dev/null`

func run() error {
	group, err := createGroup(5)

	if err != nil {
		return err
	}

	defer removeGroup(group)

	root, err := os.MkdirTemp("", "box-root")

	if err != nil {
		return err
	}

	defer os.Remove(root)

	before, err := os.Hostname()

	if err != nil {
		return err
	}

	child, err := launch(group, root, script)

	if err != nil {
		return err
	}

	child.Wait()

	count, err := refused(group)

	if err != nil {
		return err
	}

	after, err := os.Hostname()

	if err != nil {
		return err
	}

	fmt.Printf("the container ended with status %d; forks refused by its group: %s\n", child.ProcessState.ExitCode(), count)
	fmt.Printf("host name outside unchanged: %v\n", before == after)

	return nil
}

func main() {
	var err error

	if len(os.Args) == 4 && os.Args[1] == "init" {
		err = initialize(os.Args[2], os.Args[3])
	} else {
		err = run()
	}

	if err != nil {
		log.Fatal(err)
	}
}

Listing 13.14 The complete box, in Rust. code/rust/box/src/main.rs

//! Runs a shell script in a container built by hand: new user, mount, UTS,
//! PID, IPC and network namespaces, a root file system of its own, and a control group
//! with a limit on the number of processes.
//!
//! Run it from code/rust with:
//!
//! ```text
//! cargo run -p box
//! ```

use std::env;
use std::fs;
use std::io;
use std::os::unix::fs::symlink;
use std::os::unix::process::CommandExt;
use std::path::{Path, PathBuf};
use std::process::{self, Command};
use std::thread;
use std::time::Duration;

use nix::mount::{MntFlags, MsFlags, mount, umount2};
use nix::sched::{CloneFlags, unshare};
use nix::unistd::{chdir, getgid, gethostname, getuid, pivot_root, sethostname};

/// Makes a control group for the container below the nearest group that
/// passes the pids controller to its children, and sets `pids.max`.
fn create_group(pids_max: u32) -> io::Result<PathBuf> {
    let line = fs::read_to_string("/proc/self/cgroup")?;
    let path = line.trim().split_once("::").map_or("", |(_, path)| path);

    let root = Path::new("/sys/fs/cgroup");
    let own = root.join(path.trim_start_matches('/'));

    for dir in own.ancestors().take_while(|dir| *dir != root) {
        let Ok(enabled) = fs::read_to_string(dir.join("cgroup.subtree_control")) else {
            continue;
        };

        if enabled.contains("pids") {
            let group = dir.join(format!("box-{}", process::id()));

            fs::create_dir(&group)?;
            fs::write(group.join("pids.max"), pids_max.to_string())?;

            return Ok(group);
        }
    }

    Err(io::Error::other(
        "no parent group passes on the pids controller",
    ))
}

/// Returns the number of forks the group has refused.
fn refused(group: &Path) -> io::Result<String> {
    fs::read_to_string(group.join("pids.events"))?
        .lines()
        .find_map(|line| line.strip_prefix("max "))
        .map(str::to_owned)
        .ok_or_else(|| io::Error::other("pids.events has no line max"))
}

/// Waits until the group is empty and deletes it.
fn remove_group(group: &Path) -> io::Result<()> {
    while fs::read_to_string(group.join("pids.current"))?.trim() != "0" {
        thread::sleep(Duration::from_millis(1));
    }

    fs::remove_dir(group)
}

/// Runs in a helper process. It moves itself into new namespaces, maps the
/// current user to user 0 inside, and starts the first process of the
/// container. The new PID namespace takes effect for that child.
fn enter(group: &str, root: &str, script: &str) -> io::Result<()> {
    let (uid, gid) = (getuid(), getgid());

    unshare(
        CloneFlags::CLONE_NEWUSER
            | CloneFlags::CLONE_NEWNS
            | CloneFlags::CLONE_NEWUTS
            | CloneFlags::CLONE_NEWPID
            | CloneFlags::CLONE_NEWIPC
            | CloneFlags::CLONE_NEWNET,
    )?;

    fs::write("/proc/self/setgroups", "deny")?;
    fs::write("/proc/self/uid_map", format!("0 {uid} 1"))?;
    fs::write("/proc/self/gid_map", format!("0 {gid} 1"))?;

    let status = Command::new("/proc/self/exe")
        .args(["init", group, root, script])
        .status()?;

    let Some(code) = status.code() else {
        process::exit(1);
    };

    process::exit(code);
}

/// The directories of the host that the container can see.
const SHARED: [&str; 4] = ["bin", "sbin", "lib", "usr"];

/// No source, file system type or options, for the calls to `mount` that
/// need none.
const NONE: Option<&str> = None;

/// Assembles the root file system of the container in an empty directory: a
/// tmpfs, the host's program directories, the null device, and a new /proc.
fn build_root(root: &Path) -> io::Result<()> {
    // Mount changes made from here on stay in this mount namespace.
    mount(NONE, "/", NONE, MsFlags::MS_REC | MsFlags::MS_PRIVATE, NONE)?;
    mount(Some("tmpfs"), root, Some("tmpfs"), MsFlags::empty(), NONE)?;

    for name in SHARED {
        let (source, target) = (Path::new("/").join(name), root.join(name));

        // On this system /bin, /sbin and /lib are symbolic links into /usr.
        if let Ok(link) = fs::read_link(&source) {
            symlink(link, target)?;

            continue;
        }

        fs::create_dir(&target)?;
        mount(
            Some(&source),
            &target,
            NONE,
            MsFlags::MS_BIND | MsFlags::MS_REC,
            NONE,
        )?;
    }

    // The container gets a /dev with the null device and nothing else.
    let null = root.join("dev/null");

    fs::create_dir(root.join("dev"))?;
    fs::write(&null, "")?;
    mount(Some("/dev/null"), &null, NONE, MsFlags::MS_BIND, NONE)?;

    let proc = root.join("proc");

    fs::create_dir(&proc)?;
    mount(Some("proc"), &proc, Some("proc"), MsFlags::empty(), NONE)?;

    Ok(())
}

/// Makes `root` the root directory of the mount namespace and detaches the
/// host's file system from it.
fn enter_root(root: &Path) -> io::Result<()> {
    let old = root.join("old");

    fs::create_dir(&old)?;
    pivot_root(root, &old)?;
    chdir("/")?;
    umount2("/old", MntFlags::MNT_DETACH)?;

    fs::remove_dir("/old")
}

/// Runs as process 1 of the container. It moves itself into the control
/// group, where writing 0 to `cgroup.procs` means "the writing process".
/// Then it builds and enters the root file system, sets the host name, and
/// replaces itself with a shell that runs the script.
fn initialize(group: &Path, root: &Path, script: &str) -> io::Result<()> {
    fs::write(group.join("cgroup.procs"), "0")?;

    build_root(root)?;
    enter_root(root)?;
    sethostname("box")?;

    let error = Command::new("/bin/sh")
        .args(["-c", script])
        .env_clear()
        .env("PATH", "/usr/sbin:/usr/bin:/sbin:/bin")
        .exec();

    Err(error)
}

/// What the container runs.
const SCRIPT: &str = r#"echo "host name: $(hostname)"
echo "process number of the shell: $$"
echo "user number: $(id -u)"
echo "processes under /proc:" /proc/[0-9]*
echo "root directory:" /*
while read device directory type rest; do echo "mounted: $directory ($type)"; done < /proc/mounts
echo "network interfaces:" $(tail -n +3 /proc/net/dev | cut -d: -f1)
echo "effective capabilities:" $(grep CapEff /proc/self/status | cut -f2)
for i in 1 2 3 4 5 6; do sleep 1 & echo "started a background command"; done 2>/dev/null"#;

fn hostname() -> io::Result<String> {
    Ok(gethostname()?.to_string_lossy().into_owned())
}

fn run() -> io::Result<()> {
    let group = create_group(5)?;
    let root = env::temp_dir().join(format!("box-root-{}", process::id()));

    fs::create_dir(&root)?;

    let before = hostname()?;

    let status = Command::new("/proc/self/exe")
        .arg("enter")
        .args([&group, &root])
        .arg(SCRIPT)
        .status()?;

    let Some(code) = status.code() else {
        return Err(io::Error::other("the container was ended by a signal"));
    };

    let count = refused(&group)?;

    remove_group(&group)?;
    fs::remove_dir(&root)?;

    println!("the container ended with status {code}; forks refused by its group: {count}");
    println!("host name outside unchanged: {}", before == hostname()?);

    Ok(())
}

fn main() -> io::Result<()> {
    let args: Vec<String> = env::args().collect();
    let args: Vec<&str> = args.iter().map(String::as_str).collect();

    match args.as_slice() {
        [_, "enter", group, root, script] => enter(group, root, script),
        [_, "init", group, root, script] => initialize(Path::new(group), Path::new(root), script),
        _ => run(),
    }
}

References

• pivot_root(2), mount_namespaces(7) and clone(2), Linux manual pages: https://man7.org/linux/man-pages/man2/pivot_root.2.html
• Open Container Initiative Runtime Specification: https://github.com/opencontainers/runtime-spec
• runc, the reference runtime: https://github.com/opencontainers/runc
• L. Rice, Container Security, O’Reilly, 2020.