Mapping files and copying less

Part 7 works at the boundary between your program and the kernel. Chapter 2 read files with read, and chapter 15 explained pages and page faults. This chapter joins the two. It maps a file into memory and watches the page faults that load it. Then it sends a file over a socket without copying its bytes into the program at all.

Two programs carry the chapter. mmap_file maps a 64-page file, counts the faults that two passes over it take, and writes through a shared and a private mapping. zero_copy sends a 256 MiB file over a loopback socket twice: once with a read/write loop, and once with sendfile.

23.1 Where a file’s bytes live

The kernel keeps recently used file data in RAM, in the page cache. The cache is organized by file and by page. Page 0 of a file holds bytes 0 to 4,095, and page 1 holds the next 4,096. Every way of reading a file goes through it. A read of a page that is not cached makes the kernel load the page from disk first. A later read of the same page, by any process, finds it in RAM.

The page cache belongs to the kernel, so a read cannot hand your program the cached page itself. It copies the requested bytes into a buffer your program owns. After read(fd, &mut buffer) returns, the file’s bytes sit in RAM twice: once in the page cache, and once in buffer.

A memory mapping removes the second copy. The system call mmap asks the kernel to make a range of a file part of your address space. The kernel reserves a range of virtual addresses and returns a pointer to its start. When the program reads from those addresses, the page table translates them to the page cache’s own frames (figure 23.1). The program reads the file’s bytes where the kernel keeps them.

Figure 23.2 places the mapping in the process. It is one more region of the address space, beside the heap and the stack. Its page table entries point at frames the kernel already holds for the file.

mmap loads nothing when it returns. The page table entries for the range start empty. The first access to each page is a page fault, as in section 15.5. The kernel finds the file’s page in the page cache, or reads it from disk if it is not there. Then it fills in the page table entry and restarts the instruction that faulted. A fault that finds the page already cached is a minor fault, and costs microseconds. A fault that has to read the disk is a major fault, and costs at least a disk access.

Here is the plan for 64 pages:

The rest of this section builds a type for a mapping and checks the table against a real run.

23.1.1 A type that owns a mapping

The program calls mmap through the libc crate. A mapping is a pointer and a length, and it must be unmapped exactly once. That is the shape of an owning Rust type with a Drop:

use std::{
    fs::{self, File},
    io,
    os::fd::AsRawFd,
    path::Path,
    ptr::NonNull,
    slice,
};

/// A file mapped into this process's address space. Unmapped when dropped.
struct Mapping {
    ptr: NonNull<u8>,
    len: usize,
}

NonNull<u8> is a raw pointer that is never null. It records that Mapping always holds a real address. Creating the mapping is one call to mmap. The libc crate declares each C function inside an unsafe extern "C" block, with C’s types. c_int is a C int, size_t is an unsigned size, and off_t is a file offset. *mut c_void is a pointer to bytes of no particular type. Here is the declaration, with what each parameter means:

unsafe extern "C" {
    fn mmap(
        addr: *mut c_void, // where to map; null lets the kernel choose
        len: size_t,       // how many bytes to map
        prot: c_int,       // what is allowed: PROT_READ | PROT_WRITE
        flags: c_int,      // MAP_SHARED or MAP_PRIVATE (section 23.2)
        fd: c_int,         // the open file
        offset: off_t,     // where in the file; a multiple of the page size
    ) -> *mut c_void;      // the mapping's address, or MAP_FAILED
}

Every call through such a declaration is unsafe, because the compiler cannot check what the C code does with the pointers it is given. new makes the call and checks the result:

impl Mapping {
    /// Map the first `len` bytes of `file` for reading and writing.
    ///
    /// `shared` chooses `MAP_SHARED`, whose writes reach the file, over
    /// `MAP_PRIVATE`, whose writes stay in this process.
    fn new(file: &File, len: usize, shared: bool) -> io::Result<Self> {
        let flags = if shared {
            libc::MAP_SHARED
        } else {
            libc::MAP_PRIVATE
        };

        // SAFETY: a null address lets the kernel choose where to place the
        // mapping, and the descriptor is open for reading and writing. The
        // result is checked against MAP_FAILED before it is used.
        let address = unsafe {
            libc::mmap(
                std::ptr::null_mut(),
                len,
                libc::PROT_READ | libc::PROT_WRITE,
                flags,
                file.as_raw_fd(),
                0,
            )
        };
        if address == libc::MAP_FAILED {
            return Err(io::Error::last_os_error());
        }

        let ptr = NonNull::new(address.cast::<u8>()).ok_or_else(io::Error::last_os_error)?;
        Ok(Self { ptr, len })
    }
    // ...
}

On failure mmap returns MAP_FAILED, not null, so the code checks for that value first. last_os_error reads errno and turns it into an io::Error.

The program reaches the bytes through slices. bytes and bytes_mut build them from the pointer and the length:

impl Mapping {
    // ...
    fn bytes(&self) -> &[u8] {
        // SAFETY: the mapping is `len` bytes long and stays valid until drop.
        unsafe { slice::from_raw_parts(self.ptr.as_ptr(), self.len) }
    }

    fn bytes_mut(&mut self) -> &mut [u8] {
        // SAFETY: as in `bytes`, and `&mut self` makes this the only borrow.
        unsafe { slice::from_raw_parts_mut(self.ptr.as_ptr(), self.len) }
    }
    // ...
}

bytes_mut takes &mut self, so the borrow checker allows one mutable slice at a time, as with a Vec. Drop gives the address range back to the kernel with munmap, which takes the same address and length:

unsafe extern "C" {
    fn munmap(addr: *mut c_void, len: size_t) -> c_int; // 0, or -1 with errno set
}

impl Drop for Mapping {
    fn drop(&mut self) {
        // SAFETY: the range is exactly the mapping created in `new`, and no
        // borrow of it can outlive `self`.
        unsafe {
            libc::munmap(self.ptr.as_ptr().cast(), self.len);
        }
    }
}

A slice returned by bytes borrows self, so it cannot outlive the Mapping. Without that borrow, a slice could point into an unmapped range, and the next read through it would crash the process.

23.1.2 Counting the faults

The kernel counts each process’s page faults. getrusage copies the counts into a rusage struct that the caller provides, and sysconf reads system constants such as the page size:

unsafe extern "C" {
    fn getrusage(
        who: c_int,          // RUSAGE_SELF: this process
        usage: *mut rusage,  // filled in; ru_minflt counts minor faults
    ) -> c_int;              // 0, or -1 with errno set
    fn sysconf(name: c_int) -> c_long; // _SC_PAGESIZE gives the page size
}

The program reads the minor-fault count before and after a pass over the mapping:

/// Minor page faults this process has taken so far: faults the kernel served
/// from memory, without reading the disk.
fn minor_faults() -> u64 {
    // SAFETY: `getrusage` fills the struct it is given and nothing else.
    let usage = unsafe {
        let mut usage = std::mem::zeroed::<libc::rusage>();
        libc::getrusage(libc::RUSAGE_SELF, &mut usage);
        usage
    };
    u64::try_from(usage.ru_minflt).expect("fault count is not negative")
}

/// Read one byte from every page, so each page is touched once.
fn touch_every_page(bytes: &[u8], page: usize) -> u64 {
    bytes
        .iter()
        .step_by(page)
        .map(|&byte| u64::from(byte))
        .sum()
}

touch_every_page reads one byte per page. step_by(page) skips from the first byte of one page to the first byte of the next. The function returns the sum of the bytes so the compiler cannot drop the reads as unused.

main writes a file of 64 pages, maps it shared, and runs the pass twice:

fn main() -> io::Result<()> {
    // ...
    let mut shared = Mapping::new(&file, len, true)?;

    let before = minor_faults();
    touch_every_page(shared.bytes(), page);
    let first = minor_faults() - before;

    let before = minor_faults();
    touch_every_page(shared.bytes(), page);
    let second = minor_faults() - before;

    println!("first pass over the mapping:  {first} minor faults");
    println!("second pass over the mapping: {second} minor faults\n");
    // ...
}

Animation 23.1 follows both passes, the writes of section 23.2, and the truncation of section 23.2.1.

On macOS, the run matches the table:

$ cargo run --bin mmap_file
page size 4096 bytes, file 64 pages (262144 bytes)

first pass over the mapping:  64 minor faults
second pass over the mapping: 0 minor faults

The faults are minor because fs::write had put every page of the file in the page cache a moment earlier. No disk read was needed.

On Linux, the same program reports first pass over the mapping: 5 minor faults. When a read faults on a file mapping, Linux also fills the entries of the neighbouring pages that are already in the page cache. This is fault-around, and by default it covers 64 KiB, which is 16 pages, per fault. 64 pages need at least 4 such faults. The second pass is 0 faults on both systems.

23.2 Shared and private mappings

The flags argument decides where a write through the mapping goes (figure 23.3).

• MAP_SHARED: the entry points at the page cache frame itself. A write changes the file’s cached page. Every process that maps or reads the file sees the change. The kernel writes the dirty page to disk later, during writeback.
• MAP_PRIVATE: the entry also points at the page cache frame, but it is marked read-only. The first write to a page faults. The kernel copies the page into a new frame that belongs to this process and points the entry at the copy. This is copy-on-write. The file never sees the write.

Writeback happens on the kernel’s schedule, typically within about 30 seconds. To put the data on disk at a known point, call msync:

unsafe extern "C" {
    fn msync(
        addr: *mut c_void, // start of the range, as returned by mmap
        len: size_t,       // length of the range
        flags: c_int,      // MS_SYNC: return after the write completes
    ) -> c_int;            // 0, or -1 with errno set
}

With MS_SYNC, it returns after the dirty pages of the range are written:

impl Mapping {
    // ...
    /// Write the changed pages of a shared mapping back to the file now.
    fn sync(&self) -> io::Result<()> {
        // SAFETY: the range is exactly the mapping created in `new`.
        let result = unsafe { libc::msync(self.ptr.as_ptr().cast(), self.len, libc::MS_SYNC) };
        if result == 0 {
            Ok(())
        } else {
            Err(io::Error::last_os_error())
        }
    }
}

main writes HELLO through the shared mapping and syncs it. Then it maps the same file privately and writes world:

fn main() -> io::Result<()> {
    // ...
    shared.bytes_mut()[..5].copy_from_slice(b"HELLO");
    shared.sync()?;
    drop(shared);
    println!(
        "after writing HELLO through MAP_SHARED, the file starts with {:?}",
        file_prefix(&path)?
    );

    let mut private = Mapping::new(&file, len, false)?;
    private.bytes_mut()[..5].copy_from_slice(b"world");
    println!(
        "after writing world through MAP_PRIVATE, the mapping holds {:?}",
        String::from_utf8_lossy(&private.bytes()[..5])
    );
    drop(private);
    println!("and the file still starts with {:?}", file_prefix(&path)?);
    // ...
}

file_prefix reads the file with fs::read, the ordinary path through the page cache. It reports what the file holds, not what a mapping holds:

after writing HELLO through MAP_SHARED, the file starts with "HELLO"
after writing world through MAP_PRIVATE, the mapping holds "world"
and the file still starts with "HELLO"

fork uses the same mechanism for a whole process. The child gets a copy of the parent’s page tables with every writable page marked read-only. Neither process copies a page until one of them writes to it.

23.2.1 What a mapping cannot tell you

A read that fails returns an error. An access through a mapping has no return value, so the kernel reports a failure with a signal:

• If another process truncates the file, the pages past the new end no longer exist. Touching one of them raises SIGBUS, which kills the process unless it installs a handler.
• If the disk returns an error while a major fault reads a page, the access also raises SIGBUS.

There is also a rule that Rust’s types cannot check. A &[u8] promises that the bytes do not change while the slice is alive. Through a MAP_SHARED mapping, another process can write the file at any moment, and the slice’s bytes change under it. Mapping::bytes is sound only while no other process writes the file. The memmap2 crate makes Mmap::map an unsafe fn for this reason: the caller promises that the file stays unchanged.

23.2.2 When to map, and when to read

A mapping saves the copy into a buffer, but it has costs of its own. Every first touch of a page is a fault, and each fault enters the kernel. When a mapping is removed, every CPU that may hold its translations in a TLB must drop them. The kernel interrupts those CPUs to make them do it.

Databases such as LMDB map their data files for the second and third reasons. Log readers and copy tools read sequentially, and use read.

23.3 Sending a file to a socket

A static file server sends files to sockets. The simple version reads a chunk of the file into a buffer and writes the buffer to the socket, until the file ends. Figure 23.4 shows what each chunk costs. A read copies from the page cache to the buffer. A write copies from the buffer to the socket’s send buffer. The program never looks at the bytes. They pass through it only because read and write are the calls it knows.

sendfile moves the bytes from a file to a socket inside the kernel. One call names both descriptors, an offset, and a count. The kernel passes the page cache’s pages to the socket, and the bytes never enter the process. On network cards that can gather data from several places in memory, the kernel passes references to the pages instead of copying them. This is zero-copy: no copy of the payload by the CPU.

23.3.1 Measuring both

sendfile with a socket as the destination is a Linux call, so zero_copy puts its code in a module compiled only on Linux. It also needs a socket to send to. connected_pair opens a loopback connection and starts a thread that reads everything from the far end and counts it:

#[cfg(target_os = "linux")]
mod linux {
    use std::{
        fs::{self, File},
        io::{self, Read, Write},
        net::{TcpListener, TcpStream},
        os::fd::AsRawFd,
        thread,
        time::{Duration, Instant},
    };

    /// What one transfer cost.
    pub struct Transfer {
        pub bytes: u64,
        pub syscalls: u64,
        pub elapsed: Duration,
    }

    /// A connected socket pair over loopback, with a thread on the far end
    /// that reads and discards everything until the sender closes.
    fn connected_pair() -> io::Result<(TcpStream, thread::JoinHandle<u64>)> {
        let listener = TcpListener::bind("127.0.0.1:0")?;
        let address = listener.local_addr()?;
        let sink = thread::spawn(move || {
            let (mut stream, _) = listener.accept().expect("accept");
            let mut buffer = vec![0u8; 1 << 20];
            let mut total = 0u64;
            loop {
                match stream.read(&mut buffer) {
                    Ok(0) | Err(_) => return total,
                    Ok(read) => total += read as u64,
                }
            }
        });
        Ok((TcpStream::connect(address)?, sink))
    }
    // ...
}

Transfer records what one run cost. The receiving thread returns its byte count through join, so each run checks that every byte arrived.

The loop reads up to buffer_size bytes and writes what it read. It counts every system call:

mod linux {
    // ...
    /// Send `file` with `read` into a buffer and `write` out of it.
    pub fn copy_loop(path: &str, buffer_size: usize) -> io::Result<Transfer> {
        let mut file = File::open(path)?;
        let (mut socket, sink) = connected_pair()?;
        let mut buffer = vec![0u8; buffer_size];
        let mut syscalls = 0;

        let start = Instant::now();
        loop {
            let read = file.read(&mut buffer)?;
            syscalls += 1;
            if read == 0 {
                break;
            }
            socket.write_all(&buffer[..read])?;
            syscalls += 1;
        }
        drop(socket);
        let bytes = sink.join().expect("sink thread");
        Ok(Transfer {
            bytes,
            syscalls,
            elapsed: start.elapsed(),
        })
    }
    // ...
}

write_all may need more than one write when the socket’s send buffer is full. The count adds one per write_all, so it is a lower bound.

The sendfile version has no buffer. On Linux, sendfile copies from one descriptor to another inside the kernel:

unsafe extern "C" {
    fn sendfile(
        out_fd: c_int,       // where the bytes go: the socket
        in_fd: c_int,        // where they come from: the file
        offset: *mut off_t,  // where to start reading; advanced by the kernel
        count: size_t,       // at most this many bytes
    ) -> ssize_t;            // bytes sent, or -1 with errno set
}

The program calls it until the offset reaches the file’s length:

mod linux {
    // ...
    /// Send `file` with `sendfile`, which moves the bytes inside the kernel.
    pub fn send_file(path: &str) -> io::Result<Transfer> {
        let file = File::open(path)?;
        let length = file.metadata()?.len();
        let (socket, sink) = connected_pair()?;
        let mut offset: libc::off_t = 0;
        let mut syscalls = 0;

        let start = Instant::now();
        while (offset as u64) < length {
            let remaining = (length - offset as u64) as usize;
            // SAFETY: both descriptors are open for the whole call, and the
            // kernel advances `offset` by the number of bytes it sent.
            let sent = unsafe {
                libc::sendfile(socket.as_raw_fd(), file.as_raw_fd(), &mut offset, remaining)
            };
            syscalls += 1;
            if sent < 0 {
                return Err(io::Error::last_os_error());
            }
        }
        drop(socket);
        let bytes = sink.join().expect("sink thread");
        Ok(Transfer {
            bytes,
            syscalls,
            elapsed: start.elapsed(),
        })
    }
    // ...
}

The kernel advances offset by the number of bytes it sent. A single call sends at most 2,147,479,552 bytes on Linux, and a signal can cut a call short. The loop handles both, although this run needs one call.

main writes a 256 MiB file and sends it once untimed, so both timed runs start with the file in the page cache. Run in a Linux container, in release mode:

$ cargo run --release --bin zero_copy
method                        MiB   syscalls         ms        MiB/s
read/write, 64 KiB            256       8193      316.4          809
sendfile                      256          1      194.3         1318

256 MiB in 64 KiB chunks is 4,096 chunks. Each takes a read and a write, and one last read returns 0, which gives 8,193 calls. sendfile made one. The time fell by about 40%. Both runs share the costs of loopback TCP and of the receiving thread, which still copies every byte into its own buffer. Over a real network card, the receiver is on another machine, and the sender’s saving is a larger share of its work.

23.3.2 Where zero-copy stops

sendfile works only when the bytes leave the kernel exactly as they sit in the page cache. Any transformation of the bytes needs a CPU to touch them:

• TLS. A TLS library such as rustls encrypts each record in user space. The plaintext has to come into the process first. A server that sends a file over HTTPS reads it into a buffer, encrypts the buffer, and writes the ciphertext. Linux can move the encryption into the kernel with kernel TLS: the library performs the handshake, then hands the session keys to the socket. With kernel TLS on, sendfile works on a TLS connection again, and the kernel encrypts as it sends.
• Compression. A server that compresses responses on the fly reads every byte. Storing a compressed copy of each file ahead of time brings sendfile back.

23.4 The complete programs

Listing 23.1 Shared and private mappings, and the faults that load them. src/bin/mmap_file.rs

//! Mapping a file into memory with `mmap`.
//!
//! A mapping makes a file's bytes addressable like a slice. The kernel loads a
//! page the first time it is touched, which shows up as a page fault. The flags
//! decide where writes go: through a `MAP_SHARED` mapping they reach the file,
//! and through a `MAP_PRIVATE` mapping they stay in a private copy of the page.
//!
//! Run with: cargo run --bin mmap_file

use std::{
    fs::{self, File},
    io,
    os::fd::AsRawFd,
    path::Path,
    ptr::NonNull,
    slice,
};

/// A file mapped into this process's address space. Unmapped when dropped.
struct Mapping {
    ptr: NonNull<u8>,
    len: usize,
}

impl Mapping {
    /// Map the first `len` bytes of `file` for reading and writing.
    ///
    /// `shared` chooses `MAP_SHARED`, whose writes reach the file, over
    /// `MAP_PRIVATE`, whose writes stay in this process.
    fn new(file: &File, len: usize, shared: bool) -> io::Result<Self> {
        let flags = if shared {
            libc::MAP_SHARED
        } else {
            libc::MAP_PRIVATE
        };

        // SAFETY: a null address lets the kernel choose where to place the
        // mapping, and the descriptor is open for reading and writing. The
        // result is checked against MAP_FAILED before it is used.
        let address = unsafe {
            libc::mmap(
                std::ptr::null_mut(),
                len,
                libc::PROT_READ | libc::PROT_WRITE,
                flags,
                file.as_raw_fd(),
                0,
            )
        };
        if address == libc::MAP_FAILED {
            return Err(io::Error::last_os_error());
        }

        let ptr = NonNull::new(address.cast::<u8>()).ok_or_else(io::Error::last_os_error)?;
        Ok(Self { ptr, len })
    }

    fn bytes(&self) -> &[u8] {
        // SAFETY: the mapping is `len` bytes long and stays valid until drop.
        unsafe { slice::from_raw_parts(self.ptr.as_ptr(), self.len) }
    }

    fn bytes_mut(&mut self) -> &mut [u8] {
        // SAFETY: as in `bytes`, and `&mut self` makes this the only borrow.
        unsafe { slice::from_raw_parts_mut(self.ptr.as_ptr(), self.len) }
    }

    /// Write the changed pages of a shared mapping back to the file now.
    fn sync(&self) -> io::Result<()> {
        // SAFETY: the range is exactly the mapping created in `new`.
        let result = unsafe { libc::msync(self.ptr.as_ptr().cast(), self.len, libc::MS_SYNC) };
        if result == 0 {
            Ok(())
        } else {
            Err(io::Error::last_os_error())
        }
    }
}

impl Drop for Mapping {
    fn drop(&mut self) {
        // SAFETY: the range is exactly the mapping created in `new`, and no
        // borrow of it can outlive `self`.
        unsafe {
            libc::munmap(self.ptr.as_ptr().cast(), self.len);
        }
    }
}

/// The size of one page of memory on this machine.
fn page_size() -> usize {
    // SAFETY: `sysconf` reads a system constant and has no side effects.
    let size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };
    usize::try_from(size).expect("page size is positive")
}

/// Minor page faults this process has taken so far: faults the kernel served
/// from memory, without reading the disk.
fn minor_faults() -> u64 {
    // SAFETY: `getrusage` fills the struct it is given and nothing else.
    let usage = unsafe {
        let mut usage = std::mem::zeroed::<libc::rusage>();
        libc::getrusage(libc::RUSAGE_SELF, &mut usage);
        usage
    };
    u64::try_from(usage.ru_minflt).expect("fault count is not negative")
}

/// Read one byte from every page, so each page is touched once.
fn touch_every_page(bytes: &[u8], page: usize) -> u64 {
    bytes
        .iter()
        .step_by(page)
        .map(|&byte| u64::from(byte))
        .sum()
}

/// The first five bytes of the file at `path`, as text.
fn file_prefix(path: &Path) -> io::Result<String> {
    let bytes = fs::read(path)?;
    Ok(String::from_utf8_lossy(&bytes[..5]).into_owned())
}

fn main() -> io::Result<()> {
    const PAGES: usize = 64;

    let page = page_size();
    let len = PAGES * page;
    let path = std::env::temp_dir().join("mmap_file_demo.bin");
    fs::write(&path, vec![b'a'; len])?;
    let file = File::options()
        .read(true)
        .write(true)
        .open(&path)?;

    println!("page size {page} bytes, file {PAGES} pages ({len} bytes)\n");

    let mut shared = Mapping::new(&file, len, true)?;

    let before = minor_faults();
    touch_every_page(shared.bytes(), page);
    let first = minor_faults() - before;

    let before = minor_faults();
    touch_every_page(shared.bytes(), page);
    let second = minor_faults() - before;

    println!("first pass over the mapping:  {first} minor faults");
    println!("second pass over the mapping: {second} minor faults\n");

    shared.bytes_mut()[..5].copy_from_slice(b"HELLO");
    shared.sync()?;
    drop(shared);
    println!(
        "after writing HELLO through MAP_SHARED, the file starts with {:?}",
        file_prefix(&path)?
    );

    let mut private = Mapping::new(&file, len, false)?;
    private.bytes_mut()[..5].copy_from_slice(b"world");
    println!(
        "after writing world through MAP_PRIVATE, the mapping holds {:?}",
        String::from_utf8_lossy(&private.bytes()[..5])
    );
    drop(private);
    println!("and the file still starts with {:?}", file_prefix(&path)?);

    fs::remove_file(&path)
}

#[cfg(test)]
mod tests {
    use super::*;

    /// A scratch file of `pages` pages of `b'a'`, unique to one test.
    fn scratch(name: &str, pages: usize) -> (std::path::PathBuf, File, usize) {
        let len = pages * page_size();
        let path = std::env::temp_dir().join(format!("mmap_file_test_{name}.bin"));
        fs::write(&path, vec![b'a'; len]).expect("write scratch file");
        let file = File::options()
            .read(true)
            .write(true)
            .open(&path)
            .expect("open scratch file");
        (path, file, len)
    }

    #[test]
    fn shared_writes_reach_the_file() {
        let (path, file, len) = scratch("shared", 2);
        let mut mapping = Mapping::new(&file, len, true).expect("map");
        mapping.bytes_mut()[..5].copy_from_slice(b"HELLO");
        mapping.sync().expect("msync");
        drop(mapping);
        assert_eq!(file_prefix(&path).expect("read"), "HELLO");
        fs::remove_file(path).expect("remove");
    }

    #[test]
    fn private_writes_stay_in_the_process() {
        let (path, file, len) = scratch("private", 2);
        let mut mapping = Mapping::new(&file, len, false).expect("map");
        mapping.bytes_mut()[..5].copy_from_slice(b"world");
        assert_eq!(&mapping.bytes()[..5], b"world");
        drop(mapping);
        assert_eq!(file_prefix(&path).expect("read"), "aaaaa");
        fs::remove_file(path).expect("remove");
    }
}

$ cargo run --bin mmap_file
page size 4096 bytes, file 64 pages (262144 bytes)

first pass over the mapping:  64 minor faults
second pass over the mapping: 0 minor faults

after writing HELLO through MAP_SHARED, the file starts with "HELLO"
after writing world through MAP_PRIVATE, the mapping holds "world"
and the file still starts with "HELLO"

Listing 23.2 A file sent over a socket with a read/write loop and with sendfile. src/bin/zero_copy.rs

//! Sending a file over a socket two ways: a read/write loop, and `sendfile`.
//!
//! The loop moves every byte through a buffer in this process: the kernel
//! copies it into the buffer on `read`, and copies it out again on `write`.
//! `sendfile` asks the kernel to move the bytes from the file to the socket
//! itself, so they never enter user space.
//!
//! Linux only. Run with: cargo run --release --bin zero_copy

#[cfg(target_os = "linux")]
mod linux {
    use std::{
        fs::{self, File},
        io::{self, Read, Write},
        net::{TcpListener, TcpStream},
        os::fd::AsRawFd,
        thread,
        time::{Duration, Instant},
    };

    /// What one transfer cost.
    pub struct Transfer {
        pub bytes: u64,
        pub syscalls: u64,
        pub elapsed: Duration,
    }

    /// A connected socket pair over loopback, with a thread on the far end
    /// that reads and discards everything until the sender closes.
    fn connected_pair() -> io::Result<(TcpStream, thread::JoinHandle<u64>)> {
        let listener = TcpListener::bind("127.0.0.1:0")?;
        let address = listener.local_addr()?;
        let sink = thread::spawn(move || {
            let (mut stream, _) = listener.accept().expect("accept");
            let mut buffer = vec![0u8; 1 << 20];
            let mut total = 0u64;
            loop {
                match stream.read(&mut buffer) {
                    Ok(0) | Err(_) => return total,
                    Ok(read) => total += read as u64,
                }
            }
        });
        Ok((TcpStream::connect(address)?, sink))
    }

    /// Send `file` with `read` into a buffer and `write` out of it.
    pub fn copy_loop(path: &str, buffer_size: usize) -> io::Result<Transfer> {
        let mut file = File::open(path)?;
        let (mut socket, sink) = connected_pair()?;
        let mut buffer = vec![0u8; buffer_size];
        let mut syscalls = 0;

        let start = Instant::now();
        loop {
            let read = file.read(&mut buffer)?;
            syscalls += 1;
            if read == 0 {
                break;
            }
            socket.write_all(&buffer[..read])?;
            syscalls += 1;
        }
        drop(socket);
        let bytes = sink.join().expect("sink thread");
        Ok(Transfer {
            bytes,
            syscalls,
            elapsed: start.elapsed(),
        })
    }

    /// Send `file` with `sendfile`, which moves the bytes inside the kernel.
    pub fn send_file(path: &str) -> io::Result<Transfer> {
        let file = File::open(path)?;
        let length = file.metadata()?.len();
        let (socket, sink) = connected_pair()?;
        let mut offset: libc::off_t = 0;
        let mut syscalls = 0;

        let start = Instant::now();
        while (offset as u64) < length {
            let remaining = (length - offset as u64) as usize;
            // SAFETY: both descriptors are open for the whole call, and the
            // kernel advances `offset` by the number of bytes it sent.
            let sent = unsafe {
                libc::sendfile(socket.as_raw_fd(), file.as_raw_fd(), &mut offset, remaining)
            };
            syscalls += 1;
            if sent < 0 {
                return Err(io::Error::last_os_error());
            }
        }
        drop(socket);
        let bytes = sink.join().expect("sink thread");
        Ok(Transfer {
            bytes,
            syscalls,
            elapsed: start.elapsed(),
        })
    }

    /// Write a scratch file of `size` bytes and return its path.
    pub fn scratch_file(size: usize) -> io::Result<String> {
        let path = std::env::temp_dir().join("zero_copy_demo.bin");
        fs::write(&path, vec![b'z'; size])?;
        Ok(path.to_string_lossy().into_owned())
    }
}

#[cfg(target_os = "linux")]
fn main() -> std::io::Result<()> {
    const SIZE: usize = 256 << 20;

    let path = linux::scratch_file(SIZE)?;
    // One untimed pass, so the file is in the page cache for both runs.
    linux::copy_loop(&path, 64 << 10)?;

    println!(
        "{:<22} {:>10} {:>10} {:>10} {:>12}",
        "method", "MiB", "syscalls", "ms", "MiB/s"
    );
    let runs = [
        ("read/write, 64 KiB", linux::copy_loop(&path, 64 << 10)?),
        ("sendfile", linux::send_file(&path)?),
    ];
    for (name, run) in runs {
        let mib = run.bytes as f64 / (1 << 20) as f64;
        let ms = run.elapsed.as_secs_f64() * 1000.0;
        println!(
            "{name:<22} {mib:>10.0} {:>10} {ms:>10.1} {:>12.0}",
            run.syscalls,
            mib / run.elapsed.as_secs_f64()
        );
    }

    std::fs::remove_file(&path)
}

#[cfg(not(target_os = "linux"))]
fn main() {
    eprintln!("zero_copy uses Linux sendfile; run it on Linux");
}

#[cfg(all(test, target_os = "linux"))]
mod tests {
    use super::linux;

    #[test]
    fn both_methods_deliver_every_byte() {
        let path = linux::scratch_file(3 << 20).expect("scratch file");
        assert_eq!(
            linux::copy_loop(&path, 64 << 10)
                .expect("loop")
                .bytes,
            3 << 20
        );
        assert_eq!(
            linux::send_file(&path)
                .expect("sendfile")
                .bytes,
            3 << 20
        );
    }
}

23.5 Questions that come up

“Is a mapped file faster than read?” For repeated random access to a large file, usually. Each page faults once, and later reads cost a memory read. For one sequential pass, read with a reused buffer is as fast or faster, because it takes no faults.

“What happens when two processes map the same file with MAP_SHARED?” Both page tables point at the same page cache frames. A write by one is visible to the other at once, before any writeback. Shared memory between processes works this way.

“Does msync make a write durable?” MS_SYNC waits until the dirty pages are written to the device. Whether the device has stored them permanently depends on its write cache, as with fsync.

“Why does a file server stop using sendfile when it turns on TLS?” The TLS library encrypts the bytes in user space. Kernel TLS moves the encryption into the kernel and allows sendfile again.

“What does fault-around change?” It lowers the number of faults for a mapping read in order. It does not change what the program sees. It maps only neighbouring pages that are already cached, and reads nothing from disk.

Chapter 24 looks at file descriptors themselves. It covers the table each process keeps, what fork and exec do to it, and how a shell joins two programs with a pipe.

Exercises

1. Change PAGES to 4,096 and run mmap_file again. Then drop the page cache (on Linux, as root: echo 1 > /proc/sys/vm/drop_caches) between writing the file and mapping it, and read ru_majflt as well as ru_minflt. Which faults turn major?
2. Map the file read-only with PROT_READ only, and write through the slice from an unsafe block. Which signal stops the process, and which kind of fault from section 15.5 is it?
3. In copy_loop, try buffer sizes of 4 KiB, 64 KiB, and 1 MiB. Explain the change in system calls and in time.
4. Add a third method to zero_copy that maps the file and calls write_all on the whole mapping. Count its system calls and compare its time with the other two.
5. Write a test that truncates the file to one page while a two-page Mapping of it is alive. Check that the first page still reads. Do not read the second page in the test. Explain why.