Edge-triggered epoll

Section 20.4 built an echo server on epoll in its default mode, level-triggered. Epoll has a second mode, edge-triggered, selected with the flag EPOLLET. Section 20.6 noted that it needs a handler that reads until EAGAIN. This chapter shows why, by measuring what each mode reports. Then it covers two flags for loops that several threads share: EPOLLONESHOT and EPOLLEXCLUSIVE.

The program, epoll_edge, uses Linux epoll, so its output comes from a Linux container. It writes 10,000 bytes into a socket and counts the events and bytes each handler sees.

25.1 Levels and edges

A socket is readable while its receive buffer holds at least one byte. That is a state, or a level: it stays true until a read empties the buffer. The moment new bytes arrive is a change, or an edge.

Level-triggered epoll reports the level: every epoll_wait returns the socket while it is readable. Edge-triggered epoll reports the edge: one event when bytes arrive, and no more until more bytes arrive.

The difference comes from the ready list, introduced in section 20.4 (figure 25.1). When bytes arrive, a callback that epoll registered on the socket adds the socket to the ready list. epoll_wait takes entries off the list and returns them. For a level-triggered socket, the kernel puts the entry back on the list, and the next epoll_wait checks the socket again. It reports the socket only if it is still readable. An edge-triggered entry is not put back. Only the callback, on the next arrival, can add it again.

Here is what each mode reports when 10,000 bytes arrive at once and the handler reads 4,096 bytes per event:

In edge-triggered mode, 5,904 bytes stay in the buffer after the first event. No event will report them until the peer sends more. If the peer is waiting for a reply to the bytes it already sent, it never sends more, and both sides wait forever. This is the stall. Edge-triggered mode needs a handler that reads until read returns EAGAIN, which proves the buffer is empty.

25.2 Three handlers, measured

Epoll is three system calls. epoll_event is the record that carries a descriptor’s flags in and its events out:

struct epoll_event {
    events: u32, // EPOLLIN, EPOLLOUT, EPOLLET, ... combined with |
    u64: u64,    // data the caller chooses; the program stores the fd
}

unsafe extern "C" {
    fn epoll_create1(flags: c_int) -> c_int; // a new instance's fd; EPOLL_CLOEXEC
    fn epoll_ctl(
        epfd: c_int,                // the instance
        op: c_int,                  // EPOLL_CTL_ADD, EPOLL_CTL_MOD, or EPOLL_CTL_DEL
        fd: c_int,                  // the descriptor to watch
        event: *mut epoll_event,    // its flags
    ) -> c_int;                     // 0, or -1 with errno set
    fn epoll_wait(
        epfd: c_int,
        events: *mut epoll_event,   // an array the kernel fills
        maxevents: c_int,           // its length
        timeout: c_int,             // milliseconds; -1 waits forever
    ) -> c_int;                     // how many entries were filled, or -1
}

The program wraps the epoll instance in a type that closes it when dropped:

#[cfg(target_os = "linux")]
mod linux {
    use std::{
        io::{self, Read, Write},
        net::{TcpListener, TcpStream},
        os::{
            fd::{AsRawFd, FromRawFd, OwnedFd, RawFd},
            unix::net::UnixStream,
        },
        sync::{Arc, Barrier},
        thread,
        time::Duration,
    };

    /// How long `epoll_wait` waits before the program decides no event is coming.
    const QUIET_MS: i32 = 100;

    /// An epoll instance. Closed when dropped.
    pub struct Epoll(OwnedFd);

    impl Epoll {
        pub fn new() -> io::Result<Self> {
            // SAFETY: epoll_create1 takes no pointers, and the result is checked.
            let fd = unsafe { libc::epoll_create1(libc::EPOLL_CLOEXEC) };
            if fd < 0 {
                return Err(io::Error::last_os_error());
            }
            // SAFETY: the descriptor was just created, and nothing else owns it.
            Ok(Self(unsafe { OwnedFd::from_raw_fd(fd) }))
        }

        fn control(&self, op: libc::c_int, fd: RawFd, flags: u32) -> io::Result<()> {
            let mut event = libc::epoll_event {
                events: flags,
                u64: fd as u64,
            };
            // SAFETY: `event` is a valid epoll_event that outlives the call.
            let result = unsafe { libc::epoll_ctl(self.0.as_raw_fd(), op, fd, &mut event) };
            if result < 0 {
                return Err(io::Error::last_os_error());
            }
            Ok(())
        }

        /// Add `fd` to the interest list.
        pub fn add(&self, fd: RawFd, flags: u32) -> io::Result<()> {
            self.control(libc::EPOLL_CTL_ADD, fd, flags)
        }

        /// Change the flags of `fd`. This also re-arms an `EPOLLONESHOT` descriptor.
        pub fn modify(&self, fd: RawFd, flags: u32) -> io::Result<()> {
            self.control(libc::EPOLL_CTL_MOD, fd, flags)
        }

        /// Wait up to `timeout_ms` and return the number of events.
        pub fn wait(&self, timeout_ms: i32) -> io::Result<usize> {
            let mut events = [libc::epoll_event { events: 0, u64: 0 }; 8];
            // SAFETY: `events` is a writable buffer of 8 entries.
            let ready =
                unsafe { libc::epoll_wait(self.0.as_raw_fd(), events.as_mut_ptr(), 8, timeout_ms) };
            if ready < 0 {
                return Err(io::Error::last_os_error());
            }
            Ok(ready as usize)
        }
    }
    // ...
}

epoll_create1(EPOLL_CLOEXEC) creates the instance with close-on-exec set, as section 24.4 recommends. control fills an epoll_event with the flags and with the descriptor as the event’s data, then calls epoll_ctl. wait returns only the number of events, because each run watches a single socket.

The two handlers differ in one loop:

mod linux {
    // ...
    /// Read once, at most 4,096 bytes, and return how many arrived.
    pub fn read_once(socket: &mut UnixStream) -> io::Result<usize> {
        let mut buffer = [0; 4096];
        socket.read(&mut buffer)
    }

    /// Read until the socket has nothing left (`EAGAIN`), and return the total.
    pub fn read_until_eagain(socket: &mut UnixStream) -> io::Result<usize> {
        let mut buffer = [0; 4096];
        let mut total = 0;
        loop {
            match socket.read(&mut buffer) {
                Ok(0) => return Ok(total),
                Ok(read) => total += read,
                Err(error) if error.kind() == io::ErrorKind::WouldBlock => return Ok(total),
                Err(error) => return Err(error),
            }
        }
    }
    // ...
}

read_once makes one read of up to 4,096 bytes. read_until_eagain reads until the socket is empty. On a non-blocking socket, EAGAIN reaches Rust as io::ErrorKind::WouldBlock, which is the loop’s normal exit.

run connects a pair of Unix sockets and writes the bytes into one end. Then it handles events on the other end until 100 ms pass without one:

mod linux {
    // ...
    /// Write `sent` bytes into one end of a socket pair. Then wait on the other
    /// end with `flags`, and call `handler` once per event, until no event comes.
    /// Returns the bytes `handler` read at each event.
    pub fn run(
        flags: u32,
        sent: usize,
        handler: fn(&mut UnixStream) -> io::Result<usize>,
    ) -> io::Result<Vec<usize>> {
        let (mut writer, mut reader) = UnixStream::pair()?;
        reader.set_nonblocking(true)?;
        let epoll = Epoll::new()?;
        epoll.add(reader.as_raw_fd(), flags)?;
        writer.write_all(&vec![b'x'; sent])?;

        let mut per_event = Vec::new();
        while epoll.wait(QUIET_MS)? > 0 {
            per_event.push(handler(&mut reader)?);
        }
        Ok(per_event)
    }
    // ...
}

UnixStream::pair returns two sockets connected to each other, which keeps the test off the network. The reading end is non-blocking, as epoll requires. handler is a function pointer, so one run measures both handlers.

Animation 25.1 follows the three runs.

The output matches the table in section 25.1:

10000 bytes written into a socket; one handler call per epoll event

level-triggered, read once  events 3, bytes per event [4096, 4096, 1808], left unread 0
edge-triggered,  read once  events 1, bytes per event [4096], left unread 5904
edge-triggered,  to EAGAIN  events 1, bytes per event [10000], left unread 0

Level-triggered mode with one read per event delivered every byte, in three events. Edge-triggered mode with the same handler left 5,904 bytes unread. The drained handler took one event for all 10,000 bytes.

25.2.1 Why use edge-triggered mode

Edge-triggered mode reports each arrival once, which saves work in two places.

• Fewer re-checks. A level-triggered loop gets the same socket back from every epoll_wait until it is drained. The kernel re-checks every such socket on every call.
• Writes. A socket is writable whenever its send buffer has room, which is nearly always. Section 20.4 registered EPOLLOUT only while a reply was pending, because a level-triggered EPOLLOUT would report the socket on every call. In edge-triggered mode, a loop registers EPOLLIN | EPOLLOUT | EPOLLET once. The writable edge comes only after a write has returned EAGAIN and the buffer has drained. That is when the loop has a reply to finish.

The mio crate, which Tokio uses for its I/O on Linux, registers every socket edge-triggered for these reasons.

Draining to EAGAIN has a cost of its own. A client that sends faster than the loop reads keeps the handler in its read loop, and every other socket waits. A production loop caps the bytes it reads per event. It keeps the sockets it stopped early on a list of its own, and returns to them after serving the other events. Tokio’s tasks do the same with a budget: after a fixed amount of work, a task yields even if its socket still has data.

25.3 Several threads on one epoll: EPOLLONESHOT

A server can run several threads that all call epoll_wait on one epoll instance. In edge-triggered mode, a socket can then be handled by two threads at once:

1. Bytes arrive, and thread A wakes with the socket and starts reading.
2. More bytes arrive while A is still reading. The arrival is a new edge, so the socket is ready again.
3. Thread B wakes with the same socket and reads too. A and B each get part of the stream, and the order between their parts is lost.

EPOLLONESHOT disables a descriptor in the interest list after it reports one event. It stays registered, but reports nothing until the thread that handled it calls epoll_ctl with EPOLL_CTL_MOD to re-arm it. Only one thread can hold the socket between the event and the re-arm.

mod linux {
    // ...
    /// Events seen with `EPOLLONESHOT`: after the first write, after a second
    /// write, and after re-arming the descriptor.
    pub fn oneshot() -> io::Result<[usize; 3]> {
        let (mut writer, mut reader) = UnixStream::pair()?;
        reader.set_nonblocking(true)?;
        let epoll = Epoll::new()?;
        let flags = (libc::EPOLLIN | libc::EPOLLET | libc::EPOLLONESHOT) as u32;
        epoll.add(reader.as_raw_fd(), flags)?;

        writer.write_all(b"first")?;
        let first = epoll.wait(QUIET_MS)?;
        read_until_eagain(&mut reader)?;

        writer.write_all(b"second")?;
        let before_rearm = epoll.wait(QUIET_MS)?;
        epoll.modify(reader.as_raw_fd(), flags)?;
        let after_rearm = epoll.wait(QUIET_MS)?;
        Ok([first, before_rearm, after_rearm])
    }
    // ...
}

EPOLLONESHOT: first write 1 event, second write 0 events, after re-arming 1 event

The second write arrived while the descriptor was disabled, and epoll_wait timed out. The re-arm found the socket readable, so the next epoll_wait reported it at once. No data is lost while a descriptor is disabled. It waits in the socket’s buffer for the re-arm.

25.4 Many waiters on one listener: EPOLLEXCLUSIVE

A common server design gives each thread its own epoll instance, and adds the same listening socket to all of them. When one connection arrives, the listener’s callback runs for every instance, and every thread wakes. One of them wins accept. The others get EAGAIN and go back to sleep. With 64 threads, one connection costs 64 wakeups and 63 wasted system calls. This is the thundering herd.

EPOLLEXCLUSIVE, added in Linux 4.5, marks an instance’s interest in the listener as exclusive. When a connection arrives, the kernel wakes one of the exclusive waiters, or a few, instead of all of them (figure 25.3). The flag is allowed only with EPOLL_CTL_ADD.

woken_by_one_connection starts the threads, waits until each has registered the listener, connects one client, and counts the threads whose epoll_wait returned an event:

mod linux {
    // ...
    /// Start `waiters` threads, each blocked in `epoll_wait` on its own epoll
    /// instance that watches one shared listener. Connect one client, and
    /// return how many threads woke.
    pub fn woken_by_one_connection(waiters: usize, exclusive: bool) -> io::Result<usize> {
        let listener = TcpListener::bind("127.0.0.1:0")?;
        listener.set_nonblocking(true)?;
        let fd = listener.as_raw_fd();
        let flags = if exclusive {
            (libc::EPOLLIN | libc::EPOLLEXCLUSIVE) as u32
        } else {
            libc::EPOLLIN as u32
        };

        let registered = Arc::new(Barrier::new(waiters + 1));
        let threads: Vec<_> = (0..waiters)
            .map(|_| {
                let registered = Arc::clone(&registered);
                thread::spawn(move || -> io::Result<bool> {
                    let epoll = Epoll::new()?;
                    epoll.add(fd, flags)?;
                    registered.wait();
                    Ok(epoll.wait(500)? > 0)
                })
            })
            .collect();

        registered.wait();
        thread::sleep(Duration::from_millis(50));
        let _client = TcpStream::connect(listener.local_addr()?)?;

        let mut woken = 0;
        for thread in threads {
            if thread.join().expect("waiter thread")? {
                woken += 1;
            }
        }
        Ok(woken)
    }
}

The Barrier holds the main thread until every waiter has called epoll_ctl. The 50 ms sleep gives each waiter time to block in epoll_wait before the client connects.

4 threads wait on one listener, and one client connects:
  without EPOLLEXCLUSIVE: 4 threads woke
  with EPOLLEXCLUSIVE:    1 thread woke

SO_REUSEPORT solves the same problem another way. Each thread creates its own listening socket on the same port, with the option set before bind. The kernel then assigns each new connection to one of the sockets, chosen by a hash of the client’s address and port. No two threads share a listener, so none is woken for another’s connection.

25.5 The complete program

Listing 25.1 Level- and edge-triggered handlers, EPOLLONESHOT, and EPOLLEXCLUSIVE. src/bin/epoll_edge.rs

//! Edge-triggered epoll: what it reports, and the handler it needs.
//!
//! Level-triggered epoll reports a socket for as long as it has data.
//! Edge-triggered epoll reports it once per arrival, so a handler that leaves
//! data behind never hears about it again. The program also shows
//! `EPOLLONESHOT`, which disables a descriptor after one event, and
//! `EPOLLEXCLUSIVE`, which wakes one waiter instead of all of them.
//!
//! Linux only. Run with: cargo run --bin epoll_edge

#[cfg(target_os = "linux")]
mod linux {
    use std::{
        io::{self, Read, Write},
        net::{TcpListener, TcpStream},
        os::{
            fd::{AsRawFd, FromRawFd, OwnedFd, RawFd},
            unix::net::UnixStream,
        },
        sync::{Arc, Barrier},
        thread,
        time::Duration,
    };

    /// How long `epoll_wait` waits before the program decides no event is coming.
    const QUIET_MS: i32 = 100;

    /// An epoll instance. Closed when dropped.
    pub struct Epoll(OwnedFd);

    impl Epoll {
        pub fn new() -> io::Result<Self> {
            // SAFETY: epoll_create1 takes no pointers, and the result is checked.
            let fd = unsafe { libc::epoll_create1(libc::EPOLL_CLOEXEC) };
            if fd < 0 {
                return Err(io::Error::last_os_error());
            }
            // SAFETY: the descriptor was just created, and nothing else owns it.
            Ok(Self(unsafe { OwnedFd::from_raw_fd(fd) }))
        }

        fn control(&self, op: libc::c_int, fd: RawFd, flags: u32) -> io::Result<()> {
            let mut event = libc::epoll_event {
                events: flags,
                u64: fd as u64,
            };
            // SAFETY: `event` is a valid epoll_event that outlives the call.
            let result = unsafe { libc::epoll_ctl(self.0.as_raw_fd(), op, fd, &mut event) };
            if result < 0 {
                return Err(io::Error::last_os_error());
            }
            Ok(())
        }

        /// Add `fd` to the interest list.
        pub fn add(&self, fd: RawFd, flags: u32) -> io::Result<()> {
            self.control(libc::EPOLL_CTL_ADD, fd, flags)
        }

        /// Change the flags of `fd`. This also re-arms an `EPOLLONESHOT` descriptor.
        pub fn modify(&self, fd: RawFd, flags: u32) -> io::Result<()> {
            self.control(libc::EPOLL_CTL_MOD, fd, flags)
        }

        /// Wait up to `timeout_ms` and return the number of events.
        pub fn wait(&self, timeout_ms: i32) -> io::Result<usize> {
            let mut events = [libc::epoll_event { events: 0, u64: 0 }; 8];
            // SAFETY: `events` is a writable buffer of 8 entries.
            let ready =
                unsafe { libc::epoll_wait(self.0.as_raw_fd(), events.as_mut_ptr(), 8, timeout_ms) };
            if ready < 0 {
                return Err(io::Error::last_os_error());
            }
            Ok(ready as usize)
        }
    }

    /// Read once, at most 4,096 bytes, and return how many arrived.
    pub fn read_once(socket: &mut UnixStream) -> io::Result<usize> {
        let mut buffer = [0; 4096];
        socket.read(&mut buffer)
    }

    /// Read until the socket has nothing left (`EAGAIN`), and return the total.
    pub fn read_until_eagain(socket: &mut UnixStream) -> io::Result<usize> {
        let mut buffer = [0; 4096];
        let mut total = 0;
        loop {
            match socket.read(&mut buffer) {
                Ok(0) => return Ok(total),
                Ok(read) => total += read,
                Err(error) if error.kind() == io::ErrorKind::WouldBlock => return Ok(total),
                Err(error) => return Err(error),
            }
        }
    }

    /// Write `sent` bytes into one end of a socket pair. Then wait on the other
    /// end with `flags`, and call `handler` once per event, until no event comes.
    /// Returns the bytes `handler` read at each event.
    pub fn run(
        flags: u32,
        sent: usize,
        handler: fn(&mut UnixStream) -> io::Result<usize>,
    ) -> io::Result<Vec<usize>> {
        let (mut writer, mut reader) = UnixStream::pair()?;
        reader.set_nonblocking(true)?;
        let epoll = Epoll::new()?;
        epoll.add(reader.as_raw_fd(), flags)?;
        writer.write_all(&vec![b'x'; sent])?;

        let mut per_event = Vec::new();
        while epoll.wait(QUIET_MS)? > 0 {
            per_event.push(handler(&mut reader)?);
        }
        Ok(per_event)
    }

    /// Events seen with `EPOLLONESHOT`: after the first write, after a second
    /// write, and after re-arming the descriptor.
    pub fn oneshot() -> io::Result<[usize; 3]> {
        let (mut writer, mut reader) = UnixStream::pair()?;
        reader.set_nonblocking(true)?;
        let epoll = Epoll::new()?;
        let flags = (libc::EPOLLIN | libc::EPOLLET | libc::EPOLLONESHOT) as u32;
        epoll.add(reader.as_raw_fd(), flags)?;

        writer.write_all(b"first")?;
        let first = epoll.wait(QUIET_MS)?;
        read_until_eagain(&mut reader)?;

        writer.write_all(b"second")?;
        let before_rearm = epoll.wait(QUIET_MS)?;
        epoll.modify(reader.as_raw_fd(), flags)?;
        let after_rearm = epoll.wait(QUIET_MS)?;
        Ok([first, before_rearm, after_rearm])
    }

    /// Start `waiters` threads, each blocked in `epoll_wait` on its own epoll
    /// instance that watches one shared listener. Connect one client, and
    /// return how many threads woke.
    pub fn woken_by_one_connection(waiters: usize, exclusive: bool) -> io::Result<usize> {
        let listener = TcpListener::bind("127.0.0.1:0")?;
        listener.set_nonblocking(true)?;
        let fd = listener.as_raw_fd();
        let flags = if exclusive {
            (libc::EPOLLIN | libc::EPOLLEXCLUSIVE) as u32
        } else {
            libc::EPOLLIN as u32
        };

        let registered = Arc::new(Barrier::new(waiters + 1));
        let threads: Vec<_> = (0..waiters)
            .map(|_| {
                let registered = Arc::clone(&registered);
                thread::spawn(move || -> io::Result<bool> {
                    let epoll = Epoll::new()?;
                    epoll.add(fd, flags)?;
                    registered.wait();
                    Ok(epoll.wait(500)? > 0)
                })
            })
            .collect();

        registered.wait();
        thread::sleep(Duration::from_millis(50));
        let _client = TcpStream::connect(listener.local_addr()?)?;

        let mut woken = 0;
        for thread in threads {
            if thread.join().expect("waiter thread")? {
                woken += 1;
            }
        }
        Ok(woken)
    }
}

#[cfg(target_os = "linux")]
fn main() -> std::io::Result<()> {
    use linux::{read_once, read_until_eagain, run};

    const SENT: usize = 10_000;
    let level = libc::EPOLLIN as u32;
    let edge = (libc::EPOLLIN | libc::EPOLLET) as u32;

    println!("{SENT} bytes written into a socket; one handler call per epoll event\n");
    let runs = [
        ("level-triggered, read once ", run(level, SENT, read_once)?),
        ("edge-triggered,  read once ", run(edge, SENT, read_once)?),
        (
            "edge-triggered,  to EAGAIN ",
            run(edge, SENT, read_until_eagain)?,
        ),
    ];
    for (name, per_event) in runs {
        let left = SENT - per_event.iter().sum::<usize>();
        println!(
            "{name} events {}, bytes per event {per_event:?}, left unread {left}",
            per_event.len()
        );
    }

    let [first, before, after] = linux::oneshot()?;
    println!(
        "\nEPOLLONESHOT: first write {first} event, second write {before} events, \
         after re-arming {after} event"
    );

    println!("\n4 threads wait on one listener, and one client connects:");
    println!(
        "  without EPOLLEXCLUSIVE: {} threads woke",
        linux::woken_by_one_connection(4, false)?
    );
    println!(
        "  with EPOLLEXCLUSIVE:    {} thread woke",
        linux::woken_by_one_connection(4, true)?
    );
    Ok(())
}

#[cfg(not(target_os = "linux"))]
fn main() {
    eprintln!("epoll_edge uses Linux epoll; run it on Linux");
}

#[cfg(all(test, target_os = "linux"))]
mod tests {
    use super::linux::*;

    const LEVEL: u32 = libc::EPOLLIN as u32;
    const EDGE: u32 = (libc::EPOLLIN | libc::EPOLLET) as u32;

    #[test]
    fn level_triggered_reports_until_the_data_is_gone() {
        assert_eq!(
            run(LEVEL, 10_000, read_once).expect("run"),
            [4096, 4096, 1808]
        );
    }

    #[test]
    fn edge_triggered_with_one_read_leaves_data_behind() {
        assert_eq!(run(EDGE, 10_000, read_once).expect("run"), [4096]);
    }

    #[test]
    fn edge_triggered_drained_to_eagain_reads_everything() {
        assert_eq!(run(EDGE, 10_000, read_until_eagain).expect("run"), [10_000]);
    }

    #[test]
    fn oneshot_waits_for_a_rearm() {
        assert_eq!(oneshot().expect("oneshot"), [1, 0, 1]);
    }
}

$ cargo run --bin epoll_edge
10000 bytes written into a socket; one handler call per epoll event

level-triggered, read once  events 3, bytes per event [4096, 4096, 1808], left unread 0
edge-triggered,  read once  events 1, bytes per event [4096], left unread 5904
edge-triggered,  to EAGAIN  events 1, bytes per event [10000], left unread 0

EPOLLONESHOT: first write 1 event, second write 0 events, after re-arming 1 event

4 threads wait on one listener, and one client connects:
  without EPOLLEXCLUSIVE: 4 threads woke
  with EPOLLEXCLUSIVE:    1 thread woke

25.6 Questions that come up

“A server under edge-triggered epoll sometimes stops answering one client. CPU use is zero. What do you check first?” Whether every read loop runs until EAGAIN. A loop that stops after a full buffer, or after a fixed number of bytes, leaves data that no event will report.

“Does edge-triggered mode apply to accept?” Yes. An edge-triggered listener reports once when connections arrive, so the handler calls accept until EAGAIN, as accept_ready in section 20.4 does.

“What happens to an edge-triggered socket when the peer closes it?” The close is an arrival. Epoll reports EPOLLIN, and EPOLLRDHUP if it was requested. The drain loop then reads the remaining bytes and sees read return 0.

“Is kqueue edge-triggered?” kqueue offers both. EV_CLEAR gives edge-triggered behavior, and EV_DISPATCH disables an event after it fires, like EPOLLONESHOT.

“Why not have every thread call accept on one blocking listener?” That works. The kernel queues blocked accept callers and wakes one per connection. The herd appears when threads wait in epoll_wait instead, because each epoll instance is a separate waiter on the listener.

Chapter 26 goes under the locks used in chapter 16. It builds a mutex on the futex system call, and runs litmus tests that show what each memory ordering allows.

Exercises

1. Change run so that after the edge-triggered stall it writes one more byte, and count the events. How many bytes does read_once return now, and why?
2. Add EPOLLOUT | EPOLLET to a socket in run and count the events while nothing is written. Then fill the socket’s send buffer until write returns WouldBlock, read from the other end, and count again.
3. Cap read_until_eagain at 8,192 bytes per call and return whether it stopped early. Write a loop that keeps a list of sockets that stopped early, and serves them after the events from epoll_wait.
4. Run woken_by_one_connection with 16 threads and 10 connections. How many wakeups does each mode cost?
5. Rewrite woken_by_one_connection with SO_REUSEPORT, using the socket2 crate to set the option before bind. Count how the 10 connections spread across the threads.