TCP Sockets

A socket is the endpoint of a network connection, held by a process as a file descriptor. Behind the descriptor the kernel keeps the state of the connection: who the peer is, a buffer of data waiting to be sent, and a buffer of data received and not yet read. read and write work on a connected socket as they do on a pipe, and the same descriptor can be registered with epoll.

In this chapter, we will learn what the kernel does for a TCP connection between the calls a program makes. We will first measure a connection on the loopback interface with sockfacts. Then we will write sockmodel, a model of the receiving side: the table that finds the socket for an arriving packet, the accept queue of a listener, and the receive buffer that limits the sender.

From write to the network

A network carries packets: blocks of at most about 1500 bytes on Ethernet, each delivered independently, and each of which may be lost, duplicated or delivered out of order. A program wants something else: a stream of bytes that arrives complete and in order. The kernel’s network stack builds the second from the first, in layers.

An IP address names a machine, and a port is a 16-bit number that names a program on it. The loopback device, with the address 127.0.0.1, delivers packets to the same machine without any hardware, and the programs of this chapter use it.

The segments of a connection

TCP makes an unreliable network reliable by numbering. Every byte of a stream has a sequence number. Each packet of TCP, called a segment, says which bytes it carries. The receiver answers with an acknowledgement: the number of the next byte it expects, which confirms everything before it. The sender keeps each byte until it is acknowledged, and sends it again if no acknowledgement arrives in time. Out-of-order segments are put in order by their numbers, and duplicates are recognised and dropped.

An example with small numbers shows the whole mechanism. The client sends hello, 5 bytes, and its stream is currently at byte 1001.

The acknowledgement is a single number, and it confirms a whole range. This keeps the protocol simple: the receiver always says “the next byte I need is this one”, and whatever went wrong, the sender knows where to continue.

Before any data, the 2 sides have to agree on their starting sequence numbers. This is the handshake, 3 segments marked with the flags SYN and ACK.

1. SYN: the client proposes its starting number, 1000.
2. SYN and ACK: the server acknowledges it with 1001 and proposes its own, 5000.
3. ACK: the client acknowledges 5001. Both sides now know both numbers, and the connection is established.
4. The client sends 5 bytes, numbered 1001 to 1005.
5. The server acknowledges 1006 and reports its window, the space left in its receive buffer.
6. FIN: the client has nothing more to send. This closes its direction of the stream.
7. The server acknowledges. Its program’s next read returns 0. The server can still send, until it sends a FIN of its own.

All of this is done by the kernels. The programs call connect, write, read and close, and do not see a segment.

Setting up a connection

TCP gives 2 programs a reliable stream of bytes in each direction. A server and a client reach that state through different calls.

The handshake is carried out by the server’s kernel. The server program takes no part in it. An established connection waits in the accept queue of the listener until the program calls accept.

A connection is identified by 4 values: the local address, the local port, the remote address and the remote port. A listener and all the connections accepted from it have the same local port. An arriving packet, called a segment in TCP, is matched against the 4 values of the existing connections. A segment that matches none and asks to open a connection, a SYN, is given to the listener on its port.

The kernel’s structures

Behind a socket descriptor is a struct file (chapter 2) whose function table is the socket’s, and behind that a struct sock. For TCP the struct sock is the first part of a larger struct tcp_sock. These are fields from include/net/sock.h and include/linux/tcp.h of Linux 5.15.

struct sock_common {
        __be32 skc_daddr;                      /* remote IP address */
        __be32 skc_rcv_saddr;                  /* local IP address */
        __be16 skc_dport;                      /* remote port */
        __u16 skc_num;                         /* local port */
        volatile unsigned char skc_state;      /* TCP_LISTEN, TCP_ESTABLISHED, ... */
        /* ... */
};

struct sock {
        struct sock_common __sk_common;
        struct sk_buff_head sk_receive_queue;  /* segments received, not yet read */
        int sk_rcvbuf;                         /* limit of the receive queue, bytes */
        struct sk_buff_head sk_write_queue;    /* data written, not yet acknowledged */
        int sk_sndbuf;                         /* limit of the write queue, bytes */
        u32 sk_ack_backlog;                    /* connections waiting for accept */
        u32 sk_max_ack_backlog;                /* the backlog given to listen */
        /* ... */
};

struct tcp_sock {
        u32 rcv_nxt;                           /* next byte we expect to receive */
        u32 snd_nxt;                           /* next byte we will send */
        u32 snd_una;                           /* oldest byte sent and not acknowledged */
        u32 snd_wnd;                           /* the window the peer offered */
        u32 snd_cwnd;                          /* congestion window */
        /* ... */
};

The ss command prints these fields for the sockets of a system. Here a server listens on port 9099 and a client has connected to it over loopback:

$ ss -tln 'sport = :9099'
State  Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0      1          127.0.0.1:9099      0.0.0.0:*
$ ss -tn 'sport = :9099 or dport = :9099'
State Recv-Q Send-Q Local Address:Port  Peer Address:Port
ESTAB 0      0          127.0.0.1:9099     127.0.0.1:51252
ESTAB 0      0          127.0.0.1:51252    127.0.0.1:9099

The first command shows the listener. For a listener, Send-Q is the backlog, 1 here, and Recv-Q is the number of connections waiting for accept. The second shows the connection 2 times, because both ends are on this machine: the server’s socket with local port 9099, and the client’s with the port 51252 that the kernel chose for it. The 4 values of one line are the 4 values of the other, exchanged.

Measuring a connection

sockfacts makes its connections over the loopback address 127.0.0.1, with both ends in the same process. The Go version calls the socket system calls through golang.org/x/sys/unix. The Rust version uses std::net and the nix crate with the socket feature for socket options.

$ cargo new sockfacts
$ cargo add -p sockfacts nix --features socket

listen binds to port 0, which asks the kernel for any free port.

Listing 10.1 The imports of sockfacts, in Go. code/go/sockfacts/main.go

import (
	"errors"
	"fmt"
	"log"

	"golang.org/x/sys/unix"
)

Listing 10.2 The imports of sockfacts, in Rust. code/rust/sockfacts/src/main.rs

use std::io::{self, Read, Write};
use std::net::{TcpListener, TcpStream};
use std::os::fd::AsRawFd;

use nix::errno::Errno;
use nix::sys::socket::{setsockopt, sockopt};

Listing 10.3 A listener and a client on the loopback interface, in Go. code/go/sockfacts/main.go

// listen returns a TCP socket that listens on a free port of 127.0.0.1, and
// its address.
func listen() (int, *unix.SockaddrInet4, error) {
	listener, err := unix.Socket(unix.AF_INET, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)

	if err != nil {
		return 0, nil, err
	}

	if err := unix.Bind(listener, &unix.SockaddrInet4{Addr: [4]byte{127, 0, 0, 1}}); err != nil {
		return 0, nil, err
	}

	if err := unix.Listen(listener, 16); err != nil {
		return 0, nil, err
	}

	address, err := unix.Getsockname(listener)

	if err != nil {
		return 0, nil, err
	}

	return listener, address.(*unix.SockaddrInet4), nil
}

// connect returns a TCP socket connected to an address.
func connect(address *unix.SockaddrInet4) (int, error) {
	client, err := unix.Socket(unix.AF_INET, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)

	if err != nil {
		return 0, err
	}

	return client, unix.Connect(client, address)
}

// name returns the symbolic name of a system call error, such as EPIPE.
func name(err error) string {
	var errno unix.Errno

	if errors.As(err, &errno) {
		return unix.ErrnoName(errno)
	}

	return fmt.Sprint(err)
}

// port returns the port of an address.
func port(address unix.Sockaddr, err error) int {
	if inet, ok := address.(*unix.SockaddrInet4); ok && err == nil {
		return inet.Port
	}

	return 0
}

Listing 10.4 A listener and a client on the loopback interface, in Rust. code/rust/sockfacts/src/main.rs

/// Returns a TCP socket that listens on a free port of 127.0.0.1.
fn listen() -> io::Result<TcpListener> {
    TcpListener::bind("127.0.0.1:0")
}

/// Returns a TCP socket connected to the address of a listener.
fn connect(listener: &TcpListener) -> io::Result<TcpStream> {
    TcpStream::connect(listener.local_addr()?)
}

/// Returns the symbolic name of a system call error, such as `EPIPE`.
fn name(error: &io::Error) -> String {
    match error.raw_os_error() {
        Some(code) => format!("{:?}", Errno::from_raw(code)),
        None => error.to_string(),
    }
}

establish calls connect before the server has called accept. Then it accepts and compares the addresses of the ends.

Listing 10.5 Connecting before accept, and the ports of both ends, in Go. code/go/sockfacts/main.go

// establish connects before the server calls accept, and compares the
// addresses of both ends.
func establish() error {
	listener, address, err := listen()

	if err != nil {
		return err
	}

	defer unix.Close(listener)

	client, err := connect(address)
	fmt.Printf("connect before the server calls accept: succeeded %v\n", err == nil)

	if err != nil {
		return err
	}

	defer unix.Close(client)

	server, _, err := unix.Accept(listener)

	if err != nil {
		return err
	}

	defer unix.Close(server)

	fmt.Printf("accept returns a new descriptor: %v, on the listener's port: %v\n",
		server != listener, port(unix.Getsockname(server)) == address.Port)
	fmt.Printf("the client's local port is the server's peer port: %v\n",
		port(unix.Getsockname(client)) == port(unix.Getpeername(server)))

	return stream(client, server)
}

Listing 10.6 Connecting before accept, and the ports of both ends, in Rust. code/rust/sockfacts/src/main.rs

/// Connects before the server calls accept, and compares the addresses of
/// both ends.
fn establish() -> io::Result<()> {
    let listener = listen()?;

    let client = connect(&listener);
    println!(
        "connect before the server calls accept: succeeded {}",
        client.is_ok()
    );

    let client = client?;
    let (server, _) = listener.accept()?;

    println!(
        "accept returns a new descriptor: {}, on the listener's port: {}",
        server.as_raw_fd() != listener.as_raw_fd(),
        server.local_addr()?.port() == listener.local_addr()?.port()
    );
    println!(
        "the client's local port is the server's peer port: {}",
        client.local_addr()?.port() == server.peer_addr()?.port()
    );

    stream(client, server)
}

stream sends ab and cd with separate calls to write, and the server reads 3 bytes and then 1. After that the client closes its socket. The server reads again, and then writes until a write fails.

Listing 10.7 A stream of bytes, and the end of the stream, in Go. code/go/sockfacts/main.go

// readFull reads until buf is full.
func readFull(fd int, buf []byte) error {
	for filled := 0; filled < len(buf); {
		n, err := unix.Read(fd, buf[filled:])

		if err != nil {
			return err
		}

		if n == 0 {
			return errors.New("the stream ended early")
		}

		filled += n
	}

	return nil
}

// stream writes "ab" and "cd" with 2 calls and reads 3 bytes and then 1.
// Then the client closes, and the server reads and writes again.
func stream(client, server int) error {
	for _, part := range []string{"ab", "cd"} {
		if _, err := unix.Write(client, []byte(part)); err != nil {
			return err
		}
	}

	first, second := make([]byte, 3), make([]byte, 1)

	if err := readFull(server, first); err != nil {
		return err
	}

	if err := readFull(server, second); err != nil {
		return err
	}

	fmt.Printf("writes of \"ab\" and \"cd\", reads of 3 bytes and 1 byte: %q and %q\n", first, second)

	unix.Close(client)

	n, err := unix.Read(server, first)

	if err != nil {
		return err
	}

	fmt.Printf("the client closed: read returns %d\n", n)

	for err == nil {
		_, err = unix.Write(server, []byte("x"))
	}

	fmt.Printf("writes to the closed peer: the first is accepted, a later one fails with %s\n", name(err))

	return nil
}

Listing 10.8 A stream of bytes, and the end of the stream, in Rust. code/rust/sockfacts/src/main.rs

/// Writes "ab" and "cd" with 2 calls and reads 3 bytes and then 1. Then the
/// client closes, and the server reads and writes again.
fn stream(mut client: TcpStream, mut server: TcpStream) -> io::Result<()> {
    for part in ["ab", "cd"] {
        client.write_all(part.as_bytes())?;
    }

    let (mut first, mut second) = ([0u8; 3], [0u8; 1]);

    server.read_exact(&mut first)?;
    server.read_exact(&mut second)?;

    println!(
        "writes of \"ab\" and \"cd\", reads of 3 bytes and 1 byte: {:?} and {:?}",
        String::from_utf8_lossy(&first),
        String::from_utf8_lossy(&second)
    );

    drop(client);

    let n = server.read(&mut first)?;
    println!("the client closed: read returns {n}");

    let error = loop {
        if let Err(error) = server.write(b"x") {
            break error;
        }
    };

    println!(
        "writes to the closed peer: the first is accepted, a later one fails with {}",
        name(&error)
    );

    Ok(())
}

flow reduces the buffers of a connection, so that the numbers are small, and writes to the client socket in non-blocking mode until the kernel refuses a write. The server does not read during that time. Then the server reads everything, and the client tries again.

Listing 10.9 A sender whose peer does not read, in Go. code/go/sockfacts/main.go

// fill writes blocks to a non-blocking socket until the kernel refuses one
// and returns the number of bytes accepted.
func fill(fd int) (int, error) {
	block := make([]byte, 1024)
	total := 0

	for {
		n, err := unix.Write(fd, block)

		if errors.Is(err, unix.EAGAIN) {
			return total, nil
		}

		if err != nil {
			return total, err
		}

		total += n
	}
}

// flow gives a connection small buffers, fills it while the server does not
// read, lets the server read everything, and fills it again.
func flow() error {
	listener, address, err := listen()

	if err != nil {
		return err
	}

	defer unix.Close(listener)

	if err := unix.SetsockoptInt(listener, unix.SOL_SOCKET, unix.SO_RCVBUF, 4096); err != nil {
		return err
	}

	client, err := connect(address)

	if err != nil {
		return err
	}

	defer unix.Close(client)

	if err := unix.SetsockoptInt(client, unix.SOL_SOCKET, unix.SO_SNDBUF, 4096); err != nil {
		return err
	}

	server, _, err := unix.Accept(listener)

	if err != nil {
		return err
	}

	defer unix.Close(server)

	if err := unix.SetNonblock(client, true); err != nil {
		return err
	}

	sent, err := fill(client)

	if err != nil {
		return err
	}

	fmt.Printf("the server does not read: the client is stopped after %d bytes\n", sent)

	if err := readFull(server, make([]byte, sent)); err != nil {
		return err
	}

	again, err := fill(client)

	if err != nil {
		return err
	}

	fmt.Printf("the server has read everything: the client can write again: %v\n", again > 0)

	return nil
}

Listing 10.10 A sender whose peer does not read, in Rust. code/rust/sockfacts/src/main.rs

/// Writes blocks to a non-blocking socket until the kernel refuses one and
/// returns the number of bytes accepted.
fn fill(mut stream: &TcpStream) -> io::Result<usize> {
    let block = [0u8; 1024];
    let mut total = 0;

    loop {
        match stream.write(&block) {
            Ok(n) => total += n,
            Err(error) if error.kind() == io::ErrorKind::WouldBlock => return Ok(total),
            Err(error) => return Err(error),
        }
    }
}

/// Gives a connection small buffers, fills it while the server does not read,
/// lets the server read everything, and fills it again.
fn flow() -> io::Result<()> {
    let listener = listen()?;
    setsockopt(&listener, sockopt::RcvBuf, &4096)?;

    let client = connect(&listener)?;
    setsockopt(&client, sockopt::SndBuf, &4096)?;

    let (mut server, _) = listener.accept()?;

    client.set_nonblocking(true)?;

    let sent = fill(&client)?;
    println!("the server does not read: the client is stopped after {sent} bytes");

    server.read_exact(&mut vec![0u8; sent])?;

    let again = fill(&client)?;
    println!(
        "the server has read everything: the client can write again: {}",
        again > 0
    );

    Ok(())
}

connect before the server calls accept: succeeded true
accept returns a new descriptor: true, on the listener's port: true
the client's local port is the server's peer port: true
writes of "ab" and "cd", reads of 3 bytes and 1 byte: "abc" and "d"
the client closed: read returns 0
writes to the closed peer: the first is accepted, a later one fails with EPIPE
the server does not read: the client is stopped after 7168 bytes
the server has read everything: the client can write again: true

Row 4 has a practical consequence. A protocol built on TCP has to mark the boundaries of its messages itself, with a length before each message or with a delimiter such as a newline, and a reader has to be prepared to receive part of a message or several messages in a single read.

Buffers and the window

A write on a socket copies the data into the socket’s send buffer and returns. The kernel transmits from that buffer and keeps each byte until the peer has acknowledged it. On the other side, arriving data is placed in the receive buffer until the program reads it.

The 2 buffers are connected by the window. With every acknowledgement, the receiver reports how much free space its receive buffer has, and the sender transmits no more than that. When the receiving program stops reading, the receive buffer fills, the window becomes 0, the sender’s kernel stops transmitting, the send buffer fills, and write blocks. This is the flow control of a pipe, extended across a network. The 7168 bytes of the measurement are the contents of both buffers at that point.

Modelling the receiving side

The model covers the receiving half of TCP and omits transmission, acknowledgements and retransmission.

Listing 10.11 The imports of sockmodel, in Go. code/go/sockmodel/main.go

import "fmt"

Listing 10.12 The imports of sockmodel, in Rust. code/rust/sockmodel/src/main.rs

use std::collections::{HashMap, VecDeque};
use std::fmt;

Listing 10.13 Endpoints, sockets, listeners and segments, in Go. code/go/sockmodel/main.go

// An endpoint is an IP address and a port.
type endpoint struct {
	address string
	port    int
}

func (e endpoint) String() string {
	return fmt.Sprintf("%s:%d", e.address, e.port)
}

// A socket is one established connection: the bytes received and not yet
// read, the size of the receive buffer, and whether the sender has finished.
type socket struct {
	received []byte
	capacity int
	finished bool
}

// window is the free space of the receive buffer: the number of bytes the
// sender may still send.
func (s *socket) window() int {
	return s.capacity - len(s.received)
}

// A key identifies a connection on this machine: the remote endpoint and the
// local port. With the local address, which the model leaves out, it forms
// the 4 values that identify a TCP connection.
type key struct {
	remote endpoint
	port   int
}

// A listener accepts connections on a port. queue holds the connections that
// are established and have not been returned by accept.
type listener struct {
	backlog int
	queue   []key
}

// A stack is the TCP state of a machine.
type stack struct {
	listeners   map[int]*listener
	connections map[key]*socket
	bufferSize  int
}

// A segment is a packet of TCP. syn opens a connection and fin ends the
// sender's stream.
type segment struct {
	from     endpoint
	port     int
	syn, fin bool
	data     string
}

Listing 10.14 Endpoints, sockets, listeners and segments, in Rust. code/rust/sockmodel/src/main.rs

/// An IP address and a port.
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
struct Endpoint {
    address: &'static str,
    port: u16,
}

impl fmt::Display for Endpoint {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}:{}", self.address, self.port)
    }
}

/// One established connection: the bytes received and not yet read, the size
/// of the receive buffer, and whether the sender has finished.
struct Socket {
    received: VecDeque<u8>,
    capacity: usize,
    finished: bool,
}

impl Socket {
    /// The free space of the receive buffer: the number of bytes the sender
    /// may still send.
    fn window(&self) -> usize {
        self.capacity - self.received.len()
    }
}

/// Identifies a connection on this machine: the remote endpoint and the local
/// port. With the local address, which the model leaves out, it forms the 4
/// values that identify a TCP connection.
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
struct Key {
    remote: Endpoint,
    port: u16,
}

/// Accepts connections on a port. `queue` holds the connections that are
/// established and have not been returned by accept.
struct Listener {
    backlog: usize,
    queue: VecDeque<Key>,
}

/// The TCP state of a machine.
struct Stack {
    listeners: HashMap<u16, Listener>,
    connections: HashMap<Key, Socket>,
    buffer_size: usize,
}

/// A packet of TCP. `syn` opens a connection and `fin` ends the sender's
/// stream.
#[derive(Default)]
struct Segment {
    syn: bool,
    fin: bool,
    data: &'static str,
}

receive is the lookup of figure 10.4. The table of connections is consulted first, which is why data for an established connection reaches its own socket while a listener exists on the same port. A SYN for a full accept queue is dropped without an answer. The client’s kernel sends it again after a timeout, which gives the server time to call accept.

Listing 10.15 Finding the socket for a segment, in Go. code/go/sockmodel/main.go

// receive handles an arriving segment and describes the result. A segment
// belongs to the connection with its remote endpoint and local port. Only
// when no such connection exists, and the segment is a SYN, is the listener
// on the port consulted.
func (st *stack) receive(seg segment) string {
	id := key{remote: seg.from, port: seg.port}

	if s, ok := st.connections[id]; ok {
		return s.deliver(seg)
	}

	if !seg.syn {
		return "reset, no connection"
	}

	l, ok := st.listeners[seg.port]

	if !ok {
		return "reset, no listener on the port"
	}

	if len(l.queue) == l.backlog {
		return "dropped, the accept queue is full"
	}

	st.connections[id] = &socket{capacity: st.bufferSize}
	l.queue = append(l.queue, id)

	return fmt.Sprintf("established, accept queue %d", len(l.queue))
}

// deliver puts the data of a segment into the receive buffer, up to the
// window, and records a FIN.
func (s *socket) deliver(seg segment) string {
	if seg.fin {
		s.finished = true

		return "the sender's stream has ended"
	}

	accepted := min(len(seg.data), s.window())
	s.received = append(s.received, seg.data[:accepted]...)

	return fmt.Sprintf("accepted %d, window %d", accepted, s.window())
}

Listing 10.16 Finding the socket for a segment, in Rust. code/rust/sockmodel/src/main.rs

impl Stack {
    /// Handles a segment that arrives from `from` for local port `port`, and
    /// describes the result. A segment belongs to the connection with its
    /// remote endpoint and local port. Only when no such connection exists,
    /// and the segment is a SYN, is the listener on the port consulted.
    fn receive(&mut self, from: Endpoint, port: u16, segment: &Segment) -> String {
        let id = Key { remote: from, port };

        if let Some(socket) = self.connections.get_mut(&id) {
            return socket.deliver(segment);
        }

        if !segment.syn {
            return "reset, no connection".into();
        }

        let Some(listener) = self.listeners.get_mut(&port) else {
            return "reset, no listener on the port".into();
        };

        if listener.queue.len() == listener.backlog {
            return "dropped, the accept queue is full".into();
        }

        let socket = Socket {
            received: VecDeque::new(),
            capacity: self.buffer_size,
            finished: false,
        };

        self.connections.insert(id, socket);
        listener.queue.push_back(id);

        format!("established, accept queue {}", listener.queue.len())
    }
}

impl Socket {
    /// Puts the data of a segment into the receive buffer, up to the window,
    /// and records a FIN.
    fn deliver(&mut self, segment: &Segment) -> String {
        if segment.fin {
            self.finished = true;

            return "the sender's stream has ended".into();
        }

        let accepted = segment.data.len().min(self.window());
        self.received.extend(&segment.data.as_bytes()[..accepted]);

        format!("accepted {accepted}, window {}", self.window())
    }
}

deliver accepts data up to the window. window is the free space of the receive buffer, the value a real receiver reports to the sender.

accept takes connections in the order they were established. read returns the end of the stream only when the buffer is empty and the FIN, the segment that ends the sender’s stream, has arrived, so data that was sent before a close is not lost.

Listing 10.17 Accepting a connection, and reading from it, in Go. code/go/sockmodel/main.go

// accept removes the oldest connection from the accept queue of a port.
func (st *stack) accept(port int) (key, bool) {
	l, ok := st.listeners[port]

	if !ok || len(l.queue) == 0 {
		return key{}, false
	}

	id := l.queue[0]
	l.queue = l.queue[1:]

	return id, true
}

// read removes up to n bytes from the receive buffer. It returns 0 bytes
// with ended set when the buffer is empty and a FIN has arrived.
func (s *socket) read(n int) (data string, ended bool) {
	n = min(n, len(s.received))
	data, s.received = string(s.received[:n]), s.received[n:]

	return data, n == 0 && s.finished
}

Listing 10.18 Accepting a connection, and reading from it, in Rust. code/rust/sockmodel/src/main.rs

impl Stack {
    /// Removes the oldest connection from the accept queue of a port.
    fn accept(&mut self, port: u16) -> Option<Key> {
        self.listeners.get_mut(&port)?.queue.pop_front()
    }
}

impl Socket {
    /// Removes up to `n` bytes from the receive buffer. It returns 0 bytes
    /// with the second value set when the buffer is empty and a FIN has
    /// arrived.
    fn read(&mut self, n: usize) -> (String, bool) {
        let n = n.min(self.received.len());
        let data: Vec<u8> = self.received.drain(..n).collect();

        (
            String::from_utf8_lossy(&data).into_owned(),
            n == 0 && self.finished,
        )
    }
}

Listing 10.19 The trace, in Go. code/go/sockmodel/main.go

func main() {
	st := &stack{
		listeners:   map[int]*listener{80: {backlog: 2}},
		connections: map[key]*socket{},
		bufferSize:  8,
	}

	a, b, c := endpoint{"10.0.0.1", 5000}, endpoint{"10.0.0.2", 5000}, endpoint{"10.0.0.3", 5000}

	for _, seg := range []segment{{from: a, port: 80, syn: true}, {from: b, port: 80, syn: true},
		{from: c, port: 80, syn: true}, {from: c, port: 81, syn: true}} {
		fmt.Printf("SYN from %s to port %d: %s\n", seg.from, seg.port, st.receive(seg))
	}

	if id, ok := st.accept(80); ok {
		fmt.Printf("accept on port 80: the connection from %s\n", id.remote)
	}

	for _, from := range []endpoint{b, a, b, c} {
		fmt.Printf("5 bytes from %s to port 80: %s\n", from, st.receive(segment{from: from, port: 80, data: "hello"}))
	}

	conn, ok := st.connections[key{remote: b, port: 80}]

	if !ok {
		return
	}

	data, ended := conn.read(4)
	fmt.Printf("read of 4 bytes on the connection from %s: %q, window %d\n", b, data, conn.window())

	fmt.Printf("FIN from %s: %s\n", b, st.receive(segment{from: b, port: 80, fin: true}))

	data, ended = conn.read(16)
	fmt.Printf("read: %q, stream ended %v\n", data, ended)

	data, ended = conn.read(16)
	fmt.Printf("read: %q, stream ended %v\n", data, ended)
}

Listing 10.20 The trace, in Rust. code/rust/sockmodel/src/main.rs

fn endpoint(address: &'static str) -> Endpoint {
    Endpoint {
        address,
        port: 5000,
    }
}

fn main() {
    let listener = Listener {
        backlog: 2,
        queue: VecDeque::new(),
    };

    let mut stack = Stack {
        listeners: HashMap::from([(80, listener)]),
        connections: HashMap::new(),
        buffer_size: 8,
    };

    let (a, b, c) = (
        endpoint("10.0.0.1"),
        endpoint("10.0.0.2"),
        endpoint("10.0.0.3"),
    );

    let syn = Segment {
        syn: true,
        ..Segment::default()
    };

    for (from, port) in [(a, 80), (b, 80), (c, 80), (c, 81)] {
        let result = stack.receive(from, port, &syn);
        println!("SYN from {from} to port {port}: {result}");
    }

    if let Some(id) = stack.accept(80) {
        println!("accept on port 80: the connection from {}", id.remote);
    }

    let hello = Segment {
        data: "hello",
        ..Segment::default()
    };

    for from in [b, a, b, c] {
        let result = stack.receive(from, 80, &hello);
        println!("5 bytes from {from} to port 80: {result}");
    }

    let id = Key {
        remote: b,
        port: 80,
    };

    if let Some(socket) = stack.connections.get_mut(&id) {
        let (data, _) = socket.read(4);
        println!(
            "read of 4 bytes on the connection from {b}: {data:?}, window {}",
            socket.window()
        );
    }

    let fin = Segment {
        fin: true,
        ..Segment::default()
    };

    println!("FIN from {b}: {}", stack.receive(b, 80, &fin));

    if let Some(socket) = stack.connections.get_mut(&id) {
        for _ in 0..2 {
            let (data, ended) = socket.read(16);
            println!("read: {data:?}, stream ended {ended}");
        }
    }
}

SYN from 10.0.0.1:5000 to port 80: established, accept queue 1
SYN from 10.0.0.2:5000 to port 80: established, accept queue 2
SYN from 10.0.0.3:5000 to port 80: dropped, the accept queue is full
SYN from 10.0.0.3:5000 to port 81: reset, no listener on the port
accept on port 80: the connection from 10.0.0.1:5000
5 bytes from 10.0.0.2:5000 to port 80: accepted 5, window 3
5 bytes from 10.0.0.1:5000 to port 80: accepted 5, window 3
5 bytes from 10.0.0.2:5000 to port 80: accepted 3, window 0
5 bytes from 10.0.0.3:5000 to port 80: reset, no connection
read of 4 bytes on the connection from 10.0.0.2:5000: "hell", window 4
FIN from 10.0.0.2:5000: the sender's stream has ended
read: "ohel", stream ended false
read: "", stream ended true

Inspecting sockets on a live system

Recv-Q on an established connection is data that has arrived and that the program has not read. A value that stays high means the program is slow to read, and the sender is being held back by the window.

The complete programs

Listing 10.21 The complete sockfacts, in Go. code/go/sockfacts/main.go

// Command sockfacts measures TCP sockets on the loopback interface: when a
// connection is established, what identifies it, how written bytes arrive,
// what stops a writer whose peer does not read, and how each side learns
// that the other has closed.
//
// Run it from code/go with:
//
//	go run ./sockfacts
package main

import (
	"errors"
	"fmt"
	"log"

	"golang.org/x/sys/unix"
)


// listen returns a TCP socket that listens on a free port of 127.0.0.1, and
// its address.
func listen() (int, *unix.SockaddrInet4, error) {
	listener, err := unix.Socket(unix.AF_INET, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)

	if err != nil {
		return 0, nil, err
	}

	if err := unix.Bind(listener, &unix.SockaddrInet4{Addr: [4]byte{127, 0, 0, 1}}); err != nil {
		return 0, nil, err
	}

	if err := unix.Listen(listener, 16); err != nil {
		return 0, nil, err
	}

	address, err := unix.Getsockname(listener)

	if err != nil {
		return 0, nil, err
	}

	return listener, address.(*unix.SockaddrInet4), nil
}

// connect returns a TCP socket connected to an address.
func connect(address *unix.SockaddrInet4) (int, error) {
	client, err := unix.Socket(unix.AF_INET, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)

	if err != nil {
		return 0, err
	}

	return client, unix.Connect(client, address)
}

// name returns the symbolic name of a system call error, such as EPIPE.
func name(err error) string {
	var errno unix.Errno

	if errors.As(err, &errno) {
		return unix.ErrnoName(errno)
	}

	return fmt.Sprint(err)
}

// port returns the port of an address.
func port(address unix.Sockaddr, err error) int {
	if inet, ok := address.(*unix.SockaddrInet4); ok && err == nil {
		return inet.Port
	}

	return 0
}


// establish connects before the server calls accept, and compares the
// addresses of both ends.
func establish() error {
	listener, address, err := listen()

	if err != nil {
		return err
	}

	defer unix.Close(listener)

	client, err := connect(address)
	fmt.Printf("connect before the server calls accept: succeeded %v\n", err == nil)

	if err != nil {
		return err
	}

	defer unix.Close(client)

	server, _, err := unix.Accept(listener)

	if err != nil {
		return err
	}

	defer unix.Close(server)

	fmt.Printf("accept returns a new descriptor: %v, on the listener's port: %v\n",
		server != listener, port(unix.Getsockname(server)) == address.Port)
	fmt.Printf("the client's local port is the server's peer port: %v\n",
		port(unix.Getsockname(client)) == port(unix.Getpeername(server)))

	return stream(client, server)
}


// readFull reads until buf is full.
func readFull(fd int, buf []byte) error {
	for filled := 0; filled < len(buf); {
		n, err := unix.Read(fd, buf[filled:])

		if err != nil {
			return err
		}

		if n == 0 {
			return errors.New("the stream ended early")
		}

		filled += n
	}

	return nil
}

// stream writes "ab" and "cd" with 2 calls and reads 3 bytes and then 1.
// Then the client closes, and the server reads and writes again.
func stream(client, server int) error {
	for _, part := range []string{"ab", "cd"} {
		if _, err := unix.Write(client, []byte(part)); err != nil {
			return err
		}
	}

	first, second := make([]byte, 3), make([]byte, 1)

	if err := readFull(server, first); err != nil {
		return err
	}

	if err := readFull(server, second); err != nil {
		return err
	}

	fmt.Printf("writes of \"ab\" and \"cd\", reads of 3 bytes and 1 byte: %q and %q\n", first, second)

	unix.Close(client)

	n, err := unix.Read(server, first)

	if err != nil {
		return err
	}

	fmt.Printf("the client closed: read returns %d\n", n)

	for err == nil {
		_, err = unix.Write(server, []byte("x"))
	}

	fmt.Printf("writes to the closed peer: the first is accepted, a later one fails with %s\n", name(err))

	return nil
}


// fill writes blocks to a non-blocking socket until the kernel refuses one
// and returns the number of bytes accepted.
func fill(fd int) (int, error) {
	block := make([]byte, 1024)
	total := 0

	for {
		n, err := unix.Write(fd, block)

		if errors.Is(err, unix.EAGAIN) {
			return total, nil
		}

		if err != nil {
			return total, err
		}

		total += n
	}
}

// flow gives a connection small buffers, fills it while the server does not
// read, lets the server read everything, and fills it again.
func flow() error {
	listener, address, err := listen()

	if err != nil {
		return err
	}

	defer unix.Close(listener)

	if err := unix.SetsockoptInt(listener, unix.SOL_SOCKET, unix.SO_RCVBUF, 4096); err != nil {
		return err
	}

	client, err := connect(address)

	if err != nil {
		return err
	}

	defer unix.Close(client)

	if err := unix.SetsockoptInt(client, unix.SOL_SOCKET, unix.SO_SNDBUF, 4096); err != nil {
		return err
	}

	server, _, err := unix.Accept(listener)

	if err != nil {
		return err
	}

	defer unix.Close(server)

	if err := unix.SetNonblock(client, true); err != nil {
		return err
	}

	sent, err := fill(client)

	if err != nil {
		return err
	}

	fmt.Printf("the server does not read: the client is stopped after %d bytes\n", sent)

	if err := readFull(server, make([]byte, sent)); err != nil {
		return err
	}

	again, err := fill(client)

	if err != nil {
		return err
	}

	fmt.Printf("the server has read everything: the client can write again: %v\n", again > 0)

	return nil
}


func main() {
	for _, experiment := range []func() error{establish, flow} {
		if err := experiment(); err != nil {
			log.Fatal(err)
		}
	}
}

Listing 10.22 The complete sockfacts, in Rust. code/rust/sockfacts/src/main.rs

//! Measures TCP sockets on the loopback interface: when a connection is
//! established, what identifies it, how written bytes arrive, what stops a
//! writer whose peer does not read, and how each side learns that the other
//! has closed.
//!
//! Run it from code/rust with:
//!
//! ```text
//! cargo run -p sockfacts
//! ```

use std::io::{self, Read, Write};
use std::net::{TcpListener, TcpStream};
use std::os::fd::AsRawFd;

use nix::errno::Errno;
use nix::sys::socket::{setsockopt, sockopt};

/// Returns a TCP socket that listens on a free port of 127.0.0.1.
fn listen() -> io::Result<TcpListener> {
    TcpListener::bind("127.0.0.1:0")
}

/// Returns a TCP socket connected to the address of a listener.
fn connect(listener: &TcpListener) -> io::Result<TcpStream> {
    TcpStream::connect(listener.local_addr()?)
}

/// Returns the symbolic name of a system call error, such as `EPIPE`.
fn name(error: &io::Error) -> String {
    match error.raw_os_error() {
        Some(code) => format!("{:?}", Errno::from_raw(code)),
        None => error.to_string(),
    }
}

/// Connects before the server calls accept, and compares the addresses of
/// both ends.
fn establish() -> io::Result<()> {
    let listener = listen()?;

    let client = connect(&listener);
    println!(
        "connect before the server calls accept: succeeded {}",
        client.is_ok()
    );

    let client = client?;
    let (server, _) = listener.accept()?;

    println!(
        "accept returns a new descriptor: {}, on the listener's port: {}",
        server.as_raw_fd() != listener.as_raw_fd(),
        server.local_addr()?.port() == listener.local_addr()?.port()
    );
    println!(
        "the client's local port is the server's peer port: {}",
        client.local_addr()?.port() == server.peer_addr()?.port()
    );

    stream(client, server)
}

/// Writes "ab" and "cd" with 2 calls and reads 3 bytes and then 1. Then the
/// client closes, and the server reads and writes again.
fn stream(mut client: TcpStream, mut server: TcpStream) -> io::Result<()> {
    for part in ["ab", "cd"] {
        client.write_all(part.as_bytes())?;
    }

    let (mut first, mut second) = ([0u8; 3], [0u8; 1]);

    server.read_exact(&mut first)?;
    server.read_exact(&mut second)?;

    println!(
        "writes of \"ab\" and \"cd\", reads of 3 bytes and 1 byte: {:?} and {:?}",
        String::from_utf8_lossy(&first),
        String::from_utf8_lossy(&second)
    );

    drop(client);

    let n = server.read(&mut first)?;
    println!("the client closed: read returns {n}");

    let error = loop {
        if let Err(error) = server.write(b"x") {
            break error;
        }
    };

    println!(
        "writes to the closed peer: the first is accepted, a later one fails with {}",
        name(&error)
    );

    Ok(())
}

/// Writes blocks to a non-blocking socket until the kernel refuses one and
/// returns the number of bytes accepted.
fn fill(mut stream: &TcpStream) -> io::Result<usize> {
    let block = [0u8; 1024];
    let mut total = 0;

    loop {
        match stream.write(&block) {
            Ok(n) => total += n,
            Err(error) if error.kind() == io::ErrorKind::WouldBlock => return Ok(total),
            Err(error) => return Err(error),
        }
    }
}

/// Gives a connection small buffers, fills it while the server does not read,
/// lets the server read everything, and fills it again.
fn flow() -> io::Result<()> {
    let listener = listen()?;
    setsockopt(&listener, sockopt::RcvBuf, &4096)?;

    let client = connect(&listener)?;
    setsockopt(&client, sockopt::SndBuf, &4096)?;

    let (mut server, _) = listener.accept()?;

    client.set_nonblocking(true)?;

    let sent = fill(&client)?;
    println!("the server does not read: the client is stopped after {sent} bytes");

    server.read_exact(&mut vec![0u8; sent])?;

    let again = fill(&client)?;
    println!(
        "the server has read everything: the client can write again: {}",
        again > 0
    );

    Ok(())
}

fn main() -> io::Result<()> {
    establish()?;
    flow()
}

Listing 10.23 The complete sockmodel, in Go. code/go/sockmodel/main.go

// Command sockmodel models the receiving side of TCP in the kernel: a table
// of connections keyed by addresses and ports, listeners with an accept
// queue of limited length, and a receive buffer whose free space is the
// window offered to the sender.
//
// Run it from code/go with:
//
//	go run ./sockmodel
package main

import "fmt"


// An endpoint is an IP address and a port.
type endpoint struct {
	address string
	port    int
}

func (e endpoint) String() string {
	return fmt.Sprintf("%s:%d", e.address, e.port)
}

// A socket is one established connection: the bytes received and not yet
// read, the size of the receive buffer, and whether the sender has finished.
type socket struct {
	received []byte
	capacity int
	finished bool
}

// window is the free space of the receive buffer: the number of bytes the
// sender may still send.
func (s *socket) window() int {
	return s.capacity - len(s.received)
}

// A key identifies a connection on this machine: the remote endpoint and the
// local port. With the local address, which the model leaves out, it forms
// the 4 values that identify a TCP connection.
type key struct {
	remote endpoint
	port   int
}

// A listener accepts connections on a port. queue holds the connections that
// are established and have not been returned by accept.
type listener struct {
	backlog int
	queue   []key
}

// A stack is the TCP state of a machine.
type stack struct {
	listeners   map[int]*listener
	connections map[key]*socket
	bufferSize  int
}

// A segment is a packet of TCP. syn opens a connection and fin ends the
// sender's stream.
type segment struct {
	from     endpoint
	port     int
	syn, fin bool
	data     string
}


// receive handles an arriving segment and describes the result. A segment
// belongs to the connection with its remote endpoint and local port. Only
// when no such connection exists, and the segment is a SYN, is the listener
// on the port consulted.
func (st *stack) receive(seg segment) string {
	id := key{remote: seg.from, port: seg.port}

	if s, ok := st.connections[id]; ok {
		return s.deliver(seg)
	}

	if !seg.syn {
		return "reset, no connection"
	}

	l, ok := st.listeners[seg.port]

	if !ok {
		return "reset, no listener on the port"
	}

	if len(l.queue) == l.backlog {
		return "dropped, the accept queue is full"
	}

	st.connections[id] = &socket{capacity: st.bufferSize}
	l.queue = append(l.queue, id)

	return fmt.Sprintf("established, accept queue %d", len(l.queue))
}

// deliver puts the data of a segment into the receive buffer, up to the
// window, and records a FIN.
func (s *socket) deliver(seg segment) string {
	if seg.fin {
		s.finished = true

		return "the sender's stream has ended"
	}

	accepted := min(len(seg.data), s.window())
	s.received = append(s.received, seg.data[:accepted]...)

	return fmt.Sprintf("accepted %d, window %d", accepted, s.window())
}


// accept removes the oldest connection from the accept queue of a port.
func (st *stack) accept(port int) (key, bool) {
	l, ok := st.listeners[port]

	if !ok || len(l.queue) == 0 {
		return key{}, false
	}

	id := l.queue[0]
	l.queue = l.queue[1:]

	return id, true
}

// read removes up to n bytes from the receive buffer. It returns 0 bytes
// with ended set when the buffer is empty and a FIN has arrived.
func (s *socket) read(n int) (data string, ended bool) {
	n = min(n, len(s.received))
	data, s.received = string(s.received[:n]), s.received[n:]

	return data, n == 0 && s.finished
}


func main() {
	st := &stack{
		listeners:   map[int]*listener{80: {backlog: 2}},
		connections: map[key]*socket{},
		bufferSize:  8,
	}

	a, b, c := endpoint{"10.0.0.1", 5000}, endpoint{"10.0.0.2", 5000}, endpoint{"10.0.0.3", 5000}

	for _, seg := range []segment{{from: a, port: 80, syn: true}, {from: b, port: 80, syn: true},
		{from: c, port: 80, syn: true}, {from: c, port: 81, syn: true}} {
		fmt.Printf("SYN from %s to port %d: %s\n", seg.from, seg.port, st.receive(seg))
	}

	if id, ok := st.accept(80); ok {
		fmt.Printf("accept on port 80: the connection from %s\n", id.remote)
	}

	for _, from := range []endpoint{b, a, b, c} {
		fmt.Printf("5 bytes from %s to port 80: %s\n", from, st.receive(segment{from: from, port: 80, data: "hello"}))
	}

	conn, ok := st.connections[key{remote: b, port: 80}]

	if !ok {
		return
	}

	data, ended := conn.read(4)
	fmt.Printf("read of 4 bytes on the connection from %s: %q, window %d\n", b, data, conn.window())

	fmt.Printf("FIN from %s: %s\n", b, st.receive(segment{from: b, port: 80, fin: true}))

	data, ended = conn.read(16)
	fmt.Printf("read: %q, stream ended %v\n", data, ended)

	data, ended = conn.read(16)
	fmt.Printf("read: %q, stream ended %v\n", data, ended)
}

Listing 10.24 The complete sockmodel, in Rust. code/rust/sockmodel/src/main.rs

//! Models the receiving side of TCP in the kernel: a table of connections
//! keyed by addresses and ports, listeners with an accept queue of limited
//! length, and a receive buffer whose free space is the window offered to the
//! sender.
//!
//! Run it from code/rust with:
//!
//! ```text
//! cargo run -p sockmodel
//! ```

use std::collections::{HashMap, VecDeque};
use std::fmt;

/// An IP address and a port.
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
struct Endpoint {
    address: &'static str,
    port: u16,
}

impl fmt::Display for Endpoint {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}:{}", self.address, self.port)
    }
}

/// One established connection: the bytes received and not yet read, the size
/// of the receive buffer, and whether the sender has finished.
struct Socket {
    received: VecDeque<u8>,
    capacity: usize,
    finished: bool,
}

impl Socket {
    /// The free space of the receive buffer: the number of bytes the sender
    /// may still send.
    fn window(&self) -> usize {
        self.capacity - self.received.len()
    }
}

/// Identifies a connection on this machine: the remote endpoint and the local
/// port. With the local address, which the model leaves out, it forms the 4
/// values that identify a TCP connection.
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
struct Key {
    remote: Endpoint,
    port: u16,
}

/// Accepts connections on a port. `queue` holds the connections that are
/// established and have not been returned by accept.
struct Listener {
    backlog: usize,
    queue: VecDeque<Key>,
}

/// The TCP state of a machine.
struct Stack {
    listeners: HashMap<u16, Listener>,
    connections: HashMap<Key, Socket>,
    buffer_size: usize,
}

/// A packet of TCP. `syn` opens a connection and `fin` ends the sender's
/// stream.
#[derive(Default)]
struct Segment {
    syn: bool,
    fin: bool,
    data: &'static str,
}

impl Stack {
    /// Handles a segment that arrives from `from` for local port `port`, and
    /// describes the result. A segment belongs to the connection with its
    /// remote endpoint and local port. Only when no such connection exists,
    /// and the segment is a SYN, is the listener on the port consulted.
    fn receive(&mut self, from: Endpoint, port: u16, segment: &Segment) -> String {
        let id = Key { remote: from, port };

        if let Some(socket) = self.connections.get_mut(&id) {
            return socket.deliver(segment);
        }

        if !segment.syn {
            return "reset, no connection".into();
        }

        let Some(listener) = self.listeners.get_mut(&port) else {
            return "reset, no listener on the port".into();
        };

        if listener.queue.len() == listener.backlog {
            return "dropped, the accept queue is full".into();
        }

        let socket = Socket {
            received: VecDeque::new(),
            capacity: self.buffer_size,
            finished: false,
        };

        self.connections.insert(id, socket);
        listener.queue.push_back(id);

        format!("established, accept queue {}", listener.queue.len())
    }
}

impl Socket {
    /// Puts the data of a segment into the receive buffer, up to the window,
    /// and records a FIN.
    fn deliver(&mut self, segment: &Segment) -> String {
        if segment.fin {
            self.finished = true;

            return "the sender's stream has ended".into();
        }

        let accepted = segment.data.len().min(self.window());
        self.received.extend(&segment.data.as_bytes()[..accepted]);

        format!("accepted {accepted}, window {}", self.window())
    }
}

impl Stack {
    /// Removes the oldest connection from the accept queue of a port.
    fn accept(&mut self, port: u16) -> Option<Key> {
        self.listeners.get_mut(&port)?.queue.pop_front()
    }
}

impl Socket {
    /// Removes up to `n` bytes from the receive buffer. It returns 0 bytes
    /// with the second value set when the buffer is empty and a FIN has
    /// arrived.
    fn read(&mut self, n: usize) -> (String, bool) {
        let n = n.min(self.received.len());
        let data: Vec<u8> = self.received.drain(..n).collect();

        (
            String::from_utf8_lossy(&data).into_owned(),
            n == 0 && self.finished,
        )
    }
}

fn endpoint(address: &'static str) -> Endpoint {
    Endpoint {
        address,
        port: 5000,
    }
}

fn main() {
    let listener = Listener {
        backlog: 2,
        queue: VecDeque::new(),
    };

    let mut stack = Stack {
        listeners: HashMap::from([(80, listener)]),
        connections: HashMap::new(),
        buffer_size: 8,
    };

    let (a, b, c) = (
        endpoint("10.0.0.1"),
        endpoint("10.0.0.2"),
        endpoint("10.0.0.3"),
    );

    let syn = Segment {
        syn: true,
        ..Segment::default()
    };

    for (from, port) in [(a, 80), (b, 80), (c, 80), (c, 81)] {
        let result = stack.receive(from, port, &syn);
        println!("SYN from {from} to port {port}: {result}");
    }

    if let Some(id) = stack.accept(80) {
        println!("accept on port 80: the connection from {}", id.remote);
    }

    let hello = Segment {
        data: "hello",
        ..Segment::default()
    };

    for from in [b, a, b, c] {
        let result = stack.receive(from, 80, &hello);
        println!("5 bytes from {from} to port 80: {result}");
    }

    let id = Key {
        remote: b,
        port: 80,
    };

    if let Some(socket) = stack.connections.get_mut(&id) {
        let (data, _) = socket.read(4);
        println!(
            "read of 4 bytes on the connection from {b}: {data:?}, window {}",
            socket.window()
        );
    }

    let fin = Segment {
        fin: true,
        ..Segment::default()
    };

    println!("FIN from {b}: {}", stack.receive(b, 80, &fin));

    if let Some(socket) = stack.connections.get_mut(&id) {
        for _ in 0..2 {
            let (data, ended) = socket.read(16);
            println!("read: {data:?}, stream ended {ended}");
        }
    }
}

References

• tcp(7), socket(7) and listen(2), Linux manual pages: https://man7.org/linux/man-pages/man7/tcp.7.html
• W. R. Stevens, B. Fenner, A. M. Rudoff, UNIX Network Programming, Volume 1, 3rd edition, Addison-Wesley, 2003.
• RFC 9293, Transmission Control Protocol (TCP): https://www.rfc-editor.org/rfc/rfc9293
• net/ipv4/tcp_input.c and net/ipv4/inet_hashtables.c, Linux 5.15: https://elixir.bootlin.com/linux/v5.15/source/net/ipv4/tcp_input.c