Closing TCP connections
This chapter covers
- How each direction of a TCP connection closes with its own
FIN - The states of the side that closes first and of the side that closes second, read from the kernel
TIME_WAIT: what it protects, how long it lasts, and the port limit it puts on busy clientsSO_REUSEADDR, and why a restarted server can bind its portCLOSE_WAIT: the sockets a server keeps when it never closes, and how to find them- The reset that replaces
FINwhen a socket closes with unread data
Chapter 20 opened TCP connections and closed them with shutdown and drop. Each close sent a FIN, and
the other side’s read returned 0. That is all a program sees. The kernel, though, keeps each side of the
connection in a sequence of states after the program has moved on. Some of those states last a minute. One
lasts for as long as the program forgets to close the socket.
This chapter follows a connection through its close, as the kernel records it. The program, tcp_close,
reads each socket’s state from Linux’s /proc/net/tcp at every step, so its output comes from a Linux
container. It also shows the two production problems the close states cause, and the reset that a close
can send instead of a FIN.
27.1 Two directions, two FINs
A TCP connection carries bytes in both directions, and each direction closes on its own. A side that has
no more bytes to send sends a FIN, and the other side acknowledges it. After that, bytes can still
flow the other way. The connection is gone only when both directions have closed, which takes four
segments: FIN, ACK, FIN, ACK.
The side that sends the first FIN makes the active close. The other side makes the passive
close. They pass through different states (figure 27.1):
| Side | State | Waiting for |
|---|---|---|
| active | FIN_WAIT1 | the ACK of its FIN |
| active | FIN_WAIT2 | the peer’s FIN |
| active | TIME_WAIT | a fixed time, 60 seconds on Linux, before the connection is forgotten |
| passive | CLOSE_WAIT | its own program to close the socket |
| passive | LAST_ACK | the ACK of its FIN |
Two of these states wait on something other than the network. CLOSE_WAIT waits for the program, and
lasts until the program closes the socket. TIME_WAIT waits for a clock. Sections 27.2 and 27.3 cover each.
27.1.1 Reading the states
Linux lists every IPv4 TCP socket in the file /proc/net/tcp, one per line. The second and third columns
are the local and remote addresses, written as 0100007F:1F90: the IP address, then the port, in
hexadecimal. The fourth column is the state, as a number from 01 for ESTABLISHED to 0B for CLOSING.
The program parses the lines into the ports and a state name:
#[cfg(target_os = "linux")]
mod linux {
use std::{
collections::BTreeMap,
fs,
io::{self, Read, Write},
mem,
net::{SocketAddr, SocketAddrV4, TcpListener, TcpStream},
os::fd::{FromRawFd, OwnedFd},
thread,
time::Duration,
};
/// Long enough for loopback segments to be sent, received, and acknowledged.
pub const SETTLE: Duration = Duration::from_millis(50);
/// One line of /proc/net/tcp: local port, remote port, and state.
struct Entry {
local: u16,
remote: u16,
state: &'static str,
}
/// The name of a TCP state, from the number Linux prints for it.
fn state_name(code: u8) -> &'static str {
match code {
0x01 => "ESTABLISHED",
0x02 => "SYN_SENT",
0x03 => "SYN_RECV",
0x04 => "FIN_WAIT1",
0x05 => "FIN_WAIT2",
0x06 => "TIME_WAIT",
0x07 => "CLOSE",
0x08 => "CLOSE_WAIT",
0x09 => "LAST_ACK",
0x0A => "LISTEN",
0x0B => "CLOSING",
_ => "UNKNOWN",
}
}
/// The port in an address written as `0100007F:1F90`.
fn port_of(field: &str) -> Option<u16> {
let (_, port) = field.split_once(':')?;
u16::from_str_radix(port, 16).ok()
}
/// Every IPv4 TCP socket the kernel knows about.
fn entries() -> io::Result<Vec<Entry>> {
let table = fs::read_to_string("/proc/net/tcp")?;
Ok(table
.lines()
.skip(1)
.filter_map(|line| {
let fields: Vec<&str> = line.split_whitespace().collect();
Some(Entry {
local: port_of(fields.get(1)?)?,
remote: port_of(fields.get(2)?)?,
state: state_name(u8::from_str_radix(fields.get(3)?, 16).ok()?),
})
})
.collect())
}
// ...
}entries skips the header line, splits each line on whitespace, and keeps the three columns it needs. The
? operators inside filter_map skip any line that does not parse. All connections in this program run
over 127.0.0.1, so the two ports alone identify a socket.
state looks up one socket, and census counts the states of every socket on one server port:
mod linux {
// ...
/// The state of the socket whose local and remote ports are `local` and
/// `remote`, or "gone" if the kernel has forgotten it.
pub fn state(local: SocketAddr, remote: SocketAddr) -> io::Result<&'static str> {
Ok(entries()?
.into_iter()
.find(|entry| entry.local == local.port() && entry.remote == remote.port())
.map_or("gone", |entry| entry.state))
}
/// How many sockets on the server's `port` are in each state, counting
/// both the server's side and the clients' side of each connection.
pub fn census(port: u16) -> io::Result<BTreeMap<&'static str, usize>> {
let mut counts = BTreeMap::new();
for entry in entries()? {
if entry.state != "LISTEN" && (entry.local == port || entry.remote == port) {
*counts.entry(entry.state).or_insert(0) += 1;
}
}
Ok(counts)
}
/// A connected pair over loopback: (client, server's accepted socket).
pub fn connected(listener: &TcpListener) -> io::Result<(TcpStream, TcpStream)> {
let client = TcpStream::connect(listener.local_addr()?)?;
let (server, _) = listener.accept()?;
Ok((client, server))
}
// ...
}A socket the kernel has forgotten has no line in the file, and state reports it as gone. connected
returns both ends of a new connection, so one process can watch both sides.
27.1.2 A close, step by step
The first scene closes a connection the way chapter 20’s echo test did. The client half-closes with
shutdown(Write), the server reads to the end and replies, then both drop their sockets. After each step
the program waits 50 ms, so the loopback segments have arrived, and prints both states:
1. An orderly close, client first, with a half-close
step client server
connected ESTABLISHED ESTABLISHED
client shutdown(Write): FIN sent FIN_WAIT2 CLOSE_WAIT
server read to end, wrote a reply FIN_WAIT2 CLOSE_WAIT
server dropped its socket: FIN sent TIME_WAIT gone
client read the reply, dropped TIME_WAIT gone
The client never shows FIN_WAIT1. On loopback the ACK of its FIN arrived within microseconds, so by
the time the program looked, the client was in FIN_WAIT2. On a network with a 50 ms round trip,
FIN_WAIT1 would show.
The third row is the half-close at work. The client’s direction is closed, the server’s is open, and the
server’s reply still reached the client. After the server dropped its socket, the server’s side passed
through LAST_ACK and was gone. The client side went to TIME_WAIT, and stayed there after the client
dropped its socket. TIME_WAIT belongs to the kernel, not to the program.
Animation 27.1 follows these steps, then shows a server that never closes.
FIN closes one direction. A server that never closes leaves every connection in CLOSE_WAIT, holding a descriptor.27.2 TIME_WAIT
The side that closes first waits in TIME_WAIT before the kernel forgets the connection. The wait covers
two failures:
- A lost final
ACK. If the lastACKis lost, the peer, still inLAST_ACK, sends itsFINagain. A side inTIME_WAITstill knows the connection and answers with anotherACK. A side that had forgotten it would answer with a reset. The peer would then report an error for a connection that closed cleanly. - Old segments. A segment from the old connection can arrive late, delayed in some router. Suppose a new connection with the same four values, both addresses and both ports, exists by then. The old segment could be taken as part of the new stream.
TIME_WAITkeeps those four values reserved until such segments have expired.
The TCP standard sets the wait to twice the maximum segment lifetime. Linux fixes it at 60 seconds.
27.2.1 The cost: ports on a busy client
A socket in TIME_WAIT holds a few hundred bytes of kernel memory, which is cheap. The scarce resource is
the four values it reserves. A client uses a new local port for each connection to a server. The kernel picks it from the ephemeral port range: 32768 to 60999 on Linux, which is 28,232 ports. If
the client closes first, each port stays in TIME_WAIT for 60 seconds. The client can then open about 470 new connections per second to one server address and port. Beyond that, connect fails with EADDRNOTAVAIL.
Three fixes are common:
- Reuse connections. An HTTP client that keeps connections alive, or a connection pool, as in chapter 29, makes few connections and few closes.
- Let the server close first. Then
TIME_WAITlands on the server. All its connections share its one port, so the entries use up nothing. net.ipv4.tcp_tw_reuse. This Linux setting lets an outgoing connection reuse a port inTIME_WAITwhen TCP timestamps prove its segments are newer.
27.2.2 Binding a port with connections in TIME_WAIT
A server that closes first collects TIME_WAIT entries on its own port. If it restarts within a minute,
its new listening socket asks for a port that those entries still use. By default, bind refuses.
SO_REUSEADDR allows the bind as long as no socket on the port is listening. Rust’s TcpListener::bind
sets it on Unix, so the program uses raw calls to bind both ways:
unsafe extern "C" {
fn socket(domain: c_int, ty: c_int, protocol: c_int) -> c_int; // AF_INET, SOCK_STREAM, 0
fn setsockopt(
fd: c_int,
level: c_int, // SOL_SOCKET
name: c_int, // SO_REUSEADDR
value: *const c_void, // points at the option's value, a c_int here
len: socklen_t, // its size in bytes
) -> c_int;
fn bind(fd: c_int, addr: *const sockaddr, len: socklen_t) -> c_int;
}sockaddr is a generic address type. An IPv4 address is a sockaddr_in, passed as a pointer to sockaddr
with its own size:
mod linux {
// ...
/// Bind a listening socket to `port` on loopback with raw calls, setting
/// `SO_REUSEADDR` only if `reuse` is true. `TcpListener::bind` always sets it.
pub fn bind_raw(port: u16, reuse: bool) -> io::Result<OwnedFd> {
// SAFETY: socket takes no pointers, and the result is checked.
let fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_STREAM | libc::SOCK_CLOEXEC, 0) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
// SAFETY: the descriptor was just created, and nothing else owns it.
let socket = unsafe { OwnedFd::from_raw_fd(fd) };
let enable: libc::c_int = 1;
let address = SocketAddrV4::new([127, 0, 0, 1].into(), port);
// SAFETY: sockaddr_in is plain data, and all zeros is a valid value.
let mut raw: libc::sockaddr_in = unsafe { mem::zeroed() };
raw.sin_family = libc::AF_INET as libc::sa_family_t;
raw.sin_port = address.port().to_be();
raw.sin_addr.s_addr = u32::from_ne_bytes(address.ip().octets());
// SAFETY: the option value and the address are valid for the lengths
// passed, and outlive both calls.
let result = unsafe {
if reuse {
libc::setsockopt(
fd,
libc::SOL_SOCKET,
libc::SO_REUSEADDR,
(&enable as *const libc::c_int).cast(),
mem::size_of::<libc::c_int>() as libc::socklen_t,
);
}
libc::bind(
fd,
(&raw as *const libc::sockaddr_in).cast(),
mem::size_of::<libc::sockaddr_in>() as libc::socklen_t,
)
};
if result < 0 {
return Err(io::Error::last_os_error());
}
Ok(socket)
}
// ...
}The third scene closes one connection server-first, drops the listener, and binds the port again:
3. TIME_WAIT on the server's port, and binding it again
server closed first, then the client: {"TIME_WAIT": 1}
bind port again, SO_REUSEADDR false: Address already in use (os error 98)
bind port again, SO_REUSEADDR true : bound
One socket in TIME_WAIT was enough to stop a plain bind.
27.3 CLOSE_WAIT
The side that receives the first FIN moves to CLOSE_WAIT, and stays there until its own program
closes the socket. The kernel cannot leave CLOSE_WAIT on its own. Only the program knows whether it still
wants to send.
A server that never closes a connection after the client has gone leaves it in CLOSE_WAIT for as long as
the server runs. Each such connection holds a descriptor. The usual causes are a handler that ignores a read of 0, and a connection kept in a map that nothing removes. A reference cycle that keeps a socket’s owner alive does the same.
leaky_server keeps every accepted socket in a Vec, while each client connects and closes at once:
mod linux {
// ...
/// A server that accepts `clients` connections and never closes them,
/// while each client connects and closes at once. Returns the listener's
/// port and the accepted sockets, still open.
pub fn leaky_server(clients: usize) -> io::Result<(u16, Vec<TcpStream>)> {
let listener = TcpListener::bind("127.0.0.1:0")?;
let port = listener.local_addr()?.port();
let mut kept = Vec::new();
for _ in 0..clients {
let (client, server) = connected(&listener)?;
drop(client);
kept.push(server);
}
thread::sleep(SETTLE);
Ok((port, kept))
}
// ...
}2. A server that never closes what clients closed
while the server keeps 5 sockets: {"CLOSE_WAIT": 5, "FIN_WAIT2": 5}
after the server drops them: {"TIME_WAIT": 5}
Five connections waited at the halfway point of figure 27.1. The clients were in FIN_WAIT2, waiting for a FIN, and the server in CLOSE_WAIT, waiting for itself. When the server dropped the Vec, it sent five
FINs, and the clients’ sides moved to TIME_WAIT. The clients had already closed their descriptors.
Linux keeps such an orphaned FIN_WAIT2 socket for 60 seconds (tcp_fin_timeout), then drops it.
In production, ss -tan state close-wait lists the CLOSE_WAIT sockets of every process. A count that only grows is a leak in the program that owns them. It ends in EMFILE, “Too many open files”, as in section 24.1.
27.4 Reset instead of FIN
A socket that is closed while its receive buffer still holds unread bytes does not send FIN. The kernel
sends a reset (RST) instead, which tells the peer that data was thrown away. The peer’s next read
fails with ECONNRESET, and any bytes the peer had not yet read from its own buffer are discarded.
mod linux {
// ...
/// The client writes bytes the server never reads, then the server closes.
/// Returns what the client's next read reports.
pub fn close_with_unread_data() -> io::Result<io::Result<usize>> {
let listener = TcpListener::bind("127.0.0.1:0")?;
let (mut client, server) = connected(&listener)?;
client.write_all(b"a request the server never reads")?;
thread::sleep(SETTLE);
drop(server);
thread::sleep(SETTLE);
Ok(client.read(&mut [0; 64]))
}
}4. Closing with unread data
client read failed: Connection reset by peer (os error 104)
This bites HTTP servers. Suppose a server sends an error response and closes before reading the whole request body. The reset can arrive while the response is still on its way. The client then reports a reset
instead of the response. Servers avoid it with a lingering close. After the response they call shutdown(Write), read and discard the rest of the request for a short time, then close.
A program can also ask for a reset on purpose, by setting SO_LINGER with a timeout of 0 before closing.
The connection skips TIME_WAIT, and the peer gets ECONNRESET. Load balancers use it to drop connections
quickly. It also discards any unsent data, so it is a poor default.
27.5 The complete program
Listing 27.1 The states of a closing connection, read from /proc/net/tcp. src/bin/tcp_close.rs
//! Closing a TCP connection: the states each side passes through, `TIME_WAIT`
//! on the side that closes first, the `CLOSE_WAIT` sockets a server keeps when
//! it never closes, and the reset that replaces `FIN` when data goes unread.
//!
//! Linux only: the states are read from /proc/net/tcp.
//! Run with: cargo run --bin tcp_close
#[cfg(target_os = "linux")]
mod linux {
use std::{
collections::BTreeMap,
fs,
io::{self, Read, Write},
mem,
net::{SocketAddr, SocketAddrV4, TcpListener, TcpStream},
os::fd::{FromRawFd, OwnedFd},
thread,
time::Duration,
};
/// Long enough for loopback segments to be sent, received, and acknowledged.
pub const SETTLE: Duration = Duration::from_millis(50);
/// One line of /proc/net/tcp: local port, remote port, and state.
struct Entry {
local: u16,
remote: u16,
state: &'static str,
}
/// The name of a TCP state, from the number Linux prints for it.
fn state_name(code: u8) -> &'static str {
match code {
0x01 => "ESTABLISHED",
0x02 => "SYN_SENT",
0x03 => "SYN_RECV",
0x04 => "FIN_WAIT1",
0x05 => "FIN_WAIT2",
0x06 => "TIME_WAIT",
0x07 => "CLOSE",
0x08 => "CLOSE_WAIT",
0x09 => "LAST_ACK",
0x0A => "LISTEN",
0x0B => "CLOSING",
_ => "UNKNOWN",
}
}
/// The port in an address written as `0100007F:1F90`.
fn port_of(field: &str) -> Option<u16> {
let (_, port) = field.split_once(':')?;
u16::from_str_radix(port, 16).ok()
}
/// Every IPv4 TCP socket the kernel knows about.
fn entries() -> io::Result<Vec<Entry>> {
let table = fs::read_to_string("/proc/net/tcp")?;
Ok(table
.lines()
.skip(1)
.filter_map(|line| {
let fields: Vec<&str> = line.split_whitespace().collect();
Some(Entry {
local: port_of(fields.get(1)?)?,
remote: port_of(fields.get(2)?)?,
state: state_name(u8::from_str_radix(fields.get(3)?, 16).ok()?),
})
})
.collect())
}
/// The state of the socket whose local and remote ports are `local` and
/// `remote`, or "gone" if the kernel has forgotten it.
pub fn state(local: SocketAddr, remote: SocketAddr) -> io::Result<&'static str> {
Ok(entries()?
.into_iter()
.find(|entry| entry.local == local.port() && entry.remote == remote.port())
.map_or("gone", |entry| entry.state))
}
/// How many sockets on the server's `port` are in each state, counting
/// both the server's side and the clients' side of each connection.
pub fn census(port: u16) -> io::Result<BTreeMap<&'static str, usize>> {
let mut counts = BTreeMap::new();
for entry in entries()? {
if entry.state != "LISTEN" && (entry.local == port || entry.remote == port) {
*counts.entry(entry.state).or_insert(0) += 1;
}
}
Ok(counts)
}
/// A connected pair over loopback: (client, server's accepted socket).
pub fn connected(listener: &TcpListener) -> io::Result<(TcpStream, TcpStream)> {
let client = TcpStream::connect(listener.local_addr()?)?;
let (server, _) = listener.accept()?;
Ok((client, server))
}
/// Bind a listening socket to `port` on loopback with raw calls, setting
/// `SO_REUSEADDR` only if `reuse` is true. `TcpListener::bind` always sets it.
pub fn bind_raw(port: u16, reuse: bool) -> io::Result<OwnedFd> {
// SAFETY: socket takes no pointers, and the result is checked.
let fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_STREAM | libc::SOCK_CLOEXEC, 0) };
if fd < 0 {
return Err(io::Error::last_os_error());
}
// SAFETY: the descriptor was just created, and nothing else owns it.
let socket = unsafe { OwnedFd::from_raw_fd(fd) };
let enable: libc::c_int = 1;
let address = SocketAddrV4::new([127, 0, 0, 1].into(), port);
// SAFETY: sockaddr_in is plain data, and all zeros is a valid value.
let mut raw: libc::sockaddr_in = unsafe { mem::zeroed() };
raw.sin_family = libc::AF_INET as libc::sa_family_t;
raw.sin_port = address.port().to_be();
raw.sin_addr.s_addr = u32::from_ne_bytes(address.ip().octets());
// SAFETY: the option value and the address are valid for the lengths
// passed, and outlive both calls.
let result = unsafe {
if reuse {
libc::setsockopt(
fd,
libc::SOL_SOCKET,
libc::SO_REUSEADDR,
(&enable as *const libc::c_int).cast(),
mem::size_of::<libc::c_int>() as libc::socklen_t,
);
}
libc::bind(
fd,
(&raw as *const libc::sockaddr_in).cast(),
mem::size_of::<libc::sockaddr_in>() as libc::socklen_t,
)
};
if result < 0 {
return Err(io::Error::last_os_error());
}
Ok(socket)
}
/// A server that accepts `clients` connections and never closes them,
/// while each client connects and closes at once. Returns the listener's
/// port and the accepted sockets, still open.
pub fn leaky_server(clients: usize) -> io::Result<(u16, Vec<TcpStream>)> {
let listener = TcpListener::bind("127.0.0.1:0")?;
let port = listener.local_addr()?.port();
let mut kept = Vec::new();
for _ in 0..clients {
let (client, server) = connected(&listener)?;
drop(client);
kept.push(server);
}
thread::sleep(SETTLE);
Ok((port, kept))
}
/// The client writes bytes the server never reads, then the server closes.
/// Returns what the client's next read reports.
pub fn close_with_unread_data() -> io::Result<io::Result<usize>> {
let listener = TcpListener::bind("127.0.0.1:0")?;
let (mut client, server) = connected(&listener)?;
client.write_all(b"a request the server never reads")?;
thread::sleep(SETTLE);
drop(server);
thread::sleep(SETTLE);
Ok(client.read(&mut [0; 64]))
}
}
#[cfg(target_os = "linux")]
fn main() -> std::io::Result<()> {
use std::{
io::{Read, Write},
net::{Shutdown, TcpListener},
thread,
};
use linux::{SETTLE, census, connected, state};
println!("1. An orderly close, client first, with a half-close\n");
let listener = TcpListener::bind("127.0.0.1:0")?;
let (mut client, mut server) = connected(&listener)?;
let (c, s) = (client.local_addr()?, server.local_addr()?);
let show = |step: &str| -> std::io::Result<()> {
thread::sleep(SETTLE);
println!(" {step:<38} {:<12} {}", state(c, s)?, state(s, c)?);
Ok(())
};
println!(" {:<38} {:<12} server", "step", "client");
show("connected")?;
client.shutdown(Shutdown::Write)?;
show("client shutdown(Write): FIN sent")?;
let mut request = Vec::new();
server.read_to_end(&mut request)?;
server.write_all(b"reply")?;
show("server read to end, wrote a reply")?;
drop(server);
show("server dropped its socket: FIN sent")?;
let mut reply = String::new();
client.read_to_string(&mut reply)?;
drop(client);
show("client read the reply, dropped")?;
println!("\n2. A server that never closes what clients closed\n");
let (port, kept) = linux::leaky_server(5)?;
println!(" while the server keeps 5 sockets: {:?}", census(port)?);
drop(kept);
thread::sleep(SETTLE);
println!(" after the server drops them: {:?}", census(port)?);
println!("\n3. TIME_WAIT on the server's port, and binding it again\n");
let listener = TcpListener::bind("127.0.0.1:0")?;
let port = listener.local_addr()?.port();
let (client, server) = connected(&listener)?;
drop(server);
thread::sleep(SETTLE);
drop(client);
drop(listener);
thread::sleep(SETTLE);
println!(
" server closed first, then the client: {:?}",
census(port)?
);
for reuse in [false, true] {
let outcome = match linux::bind_raw(port, reuse) {
Ok(_) => "bound".to_string(),
Err(error) => error.to_string(),
};
println!(" bind port again, SO_REUSEADDR {reuse:<5}: {outcome}");
}
println!("\n4. Closing with unread data\n");
match linux::close_with_unread_data()? {
Ok(read) => println!(" client read returned {read}"),
Err(error) => println!(" client read failed: {error}"),
}
Ok(())
}
#[cfg(not(target_os = "linux"))]
fn main() {
eprintln!("tcp_close reads TCP states from Linux's /proc/net/tcp; run it on Linux");
}
#[cfg(all(test, target_os = "linux"))]
mod tests {
use std::{io::ErrorKind, net::TcpListener, thread};
use super::linux::*;
#[test]
fn the_side_that_closes_first_holds_time_wait() {
let listener = TcpListener::bind("127.0.0.1:0").expect("bind");
let (client, server) = connected(&listener).expect("connect");
let (c, s) = (client.local_addr().unwrap(), server.local_addr().unwrap());
drop(server);
thread::sleep(SETTLE);
drop(client);
thread::sleep(SETTLE);
assert_eq!(state(s, c).expect("state"), "TIME_WAIT");
assert_eq!(state(c, s).expect("state"), "gone");
}
#[test]
fn a_server_that_never_closes_holds_close_wait() {
let (port, _kept) = leaky_server(3).expect("server");
assert_eq!(
census(port)
.expect("census")
.get("CLOSE_WAIT"),
Some(&3)
);
}
#[test]
fn unread_data_turns_close_into_reset() {
let error = close_with_unread_data()
.expect("setup")
.expect_err("read should fail");
assert_eq!(error.kind(), ErrorKind::ConnectionReset);
}
}$ cargo run --bin tcp_close
1. An orderly close, client first, with a half-close
step client server
connected ESTABLISHED ESTABLISHED
client shutdown(Write): FIN sent FIN_WAIT2 CLOSE_WAIT
server read to end, wrote a reply FIN_WAIT2 CLOSE_WAIT
server dropped its socket: FIN sent TIME_WAIT gone
client read the reply, dropped TIME_WAIT gone
2. A server that never closes what clients closed
while the server keeps 5 sockets: {"CLOSE_WAIT": 5, "FIN_WAIT2": 5}
after the server drops them: {"TIME_WAIT": 5}
3. TIME_WAIT on the server's port, and binding it again
server closed first, then the client: {"TIME_WAIT": 1}
bind port again, SO_REUSEADDR false: Address already in use (os error 98)
bind port again, SO_REUSEADDR true : bound
4. Closing with unread data
client read failed: Connection reset by peer (os error 104)
27.6 Questions that come up
“A server has 30,000 sockets in TIME_WAIT. Is that a problem?”
Usually not on the server. They share the server’s port and cost little memory. It is a problem on a
client, where each one holds an ephemeral port to the same destination.
“A server has 3,000 sockets in CLOSE_WAIT and the number keeps rising. What happened?”
The clients closed, and the server never closed its side. Look for a code path that returns without
dropping the connection, or a collection that keeps connections after they end.
“Why not shorten TIME_WAIT?”
Linux has no setting for it: the 60 seconds is a constant in the kernel. Reusing connections removes the need, and tcp_tw_reuse covers the outgoing case safely.
“What is the difference between shutdown(Write) and close?”
shutdown(Write) sends FIN and keeps the socket open for reading. close releases the descriptor. If no
other descriptor refers to the socket, the kernel sends FIN, or a reset if unread data remains.
“Both sides send FIN at the same moment. What happens?”
Both go from FIN_WAIT1 to CLOSING, a state for this simultaneous close, then both to TIME_WAIT.
Summary
- Each direction of a connection closes with its own
FINandACK, four segments in all. - The side that closes first passes through
FIN_WAIT1,FIN_WAIT2, andTIME_WAIT. The other passes throughCLOSE_WAITandLAST_ACK. TIME_WAITlasts 60 seconds on Linux. It answers a retransmittedFINand keeps late segments out of a new connection with the same addresses and ports.- On a client,
TIME_WAITholds ephemeral ports: about 470 new connections per second to one server. Reusing connections is the fix. SO_REUSEADDRlets a server bind a port that has connections inTIME_WAIT.TcpListener::bindsets it.CLOSE_WAITlasts until the program closes the socket. A growing count is a leak of descriptors.- Closing with unread data sends a reset, and the peer’s
readfails withECONNRESET.
Chapter 28 adds encryption to a connection. It runs TLS with rustls, checks the server’s certificate,
and then has the server check a certificate from the client.
Exercises
- Add
thread::sleep(Duration::from_secs(61))at the end of the first scene and print the client’s state again. What does it show? - Change
leaky_serverso the server closes first and the clients never do. Which side holdsFIN_WAIT2, which holdsCLOSE_WAIT, and which ends up inTIME_WAIT? - Open 1,000 connections to one listener in a loop, closing each from the client side, and print the
censusafterwards. Then make the server close first and compare. - In
close_with_unread_data, read the client’s bytes on the server before dropping it. What does the client’sreadreturn now? - Set
SO_LINGERto{ l_onoff: 1, l_linger: 0 }on the server’s socket withlibc::setsockoptbefore dropping it in the first scene. Which states disappear, and what does the client’sreadreport?