Parsing and serving HTTP/1.1

HTTP looks like text with line breaks, but it is a byte stream whose message boundaries are decided by headers. The security bugs in real servers come from disagreeing about where one request ends and the next begins. The parser in this chapter is built around that boundary. The server on top of it keeps protocol handling separate from routing, so each can be tested alone.

21.1 Requests and responses on the wire

HTTP/1.1 runs over one TCP connection, the byte stream from chapter 20. The client writes a request into the connection. The server reads it, works out an answer, and writes a response back on the same connection.

A request is a few lines of text, then an optional body. Every line ends with two bytes, a carriage return and a line feed, written \r\n in Rust strings and called CRLF. Figure 21.1 shows a request that creates a GPU workload, byte for byte.

The first line is the request line. It has three parts separated by spaces:

• The method says what the client wants done. GET reads, POST creates, and DELETE removes.
• The target names the thing to act on, here the path /v1/gpu-workloads.
• The version is HTTP/1.1.

Each following line is a header: a name, a colon, and a value. Host names the server the client meant. Content-Type says what kind of data the body holds. Content-Length says how many bytes the body has, here 14.

An empty line ends the head. The body follows it, and it is exactly Content-Length bytes long: {"gpuCount":1} is 14 bytes.

A response has the same shape. Its first line is a status line: the version, a three-digit status code, and a short reason phrase. Codes from 200 to 299 mean success, 400 to 499 mean the client sent something wrong, and 500 to 599 mean the server failed.

HTTP/1.1 201 Created
Content-Length: 49
Content-Type: application/json
Connection: keep-alive

{"id":"wl-a100-2","gpuCount":1,"state":"Pending"}

21.1.1 Where one request ends

TCP delivers bytes, not messages. As chapter 20 showed, one read can return part of a request, exactly one request, or one request and the start of the next. The bytes carry no marker that says “a request starts here”. The server has to find the end of each request from the bytes themselves:

1. The head ends at the first empty line, the four bytes \r\n\r\n.
2. The head then says how long the body is. Content-Length gives a byte count. Section 21.2.4 covers the other way, a body sent in sized pieces.

The next request depends on it too. HTTP/1.1 keeps a connection open after a response, so the client can send another request without a new TCP handshake. This is called keep-alive. The server can read the second request only if it knows exactly where the first one stopped.

21.1.2 The parts of the server

Figure 21.2 shows the pieces of the server in this chapter and the order in which a request passes through them.

• The server reads bytes from the connection and appends them to a buffer.
• parse_request looks at the whole buffer. If the request is not complete yet, it says Incomplete, and the server reads more. If it is complete, it returns a Request value.
• route decides the answer from the Request: a status code and a body. It never sees bytes or sockets.
• build_response turns the answer into the bytes of a response, which the server writes back.

The parser, the read loop, and the response bytes are the protocol. route is the application. Keeping them apart means each can be tested without the other: the parser with byte strings, and route with Request values.

21.2 The parser

The parser is one file, src/problems/http_request.rs. Listing 21.1 in section 21.4 is the whole file.

21.2.1 Types first

The file opens with its vocabulary. Method is an enum with a parse that returns None for an unknown token, and two predicates from RFC 9110:

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Method {
    Get,
    Head,
    Post,
    Put,
    Patch,
    Delete,
    Options,
}

impl Method {
    pub fn parse(token: &str) -> Option<Self> {
        Some(match token {
            "GET" => Self::Get,
            "HEAD" => Self::Head,
            "POST" => Self::Post,
            "PUT" => Self::Put,
            "PATCH" => Self::Patch,
            "DELETE" => Self::Delete,
            "OPTIONS" => Self::Options,
            _ => return None,
        })
    }

    pub fn is_safe(self) -> bool {
        matches!(self, Self::Get | Self::Head | Self::Options)
    }

    pub fn is_idempotent(self) -> bool {
        matches!(
            self,
            Self::Get | Self::Head | Self::Options | Self::Put | Self::Delete
        )
    }
}

is_safe is true for GET, HEAD, and OPTIONS, which do not change state. is_idempotent adds PUT and DELETE to those. Chapter 19’s retry loop is the consumer that needs is_idempotent.

Version has two variants because the parser accepts two. Limits carries the header and body caps, with defaults of 16 KiB and 8 MiB:

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Version {
    Http10,
    Http11,
}

#[derive(Debug, Clone, Copy)]
pub struct Limits {
    pub max_header_bytes: usize,
    pub max_body_bytes: usize,
}

impl Default for Limits {
    fn default() -> Self {
        Self {
            max_header_bytes: 16 * 1024,
            max_body_bytes: 8 * 1024 * 1024,
        }
    }
}

Request holds what the parser found:

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Request {
    pub method: Method,
    pub target: String,
    pub version: Version,
    pub headers: Vec<(String, String)>,
    pub body: Vec<u8>,
}

impl Request {
    /// First header value matching `name`, case-insensitively.
    pub fn header(&self, name: &str) -> Option<&str> {
        self.headers
            .iter()
            .find(|(key, _)| key.eq_ignore_ascii_case(name))
            .map(|(_, value)| value.as_str())
    }

    /// HTTP/1.1 reuses the connection unless `Connection: close`; HTTP/1.0 only
    /// reuses it with `Connection: keep-alive`.
    pub fn should_keep_alive(&self) -> bool {
        let connection = self.header("connection").unwrap_or("");
        let has = |token: &str| {
            connection
                .split(',')
                .any(|part| part.trim().eq_ignore_ascii_case(token))
        };
        match self.version {
            Version::Http11 => !has("close"),
            Version::Http10 => has("keep-alive"),
        }
    }
}

Request::header finds a header case-insensitively, because header names are case-insensitive. Then should_keep_alive encodes the one rule that differs by version. HTTP/1.1 keeps the connection open unless the client sends Connection: close. HTTP/1.0 closes it unless the client sends Connection: keep-alive. The has closure splits the header on commas because Connection carries a list.

ParseError distinguishes the cases a server must treat differently:

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ParseError {
    /// Not a complete request yet; read more and retry.
    Incomplete,
    Malformed(&'static str),
    MissingHost,
    /// `Content-Length` and `Transfer-Encoding` were both present.
    AmbiguousBodyLength,
    UnsupportedTransferEncoding,
    PayloadTooLarge,
}

Incomplete is not a failure: it means “read more bytes and try again”, and the server’s loop in section 21.3 depends on it. PayloadTooLarge maps to status 413, MissingHost and Malformed to 400.

21.2.2 The head of the request

parse_request works on &[u8], not &str, because an HTTP body is arbitrary bytes and only the head is text. It first looks for the blank line that ends the head, \r\n\r\n, with windows(4).position(...):

/// Parse one request from the start of `input`.
pub fn parse_request(input: &[u8], limits: Limits) -> Result<Request, ParseError> {
    let Some(header_end) = input
        .windows(4)
        .position(|window| window == b"\r\n\r\n")
    else {
        return Err(if input.len() > limits.max_header_bytes {
            ParseError::PayloadTooLarge
        } else {
            ParseError::Incomplete
        });
    };
    if header_end + 4 > limits.max_header_bytes {
        return Err(ParseError::PayloadTooLarge);
    }

    let head = &input[..header_end];
    let body_start = header_end + 4;
    // ...
}

If the blank line is missing, the request is Incomplete. The exception is a buffer already larger than max_header_bytes: then the client is sending an endless header and gets PayloadTooLarge. That check stops a slow client from holding memory with a header that never ends. Figure 21.3 shows the order of the remaining checks.

The request line must be exactly three space-separated parts: a known method, a non-empty target, and HTTP/1.1 or HTTP/1.0. A fourth part is an error.

pub fn parse_request(input: &[u8], limits: Limits) -> Result<Request, ParseError> {
    // ...
    let mut lines = head
        .split(|byte| *byte == b'\n')
        .map(|line| line.strip_suffix(b"\r").unwrap_or(line));

    let request_line = std::str::from_utf8(
        lines
            .next()
            .ok_or(ParseError::Malformed("empty"))?,
    )
    .map_err(|_| ParseError::Malformed("request line"))?;
    let mut parts = request_line.split(' ');
    let method = Method::parse(
        parts
            .next()
            .ok_or(ParseError::Malformed("method"))?,
    )
    .ok_or(ParseError::Malformed("unknown method"))?;
    let target = parts
        .next()
        .ok_or(ParseError::Malformed("target"))?;
    let version = match parts.next() {
        Some("HTTP/1.1") => Version::Http11,
        Some("HTTP/1.0") => Version::Http10,
        _ => return Err(ParseError::Malformed("version")),
    };
    if parts.next().is_some() || target.is_empty() {
        return Err(ParseError::Malformed("request line"));
    }
    // ...
}

Header lines are split at the first colon. A name with whitespace is rejected, because RFC 9112 forbids whitespace between the name and the colon. A line that starts with a space or tab is obsolete line folding, a continuation syntax the RFC deprecates. Since different servers interpret it differently, the parser rejects it.

pub fn parse_request(input: &[u8], limits: Limits) -> Result<Request, ParseError> {
    // ...
    let mut headers = Vec::new();
    for line in lines {
        if line.is_empty() {
            continue;
        }
        if matches!(line.first(), Some(b' ') | Some(b'\t')) {
            return Err(ParseError::Malformed("obsolete line folding"));
        }
        let colon = line
            .iter()
            .position(|byte| *byte == b':')
            .ok_or(ParseError::Malformed("header without colon"))?;
        let name = std::str::from_utf8(&line[..colon])
            .map_err(|_| ParseError::Malformed("header name"))?;
        if name.is_empty()
            || name
                .bytes()
                .any(|byte| byte.is_ascii_whitespace())
        {
            return Err(ParseError::Malformed("invalid header name"));
        }
        let value = std::str::from_utf8(&line[colon + 1..])
            .map_err(|_| ParseError::Malformed("header value"))?;
        headers.push((name.to_string(), value.trim().to_string()));
    }
    // ...
}

HTTP/1.1 requires exactly one Host header. The count uses filter(...).count() != 1, which rejects both zero and two.

pub fn parse_request(input: &[u8], limits: Limits) -> Result<Request, ParseError> {
    // ...
    if version == Version::Http11
        && headers
            .iter()
            .filter(|(name, _)| name.eq_ignore_ascii_case("host"))
            .count()
            != 1
    {
        return Err(ParseError::MissingHost);
    }
    // ...
}

21.2.3 Body framing and request smuggling

parse_body decides how long the body is. A request with both Content-Length and Transfer-Encoding is rejected as AmbiguousBodyLength:

fn parse_body(
    input: &[u8],
    body_start: usize,
    headers: &[(String, String)],
    limits: Limits,
) -> Result<Vec<u8>, ParseError> {
    let collect = |name: &str| -> Vec<&str> {
        headers
            .iter()
            .filter(|(key, _)| key.eq_ignore_ascii_case(name))
            .map(|(_, value)| value.as_str())
            .collect()
    };
    let transfer_encoding = collect("transfer-encoding");
    let content_length = collect("content-length");

    // Two framing headers is the classic request-smuggling setup.
    if !transfer_encoding.is_empty() && !content_length.is_empty() {
        return Err(ParseError::AmbiguousBodyLength);
    }
    // ...
}

The reason is request smuggling (figure 21.4). A front-end proxy may frame a request by Content-Length, while the back-end server frames the same bytes by Transfer-Encoding: chunked. The two then disagree about where the request ends. The attacker places a second request in the part the back-end treats as “after the body”. The back-end runs it as though it arrived on its own, past whatever checks the proxy applied. Refusing ambiguous framing outright, and closing the connection, removes the disagreement.

Transfer-Encoding values from every such header are joined and split into codings. The parser supports only chunked, and requires it to be last, because a body whose final coding is not chunked has no defined end. Anything else is UnsupportedTransferEncoding.

fn parse_body(
    input: &[u8],
    body_start: usize,
    headers: &[(String, String)],
    limits: Limits,
) -> Result<Vec<u8>, ParseError> {
    // ...
    if !transfer_encoding.is_empty() {
        let joined = transfer_encoding.join(",");
        let codings: Vec<&str> = joined
            .split(',')
            .map(str::trim)
            .filter(|token| !token.is_empty())
            .collect();
        let all_chunked = codings
            .iter()
            .all(|token| token.eq_ignore_ascii_case("chunked"));
        if !all_chunked
            || !codings
                .last()
                .is_some_and(|t| t.eq_ignore_ascii_case("chunked"))
        {
            return Err(ParseError::UnsupportedTransferEncoding);
        }
        return decode_chunked(&input[body_start..], limits.max_body_bytes);
    }
    // ...
}

Multiple Content-Length headers are allowed only if they all agree. The check parses each value and compares with lengths.any(|length| length != Ok(first)), comparing Results directly. A length above max_body_bytes is rejected before the server waits for the bytes. A client cannot claim a huge body and make the server buffer it. If fewer bytes than the length have arrived, the result is Incomplete.

fn parse_body(
    input: &[u8],
    body_start: usize,
    headers: &[(String, String)],
    limits: Limits,
) -> Result<Vec<u8>, ParseError> {
    // ...
    if content_length.is_empty() {
        return Ok(Vec::new());
    }

    let mut lengths = content_length.iter().map(|value| {
        value
            .trim()
            .parse::<usize>()
            .map_err(|_| ParseError::Malformed("invalid Content-Length"))
    });
    let first = lengths.next().expect("non-empty")?;
    if lengths.any(|length| length != Ok(first)) {
        return Err(ParseError::Malformed("conflicting Content-Length"));
    }
    if first > limits.max_body_bytes {
        return Err(ParseError::PayloadTooLarge);
    }
    let end = body_start + first;
    if input.len() < end {
        return Err(ParseError::Incomplete);
    }
    Ok(input[body_start..end].to_vec())
}

21.2.4 Chunked bodies

decode_chunked walks a cursor through size-in-hex CRLF data CRLF records until a size of 0. Chunk extensions after a ; are ignored, as the RFC permits.

fn decode_chunked(data: &[u8], max_body: usize) -> Result<Vec<u8>, ParseError> {
    let mut cursor = 0;
    let mut body = Vec::new();

    loop {
        let line_end = find_crlf(data, cursor).ok_or(ParseError::Incomplete)?;
        let size_token = data[cursor..line_end]
            .split(|byte| *byte == b';')
            .next()
            .unwrap_or(&[]);
        let size = usize::from_str_radix(
            std::str::from_utf8(size_token)
                .map_err(|_| ParseError::Malformed("chunk size"))?
                .trim(),
            16,
        )
        .map_err(|_| ParseError::Malformed("chunk size"))?;
        cursor = line_end + 2;
        // ...
    }
}

A size of 0 ends the body, after optional trailer lines up to an empty line:

fn decode_chunked(data: &[u8], max_body: usize) -> Result<Vec<u8>, ParseError> {
    // ...
    loop {
        // ...
        if size == 0 {
            // Consume trailers up to the final empty line.
            loop {
                let trailer_end = find_crlf(data, cursor).ok_or(ParseError::Incomplete)?;
                let is_empty = trailer_end == cursor;
                cursor = trailer_end + 2;
                if is_empty {
                    return Ok(body);
                }
            }
        }
        // ...
    }
}

Each other chunk checks the running total against the body limit. It computes the chunk’s end with checked_add, returns Incomplete if the data has not all arrived, and insists on the CRLF after the data.

fn decode_chunked(data: &[u8], max_body: usize) -> Result<Vec<u8>, ParseError> {
    // ...
    loop {
        // ...
        if body.len() + size > max_body {
            return Err(ParseError::PayloadTooLarge);
        }
        let chunk_end = cursor
            .checked_add(size)
            .ok_or(ParseError::Malformed("chunk overflow"))?;
        if chunk_end + 2 > data.len() {
            return Err(ParseError::Incomplete);
        }
        body.extend_from_slice(&data[cursor..chunk_end]);
        if &data[chunk_end..chunk_end + 2] != b"\r\n" {
            return Err(ParseError::Malformed("chunk not CRLF-terminated"));
        }
        cursor = chunk_end + 2;
    }
}

find_crlf uses data.get(from..)? so that a cursor past the end is None rather than a panic.

fn find_crlf(data: &[u8], from: usize) -> Option<usize> {
    data.get(from..)?
        .windows(2)
        .position(|window| window == b"\r\n")
        .map(|offset| offset + from)
}

One arithmetic detail needs tightening. body.len() + size > max_body runs before the checked_add, and a chunk size near usize::MAX overflows that addition. from_str_radix parses such a size from sixteen fs. The form size > max_body - body.len() cannot overflow.

parse_request returns the request but not how many bytes it consumed. A caller therefore cannot find the start of a second, pipelined request in the same buffer. The server below accepts that limitation, and its module comment says so.

21.3 A small REST server

The server is src/bin/http_server.rs. Listing 21.2 in section 21.4 is the whole file.

21.3.1 Routing as a pure function

route(&Request) -> (u16, &'static str, Vec<u8>) decides the status, content type, and body without touching a socket:

/// Map a request to `(status, content_type, body)`.
///
/// Pure, so the routing rules are unit-testable without a socket.
pub fn route(request: &Request) -> (u16, &'static str, Vec<u8>) {
    let target = request.target.as_str();

    if target == "/healthz" {
        return match request.method {
            Method::Get | Method::Head => (200, "text/plain", b"ok\n".to_vec()),
            _ => method_not_allowed(),
        };
    }
    // ...
}

A known path with an unsupported method returns 405, an unknown path 404.

pub fn route(request: &Request) -> (u16, &'static str, Vec<u8>) {
    // ...
    if target == "/v1/gpu-workloads" {
        return match request.method {
            Method::Get | Method::Head => (
                200,
                "application/json",
                br#"[{"id":"wl-a100-1","gpuCount":1,"state":"Running"}]"#.to_vec(),
            ),
            Method::Post => (
                201,
                "application/json",
                br#"{"id":"wl-a100-2","gpuCount":1,"state":"Pending"}"#.to_vec(),
            ),
            _ => method_not_allowed(),
        };
    }
    // ...
}

For /v1/gpu-workloads/{id}, strip_prefix extracts the id, and an empty id or one containing / is a 404.

pub fn route(request: &Request) -> (u16, &'static str, Vec<u8>) {
    // ...
    if let Some(id) = target.strip_prefix("/v1/gpu-workloads/") {
        if id.is_empty() || id.contains('/') {
            return (404, "text/plain", b"not found\n".to_vec());
        }
        return match request.method {
            Method::Get | Method::Head => (
                200,
                "application/json",
                format!(r#"{{"id":"{id}","state":"Running"}}"#).into_bytes(),
            ),
            Method::Delete => (204, "", Vec::new()),
            _ => method_not_allowed(),
        };
    }

    (404, "text/plain", b"not found\n".to_vec())
}

Its doc comment gives the reason it is a separate function: it is unit-testable. The tests at the bottom of the file exercise every route by building a Request value directly.

build_response writes the status line, Content-Length, an optional Content-Type, and a Connection header that tells the client what the server will do:

fn build_response(status: u16, content_type: &str, body: &[u8], keep_alive: bool) -> Vec<u8> {
    let mut response = format!("HTTP/1.1 {} {}\r\n", status, status_reason(status));
    response.push_str(&format!("Content-Length: {}\r\n", body.len()));
    if !content_type.is_empty() {
        response.push_str(&format!("Content-Type: {content_type}\r\n"));
    }
    response.push_str(if keep_alive {
        "Connection: keep-alive\r\n"
    } else {
        "Connection: close\r\n"
    });
    response.push_str("\r\n");

    let mut bytes = response.into_bytes();
    bytes.extend_from_slice(body);
    bytes
}

21.3.2 The connection loop

handle_connection accumulates bytes in buffer and asks the parser what it has (figure 21.5).

On Ok, it routes, writes the response, and either closes or clears the buffer for the next request:

fn handle_connection(mut stream: TcpStream) -> std::io::Result<()> {
    stream.set_nodelay(true)?;
    let limits = Limits::default();
    let mut buffer = Vec::new();
    let mut chunk = [0u8; 4096];

    loop {
        match parse_request(&buffer, limits) {
            Ok(request) => {
                let keep_alive = request.should_keep_alive();
                let (status, content_type, body) = route(&request);
                let response = build_response(status, content_type, &body, keep_alive);
                stream.write_all(&response)?;

                if !keep_alive {
                    return Ok(());
                }
                buffer.clear();
            }
            // ...
        }
    }
}

On Incomplete, it reads more; a read of 0 means the client left mid-request. Any other error gets a 400 whose body is the error’s Display text, and the connection closes. After a framing error the server cannot know where the next request starts.

fn handle_connection(mut stream: TcpStream) -> std::io::Result<()> {
    // ...
    loop {
        match parse_request(&buffer, limits) {
            // ...
            Err(ParseError::Incomplete) => {
                let read = stream.read(&mut chunk)?;
                if read == 0 {
                    // Peer closed mid-request.
                    return Ok(());
                }
                buffer.extend_from_slice(&chunk[..read]);
            }
            Err(error) => {
                let response =
                    build_response(400, "text/plain", error.to_string().as_bytes(), false);
                stream.write_all(&response)?;
                return Ok(());
            }
        }
    }
}

A session with curl and nc against the server, started with cargo run --bin http_server 127.0.0.1:18080:

$ curl -si http://127.0.0.1:18080/healthz
HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain
Connection: keep-alive

ok
$ curl -si -X POST -d '{"gpuCount":1}' http://127.0.0.1:18080/v1/gpu-workloads
HTTP/1.1 201 Created
Content-Length: 49
Content-Type: application/json
Connection: keep-alive

{"id":"wl-a100-2","gpuCount":1,"state":"Pending"}
$ printf 'POST / HTTP/1.1\r\nHost: x\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\nhello' | nc 127.0.0.1 18080
HTTP/1.1 400 Bad Request
Content-Length: 59
Content-Type: text/plain
Connection: close

Content-Length and Transfer-Encoding are mutually exclusive

21.3.3 Reading the server against the RFC

The same session shows three places where the server departs from RFC 9110. None of them is hard to fix, and finding them is good practice.

$ curl -si -X DELETE http://127.0.0.1:18080/v1/gpu-workloads/wl-7
HTTP/1.1 204 No Content
Content-Length: 0
Connection: keep-alive

$ printf 'HEAD /healthz HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n' | nc 127.0.0.1 18080
HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain
Connection: close

ok

• HEAD must not carry a body. A HEAD response repeats the GET headers, including the Content-Length of a body it does not send. The server sends ok. A client that trusts the spec reads those three bytes as the start of the next response. The fix is one line in handle_connection: skip body when request.method == Method::Head.
• 204 must not carry Content-Length. build_response always writes it. Skipping it for 204 (and 1xx) fixes that.
• Errors all become 400. PayloadTooLarge deserves 413, and status_reason already has its text. Mapping ParseError to a status in one match fixes it. A 405 should also carry an Allow header listing the methods that path supports.

Two more points concern safety rather than the RFC. The id is interpolated into JSON with format!, so an id containing " produces invalid JSON; a real server serializes with serde_json. And buffer.clear() after each request discards any pipelined bytes that arrived with it, which the module comment acknowledges.

21.4 The complete programs

Listing 21.1 An HTTP/1.1 request parser with framing checks and limits. src/problems/http_request.rs

//! HTTP/1.1 request parsing: request line, headers, and body framing.
//!
//! The framing checks are the point. `Content-Length` together with
//! `Transfer-Encoding` is rejected as ambiguous (request smuggling), as are
//! obsolete line folding and a missing `Host`. Header and body sizes are bounded
//! so a slow client cannot hold a connection open.

use std::fmt;

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Method {
    Get,
    Head,
    Post,
    Put,
    Patch,
    Delete,
    Options,
}

impl Method {
    pub fn parse(token: &str) -> Option<Self> {
        Some(match token {
            "GET" => Self::Get,
            "HEAD" => Self::Head,
            "POST" => Self::Post,
            "PUT" => Self::Put,
            "PATCH" => Self::Patch,
            "DELETE" => Self::Delete,
            "OPTIONS" => Self::Options,
            _ => return None,
        })
    }

    pub fn is_safe(self) -> bool {
        matches!(self, Self::Get | Self::Head | Self::Options)
    }

    pub fn is_idempotent(self) -> bool {
        matches!(
            self,
            Self::Get | Self::Head | Self::Options | Self::Put | Self::Delete
        )
    }
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Version {
    Http10,
    Http11,
}

#[derive(Debug, Clone, Copy)]
pub struct Limits {
    pub max_header_bytes: usize,
    pub max_body_bytes: usize,
}

impl Default for Limits {
    fn default() -> Self {
        Self {
            max_header_bytes: 16 * 1024,
            max_body_bytes: 8 * 1024 * 1024,
        }
    }
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Request {
    pub method: Method,
    pub target: String,
    pub version: Version,
    pub headers: Vec<(String, String)>,
    pub body: Vec<u8>,
}

impl Request {
    /// First header value matching `name`, case-insensitively.
    pub fn header(&self, name: &str) -> Option<&str> {
        self.headers
            .iter()
            .find(|(key, _)| key.eq_ignore_ascii_case(name))
            .map(|(_, value)| value.as_str())
    }

    /// HTTP/1.1 reuses the connection unless `Connection: close`; HTTP/1.0 only
    /// reuses it with `Connection: keep-alive`.
    pub fn should_keep_alive(&self) -> bool {
        let connection = self.header("connection").unwrap_or("");
        let has = |token: &str| {
            connection
                .split(',')
                .any(|part| part.trim().eq_ignore_ascii_case(token))
        };
        match self.version {
            Version::Http11 => !has("close"),
            Version::Http10 => has("keep-alive"),
        }
    }
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ParseError {
    /// Not a complete request yet; read more and retry.
    Incomplete,
    Malformed(&'static str),
    MissingHost,
    /// `Content-Length` and `Transfer-Encoding` were both present.
    AmbiguousBodyLength,
    UnsupportedTransferEncoding,
    PayloadTooLarge,
}

impl fmt::Display for ParseError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::Incomplete => write!(f, "incomplete request"),
            Self::Malformed(reason) => write!(f, "malformed request: {reason}"),
            Self::MissingHost => write!(f, "HTTP/1.1 requires exactly one Host header"),
            Self::AmbiguousBodyLength => {
                write!(
                    f,
                    "Content-Length and Transfer-Encoding are mutually exclusive"
                )
            }
            Self::UnsupportedTransferEncoding => write!(f, "unsupported Transfer-Encoding"),
            Self::PayloadTooLarge => write!(f, "request exceeds configured limits"),
        }
    }
}

impl std::error::Error for ParseError {}

/// Parse one request from the start of `input`.
pub fn parse_request(input: &[u8], limits: Limits) -> Result<Request, ParseError> {
    let Some(header_end) = input
        .windows(4)
        .position(|window| window == b"\r\n\r\n")
    else {
        return Err(if input.len() > limits.max_header_bytes {
            ParseError::PayloadTooLarge
        } else {
            ParseError::Incomplete
        });
    };
    if header_end + 4 > limits.max_header_bytes {
        return Err(ParseError::PayloadTooLarge);
    }

    let head = &input[..header_end];
    let body_start = header_end + 4;

    let mut lines = head
        .split(|byte| *byte == b'\n')
        .map(|line| line.strip_suffix(b"\r").unwrap_or(line));

    let request_line = std::str::from_utf8(
        lines
            .next()
            .ok_or(ParseError::Malformed("empty"))?,
    )
    .map_err(|_| ParseError::Malformed("request line"))?;
    let mut parts = request_line.split(' ');
    let method = Method::parse(
        parts
            .next()
            .ok_or(ParseError::Malformed("method"))?,
    )
    .ok_or(ParseError::Malformed("unknown method"))?;
    let target = parts
        .next()
        .ok_or(ParseError::Malformed("target"))?;
    let version = match parts.next() {
        Some("HTTP/1.1") => Version::Http11,
        Some("HTTP/1.0") => Version::Http10,
        _ => return Err(ParseError::Malformed("version")),
    };
    if parts.next().is_some() || target.is_empty() {
        return Err(ParseError::Malformed("request line"));
    }

    let mut headers = Vec::new();
    for line in lines {
        if line.is_empty() {
            continue;
        }
        if matches!(line.first(), Some(b' ') | Some(b'\t')) {
            return Err(ParseError::Malformed("obsolete line folding"));
        }
        let colon = line
            .iter()
            .position(|byte| *byte == b':')
            .ok_or(ParseError::Malformed("header without colon"))?;
        let name = std::str::from_utf8(&line[..colon])
            .map_err(|_| ParseError::Malformed("header name"))?;
        if name.is_empty()
            || name
                .bytes()
                .any(|byte| byte.is_ascii_whitespace())
        {
            return Err(ParseError::Malformed("invalid header name"));
        }
        let value = std::str::from_utf8(&line[colon + 1..])
            .map_err(|_| ParseError::Malformed("header value"))?;
        headers.push((name.to_string(), value.trim().to_string()));
    }

    if version == Version::Http11
        && headers
            .iter()
            .filter(|(name, _)| name.eq_ignore_ascii_case("host"))
            .count()
            != 1
    {
        return Err(ParseError::MissingHost);
    }

    let body = parse_body(input, body_start, &headers, limits)?;

    Ok(Request {
        method,
        target: target.to_string(),
        version,
        headers,
        body,
    })
}

fn parse_body(
    input: &[u8],
    body_start: usize,
    headers: &[(String, String)],
    limits: Limits,
) -> Result<Vec<u8>, ParseError> {
    let collect = |name: &str| -> Vec<&str> {
        headers
            .iter()
            .filter(|(key, _)| key.eq_ignore_ascii_case(name))
            .map(|(_, value)| value.as_str())
            .collect()
    };
    let transfer_encoding = collect("transfer-encoding");
    let content_length = collect("content-length");

    // Two framing headers is the classic request-smuggling setup.
    if !transfer_encoding.is_empty() && !content_length.is_empty() {
        return Err(ParseError::AmbiguousBodyLength);
    }

    if !transfer_encoding.is_empty() {
        let joined = transfer_encoding.join(",");
        let codings: Vec<&str> = joined
            .split(',')
            .map(str::trim)
            .filter(|token| !token.is_empty())
            .collect();
        let all_chunked = codings
            .iter()
            .all(|token| token.eq_ignore_ascii_case("chunked"));
        if !all_chunked
            || !codings
                .last()
                .is_some_and(|t| t.eq_ignore_ascii_case("chunked"))
        {
            return Err(ParseError::UnsupportedTransferEncoding);
        }
        return decode_chunked(&input[body_start..], limits.max_body_bytes);
    }

    if content_length.is_empty() {
        return Ok(Vec::new());
    }

    let mut lengths = content_length.iter().map(|value| {
        value
            .trim()
            .parse::<usize>()
            .map_err(|_| ParseError::Malformed("invalid Content-Length"))
    });
    let first = lengths.next().expect("non-empty")?;
    if lengths.any(|length| length != Ok(first)) {
        return Err(ParseError::Malformed("conflicting Content-Length"));
    }
    if first > limits.max_body_bytes {
        return Err(ParseError::PayloadTooLarge);
    }
    let end = body_start + first;
    if input.len() < end {
        return Err(ParseError::Incomplete);
    }
    Ok(input[body_start..end].to_vec())
}

fn decode_chunked(data: &[u8], max_body: usize) -> Result<Vec<u8>, ParseError> {
    let mut cursor = 0;
    let mut body = Vec::new();

    loop {
        let line_end = find_crlf(data, cursor).ok_or(ParseError::Incomplete)?;
        let size_token = data[cursor..line_end]
            .split(|byte| *byte == b';')
            .next()
            .unwrap_or(&[]);
        let size = usize::from_str_radix(
            std::str::from_utf8(size_token)
                .map_err(|_| ParseError::Malformed("chunk size"))?
                .trim(),
            16,
        )
        .map_err(|_| ParseError::Malformed("chunk size"))?;
        cursor = line_end + 2;

        if size == 0 {
            // Consume trailers up to the final empty line.
            loop {
                let trailer_end = find_crlf(data, cursor).ok_or(ParseError::Incomplete)?;
                let is_empty = trailer_end == cursor;
                cursor = trailer_end + 2;
                if is_empty {
                    return Ok(body);
                }
            }
        }

        if body.len() + size > max_body {
            return Err(ParseError::PayloadTooLarge);
        }
        let chunk_end = cursor
            .checked_add(size)
            .ok_or(ParseError::Malformed("chunk overflow"))?;
        if chunk_end + 2 > data.len() {
            return Err(ParseError::Incomplete);
        }
        body.extend_from_slice(&data[cursor..chunk_end]);
        if &data[chunk_end..chunk_end + 2] != b"\r\n" {
            return Err(ParseError::Malformed("chunk not CRLF-terminated"));
        }
        cursor = chunk_end + 2;
    }
}

fn find_crlf(data: &[u8], from: usize) -> Option<usize> {
    data.get(from..)?
        .windows(2)
        .position(|window| window == b"\r\n")
        .map(|offset| offset + from)
}

#[cfg(test)]
mod tests {
    use super::*;

    fn parse(raw: &[u8]) -> Result<Request, ParseError> {
        parse_request(raw, Limits::default())
    }

    #[test]
    fn parses_a_get_and_a_post_with_content_length() {
        let get = parse(b"GET /v1/gpu-workloads HTTP/1.1\r\nHost: api\r\nAccept: */*\r\n\r\n")
            .expect("valid");
        assert_eq!(get.method, Method::Get);
        assert_eq!(get.header("HOST"), Some("api"));
        assert!(get.body.is_empty());
        assert!(get.should_keep_alive());

        let post = parse(b"POST / HTTP/1.1\r\nHost: api\r\nContent-Length: 5\r\n\r\nhello")
            .expect("valid");
        assert_eq!(post.body, b"hello");
        assert!(!post.method.is_idempotent());
    }

    #[test]
    fn decodes_chunked_bodies() {
        let raw = b"POST /u HTTP/1.1\r\nHost: api\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n6\r\n world\r\n0\r\n\r\n";
        assert_eq!(parse(raw).expect("valid").body, b"hello world");
    }

    #[test]
    fn rejects_ambiguous_framing_and_bad_framing() {
        let smuggling = b"POST / HTTP/1.1\r\nHost: api\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\nhello";
        assert_eq!(parse(smuggling), Err(ParseError::AmbiguousBodyLength));

        let gzip = b"POST / HTTP/1.1\r\nHost: api\r\nTransfer-Encoding: gzip\r\n\r\n";
        assert_eq!(parse(gzip), Err(ParseError::UnsupportedTransferEncoding));
    }

    #[test]
    fn rejects_missing_host_and_line_folding() {
        assert_eq!(
            parse(b"GET / HTTP/1.1\r\nAccept: */*\r\n\r\n"),
            Err(ParseError::MissingHost)
        );
        let folded = b"GET / HTTP/1.1\r\nHost: api\r\nX-A: 1\r\n continued\r\n\r\n";
        assert!(matches!(parse(folded), Err(ParseError::Malformed(_))));
    }

    #[test]
    fn reports_incomplete_and_enforces_limits() {
        assert_eq!(
            parse(b"GET / HTTP/1.1\r\nHost: api\r\n"),
            Err(ParseError::Incomplete)
        );
        let limits = Limits {
            max_header_bytes: 1024,
            max_body_bytes: 10,
        };
        let raw = b"POST / HTTP/1.1\r\nHost: api\r\nContent-Length: 999\r\n\r\n";
        assert_eq!(parse_request(raw, limits), Err(ParseError::PayloadTooLarge));
    }

    #[test]
    fn http_1_0_needs_keep_alive_opt_in() {
        let raw = b"GET / HTTP/1.0\r\n\r\n";
        assert!(
            !parse(raw)
                .expect("valid")
                .should_keep_alive()
        );
        let raw = b"GET / HTTP/1.0\r\nConnection: keep-alive\r\n\r\n";
        assert!(
            parse(raw)
                .expect("valid")
                .should_keep_alive()
        );
    }
}

Listing 21.2 Routes, responses, and a keep-alive loop over std::net. src/bin/http_server.rs

//! Minimal HTTP/1.1 server with REST-style routes.
//!
//! It uses only `std::net` plus the request parser in
//! [`systems_lab::problems::http_request`], so the interesting parts
//! stay visible: the accept loop, the keep-alive loop, status codes, and the
//! routing decision.
//!
//! Routes:
//!
//! | Method | Path | Response |
//! |---|---|---|
//! | `GET` | `/healthz` | `200` plain text |
//! | `GET` | `/v1/gpu-workloads` | `200` JSON list |
//! | `POST` | `/v1/gpu-workloads` | `201` created |
//! | `GET` | `/v1/gpu-workloads/{id}` | `200` one workload |
//! | `DELETE` | `/v1/gpu-workloads/{id}` | `204` no content |
//!
//! A known path with the wrong method returns `405`, and an unknown path returns
//! `404`. Pipelined requests are not supported: the loop handles one request per
//! read buffer, which matches how most servers treat pipelining in practice.
//!
//! ```bash
//! cargo run --bin http_server            # binds 127.0.0.1:8080
//! curl -i http://127.0.0.1:8080/healthz
//! curl -i -X POST -d '{"gpuCount":1}' http://127.0.0.1:8080/v1/gpu-workloads
//! ```

use std::{
    io::{Read, Write},
    net::{TcpListener, TcpStream},
    thread,
};

use systems_lab::problems::http_request::{Limits, Method, ParseError, Request, parse_request};

/// Map a request to `(status, content_type, body)`.
///
/// Pure, so the routing rules are unit-testable without a socket.
pub fn route(request: &Request) -> (u16, &'static str, Vec<u8>) {
    let target = request.target.as_str();

    if target == "/healthz" {
        return match request.method {
            Method::Get | Method::Head => (200, "text/plain", b"ok\n".to_vec()),
            _ => method_not_allowed(),
        };
    }

    if target == "/v1/gpu-workloads" {
        return match request.method {
            Method::Get | Method::Head => (
                200,
                "application/json",
                br#"[{"id":"wl-a100-1","gpuCount":1,"state":"Running"}]"#.to_vec(),
            ),
            Method::Post => (
                201,
                "application/json",
                br#"{"id":"wl-a100-2","gpuCount":1,"state":"Pending"}"#.to_vec(),
            ),
            _ => method_not_allowed(),
        };
    }

    if let Some(id) = target.strip_prefix("/v1/gpu-workloads/") {
        if id.is_empty() || id.contains('/') {
            return (404, "text/plain", b"not found\n".to_vec());
        }
        return match request.method {
            Method::Get | Method::Head => (
                200,
                "application/json",
                format!(r#"{{"id":"{id}","state":"Running"}}"#).into_bytes(),
            ),
            Method::Delete => (204, "", Vec::new()),
            _ => method_not_allowed(),
        };
    }

    (404, "text/plain", b"not found\n".to_vec())
}

fn method_not_allowed() -> (u16, &'static str, Vec<u8>) {
    (405, "text/plain", b"method not allowed\n".to_vec())
}

fn status_reason(status: u16) -> &'static str {
    match status {
        200 => "OK",
        201 => "Created",
        204 => "No Content",
        400 => "Bad Request",
        404 => "Not Found",
        405 => "Method Not Allowed",
        413 => "Payload Too Large",
        _ => "Internal Server Error",
    }
}

fn build_response(status: u16, content_type: &str, body: &[u8], keep_alive: bool) -> Vec<u8> {
    let mut response = format!("HTTP/1.1 {} {}\r\n", status, status_reason(status));
    response.push_str(&format!("Content-Length: {}\r\n", body.len()));
    if !content_type.is_empty() {
        response.push_str(&format!("Content-Type: {content_type}\r\n"));
    }
    response.push_str(if keep_alive {
        "Connection: keep-alive\r\n"
    } else {
        "Connection: close\r\n"
    });
    response.push_str("\r\n");

    let mut bytes = response.into_bytes();
    bytes.extend_from_slice(body);
    bytes
}

/// Serve connections forever, one thread per connection.
pub fn serve(listener: TcpListener) {
    for incoming in listener.incoming() {
        match incoming {
            Ok(stream) => {
                thread::spawn(move || {
                    if let Err(error) = handle_connection(stream) {
                        eprintln!("connection error: {error}");
                    }
                });
            }
            Err(error) => eprintln!("accept error: {error}"),
        }
    }
}

fn handle_connection(mut stream: TcpStream) -> std::io::Result<()> {
    stream.set_nodelay(true)?;
    let limits = Limits::default();
    let mut buffer = Vec::new();
    let mut chunk = [0u8; 4096];

    loop {
        match parse_request(&buffer, limits) {
            Ok(request) => {
                let keep_alive = request.should_keep_alive();
                let (status, content_type, body) = route(&request);
                let response = build_response(status, content_type, &body, keep_alive);
                stream.write_all(&response)?;

                if !keep_alive {
                    return Ok(());
                }
                buffer.clear();
            }
            Err(ParseError::Incomplete) => {
                let read = stream.read(&mut chunk)?;
                if read == 0 {
                    // Peer closed mid-request.
                    return Ok(());
                }
                buffer.extend_from_slice(&chunk[..read]);
            }
            Err(error) => {
                let response =
                    build_response(400, "text/plain", error.to_string().as_bytes(), false);
                stream.write_all(&response)?;
                return Ok(());
            }
        }
    }
}

fn main() {
    let address = std::env::args()
        .nth(1)
        .unwrap_or_else(|| "127.0.0.1:8080".to_string());

    let listener = TcpListener::bind(&address).expect("failed to bind");
    let local = listener
        .local_addr()
        .expect("failed to read local address");
    println!("http server listening on http://{local}");

    serve(listener);
}

#[cfg(test)]
mod tests {
    use systems_lab::problems::http_request::Version;

    use super::*;

    fn request(method: Method, target: &str) -> Request {
        Request {
            method,
            target: target.to_string(),
            version: Version::Http11,
            headers: vec![("host".to_string(), "localhost".to_string())],
            body: Vec::new(),
        }
    }

    #[test]
    fn healthz_is_ok() {
        let (status, content_type, body) = route(&request(Method::Get, "/healthz"));
        assert_eq!(status, 200);
        assert_eq!(content_type, "text/plain");
        assert_eq!(body, b"ok\n");
    }

    #[test]
    fn lists_and_creates_workloads() {
        let (status, _, _) = route(&request(Method::Get, "/v1/gpu-workloads"));
        assert_eq!(status, 200);

        let (status, content_type, body) = route(&request(Method::Post, "/v1/gpu-workloads"));
        assert_eq!(status, 201);
        assert_eq!(content_type, "application/json");
        assert!(
            String::from_utf8(body)
                .expect("json")
                .contains("Pending")
        );
    }

    #[test]
    fn gets_and_deletes_one_workload() {
        let (status, _, body) = route(&request(Method::Get, "/v1/gpu-workloads/wl-7"));
        assert_eq!(status, 200);
        assert!(
            String::from_utf8(body)
                .expect("json")
                .contains("wl-7")
        );

        let (status, _, body) = route(&request(Method::Delete, "/v1/gpu-workloads/wl-7"));
        assert_eq!(status, 204);
        assert!(body.is_empty());
    }

    #[test]
    fn wrong_method_on_known_path_is_405() {
        let (status, _, _) = route(&request(Method::Put, "/healthz"));
        assert_eq!(status, 405);
        let (status, _, _) = route(&request(Method::Put, "/v1/gpu-workloads"));
        assert_eq!(status, 405);
    }

    #[test]
    fn unknown_path_is_404() {
        let (status, _, _) = route(&request(Method::Get, "/nope"));
        assert_eq!(status, 404);
    }
}

21.5 Questions that come up

“How does a server know where an HTTP/1.1 request body ends?” Transfer-Encoding: chunked means read chunks until a zero-size chunk. Else Content-Length gives the count. With neither header there is no body. For responses, closing the connection can also end the body.

“What is request smuggling, and how do you prevent it?” Two servers frame the same bytes differently, so each sees a different message. Reject requests with both framing headers. Reject conflicting Content-Lengths. Close the connection after any framing error.

“How do you stop a client from exhausting your server’s memory?” Limit header size and body size. Check the declared length before reading the body. Time out idle and slow connections.

“Why is routing a separate pure function?” So the rules are testable without sockets. The protocol code (framing, keep-alive) can then be changed without touching the application logic.

“What does HTTP/2 change?” Binary framing with explicit lengths, many concurrent streams on one connection, and header compression. Framing ambiguity of the HTTP/1.1 kind disappears inside HTTP/2, although it can return where a proxy downgrades to HTTP/1.1.