Cover

About this book

A distributed system is a set of programs on separate machines that work together over a network and present themselves to their users as a single service. The machines are not reliable. Any of them may stop at any moment, a disk may be lost, and a message may arrive late, twice, or not at all. This book is about designing services that keep their promises while such things are happening, and about understanding exactly which promises can be kept.

It has 2 parts, of different character.

The first part designs and builds a single system from its foundations, a key-value store that runs on many machines. Chapter 1 is its system design: the requirements, the interfaces, the components, and the reasoning about sharding, replication and failure. Chapters 2 to 6 are its low-level design, in which each component is written in Rust, measured, and subjected to the failures it is claimed to survive. These are the chapters that print code. The ideas established here, the log, the majority, the leader, the lease and the conditional write, are the vocabulary of everything that follows.

The second part consists of system designs, each complete in itself: a rate limiter, a cache, a message queue, a payment system, a chat service, an exchange and others. Each begins from what the system is for, considers the algorithms and arrangements that could serve, and explains why one is chosen and what it costs. Where an algorithm proceeds in steps, a picture shows it step by step, at the reader’s own pace. These chapters print no code. Every one of them has a small simulation in the book’s repository, which produced the figures the chapter quotes, and which the reader is encouraged to run and to alter.

Running the code

The programs are in code/, a Cargo workspace. They use the Rust 2024 edition and are built with Rust 1.99.0, which code/rust-toolchain.toml selects.

$ cd code
$ cargo build --release
$ cargo run -p crashsim
$ cargo run -p ratelimit

The programs run on any Linux machine. A single small board is enough for all of them, and 2 or 3 boards or cloud machines on a network make it possible to stop a real machine and watch the system continue. The measurements quoted in the book were taken on an NVMe disk, and timings on other machines will differ.

What we assume

We assume that the reader has used a Linux shell, curl and HTTP. The first part assumes in addition that the reader can read Rust. Every other term is explained where it first appears, and each chapter ends with a table of the terms it introduced.

About the cover

The cover shows 5 lighthouses in a storm, each kept by a crab. Beams of light join their lamps. Lightning has struck one lighthouse and its lamp is dark, and the other 4 still shine, so the paper boat that carries a PUT between them still finds its way.

Each chapter opens with an animal at work on the chapter’s subject.

The crab that appears beside the listings is Ferris, the mascot of the Rust community.

Key-Value Store: System Design

A key-value store is the simplest database that is of any practical use. It offers 2 operations, to store a value under a key and to return the value that was stored under a key, and it makes no attempt to understand what the values mean. Simple as it is, it lies underneath a remarkable number of other systems. A web application keeps its users’ sessions in one. A cluster of machines keeps its configuration in one. A file system uses one for its directory of files, and an object store for its index of objects. Most of the larger storage systems in use today contain a key-value store as their innermost component, and the designs in the later chapters of this book rely on one repeatedly.

On a single machine, a key-value store is an afternoon’s work: a table in memory and a file on disk. The difficulty, and the subject of this chapter and the 5 that follow, begins when the store has to outgrow 1 machine and to outlive it. The data may be larger than any 1 disk. The requests may be more than any 1 processor can answer. And the machine will certainly fail at some point, by losing power, or its disk, or its connection to the network, and a store on which other systems depend cannot be permitted to lose its data or to stop answering on that account. Meeting these demands means spreading the data across many machines and keeping several copies of every part of it, and the moment there are several copies, the questions arise that define the whole field: which copy is right when they differ, who decides, and what happens when the machines cannot reach each other to agree.

This chapter designs such a store from its requirements. It is the system design, and it stays at the level of components and the messages between them. Chapters 2 to 6 then build each component in Rust and measure it.

Why such a store is needed

Consider a shop whose web site runs on 3 web servers behind a load balancer. A customer’s requests are distributed among the 3, so that the request which adds an item to the basket may be handled by 1 server and the request which displays the basket by another. The basket therefore cannot be kept in the memory of any 1 web server. It has to be held by some service that all 3 can reach.

This arrangement is typical, and it divides the system into parts of 2 very different characters. The web servers hold nothing that matters. Any of them can be restarted, replaced or duplicated at any moment without loss, and a component of this kind is called stateless. The shared store is the opposite. Everything of value is in it, a failure of it is a failure of every web server at once, and it cannot be restarted on a fresh machine, since a fresh machine would not have the data. The simplicity of the stateless part of a system is paid for by the difficulty of the part that holds the state, and this chapter is about that part.

What is demanded of the store follows directly from its position. Each demand has a consequence for the design, and each consequence is the subject of a later section.

Use cases

The store is used by 3 kinds of party. An application is any service that keeps its data in the store, such as the web servers of the shop. An operator is whoever looks after the machines, whether a person or an automated system. And monitoring is the system that watches the health of every machine and raises an alarm when something is wrong.

Functional requirements

The functional requirements state what the store does. Each is derived from a use case and is specific enough that a test could establish whether it is met.

Non-functional requirements

The non-functional requirements state how well the store must do these things: how much data, how many requests, how quickly, and through which failures. It is these, far more than the functional requirements, that determine the architecture. A program that kept 1 file on 1 machine would satisfy functional requirements 0 to 6 completely, and would satisfy almost none of what follows.

Durability and availability are easily confused and should be kept apart. Durability concerns data that has already been written, and requires that none of it be lost. Availability concerns requests that are yet to come, and requires that they be answered. Data can be perfectly durable and entirely unavailable, as when every copy of a key is on a machine that has been switched off for an hour and will return intact. The design provides for the 2 properties by different mechanisms, and they sometimes pull in opposite directions.

Which kind of consistency

The word “consistency” has to be given an exact meaning before anything can be designed to provide it, since storage systems offer 3 quite different things under that name. They are best distinguished by a single situation: a value has just been written, 1 of the copies of the data has not yet received the change, and a reader happens to be answered by that copy.

Under strong consistency, the reader receives the new value. The system behaves, as far as any client can tell, as though there were only 1 copy of the data, and a write that has been acknowledged is visible to every subsequent read. The precise form of this property is called linearizability. Its cost is that reads and writes must involve a leader or a majority of the copies, which makes them slower and means that they fail when a majority cannot be reached.

Under eventual consistency, the reader may receive the old value. The system promises only that, once writes cease, all copies will in time come to hold the same value. Any copy can answer any request, which makes the system fast and keeps it available through almost any failure, and the burden passes to the applications, which have to cope with stale values and with conflicting ones.

Tunable consistency lets each request choose. With N copies, a write waits for W of them to confirm and a read consults R of them. Whenever R + W is greater than N, the copies consulted by any read must include at least 1 that took part in the latest write, and the read sees that write. With 3 copies, the choice of W = 2 and R = 2 behaves strongly, and W = 1 and R = 1 is fast and only eventually consistent. The flexibility is real, and so is its cost, which is that every caller must choose correctly for every request.

This design chooses strong consistency, and the reason lies in the fourth use case. “Add an item to the basket, unless somebody else has changed the basket in the meantime” works only if the version that a client reads is the latest. If reads could be stale, 2 web servers might each read version 1, each conclude that nobody had changed the basket, and each write, with 1 of the 2 items lost. This is precisely the lost update that conditional writes exist to prevent, and a conditional write over stale reads prevents nothing. What strong consistency costs is set out in the section on guarantees: during a split of the network, the smaller side stops answering.

Core entities

The core entities are the things that the system stores, together with the things it stores them on. Naming them at this stage, with their fields, provides a vocabulary that the API, the protocol and the code of later chapters all share.

The first 2 entities are 2 representations of the same data, and the distinction between them runs through every chapter that follows. The records are a history: “set user-1 to ada”, “set user-1 to ada lovelace”, “delete user-2”. The entries are the state that results from applying that history in order. Both are kept, because they serve different purposes. Records are what is written to disk and copied to other nodes, since a sequence of changes is easy to append to, to transmit and to compare. Entries are what reads are answered from, since nobody wishes to replay a history in order to learn the present value. The version of an entry is the sequence number of the last record that changed it, which fastens every entry to a definite point in the history.

Capacity estimate

Arithmetic ahead. This estimate turns non-functional requirement 4 into a number of machines. The result is 24 storage nodes, each holding 138 GB, and nothing else from this section is needed later.

Skip to the high-level design

Show the calculation

The estimate depends on 1 quantity that has to be measured and cannot be assumed: the time a disk takes to store a write durably. Chapter 2 measures it with a program called synccost. On an NVMe disk, appending a record and waiting for the disk to confirm took 5.4 ms, so that a node which waits once for each record stores 185 records per second. Readers can run synccost on their own hardware and repeat the calculation with their own figure.

Storage. An entry consists of a key of about 32 bytes, a value of 1024 bytes and about 44 bytes of bookkeeping, 1100 bytes in all.

Quantity	Calculation	Result
Size of 1 copy of the data	1000000000 × 1100 bytes	1.1 TB
Size of 3 copies, as non-functional requirement 0 demands	1.1 TB × 3	3.3 TB

Writes. Every write is stored on 3 nodes, so that 20000 writes per second become 60000 records per second across the cluster. At 185 records per second for each node, 325 nodes would be needed. The time the disk takes, however, is nearly the same for 1 record as for 20, and a node can therefore gather the records that arrive during 1 wait and store them together. Storing several records with a single wait is called group commit, and it is the subject of chapter 6. With 20 records for each wait on average, a node stores 185 × 20 = 3700 records per second.

Quantity	Calculation	Result
Records to store per second	20000 writes × 3 copies	60000
Records per node per second, with 1 wait for each record	1 ÷ 0.0054 s	185
Records per node per second, with 20 records for each wait	185 × 20	3700
Nodes required	60000 ÷ 3700	16.2, rounded up to 17
Nodes chosen		24
Capacity of 24 nodes	24 × 3700	88800 records per second
Data on each node	3.3 TB ÷ 24	138 GB

We choose 24 nodes. At the required rate of writes, a cluster of that size works at 68% of its capacity, which leaves the surviving nodes room to carry the load of one that has failed.

Reads. Reads are answered from memory and do not wait for the disk. Spread over 24 nodes, 100000 reads per second come to about 4200 for each node, far below what a node can answer from memory. The rate of reads therefore has no influence on the number of nodes.

The API

Applications reach the store over HTTP. The API follows the style known as REST, in which the things a service holds are presented as resources, each identified by a URL, and the standard methods of HTTP are the operations upon them. Adhering to the standard meanings of the methods and status codes has a practical reward. Every existing HTTP client, proxy and cache already understands them, and so handles the API correctly without knowing anything about key-value stores.

The rules the API follows

Reference

Each operation is specified below by its request and its possible responses. store.example stands for the address of the load balancer.

Set the value of a key

PUT /v1/keys/user-1 HTTP/1.1
Host: store.example
Content-Type: application/octet-stream
Content-Length: 3

ada

HTTP/1.1 201 Created
ETag: "1"
Content-Length: 0

Read the value of a key

GET /v1/keys/user-1 HTTP/1.1
Host: store.example

HTTP/1.1 200 OK
ETag: "1"
Content-Type: application/octet-stream
Content-Length: 3

ada

Delete a key

DELETE /v1/keys/user-2 HTTP/1.1
Host: store.example
If-Match: "3"

HTTP/1.1 204 No Content

List keys

GET /v1/keys?prefix=user-&limit=2 HTTP/1.1
Host: store.example

HTTP/1.1 200 OK
Content-Type: application/json

{
  "keys": [
    { "key": "user-1", "version": 2, "size": 12 },
    { "key": "user-2", "version": 3, "size": 5 }
  ],
  "next_after": "user-2"
}

Errors

Every error response has the content type application/problem+json and a body with these fields.

HTTP/1.1 412 Precondition Failed
Content-Type: application/problem+json

{
  "type": "/problems/condition-failed",
  "title": "The condition on the key's version does not hold",
  "status": 412,
  "detail": "the key is at version 2"
}

Health of a node

HTTP/1.1 200 OK
Content-Type: application/json

{ "status": "ok", "keys": 2, "last_sequence": 5 }

The sections that follow take up the parts of this reference that embody a decision, and show each exchange as it was actually performed by the server of chapter 2.

Writing and reading a key

PUT stores the body of the request as the value of the key. Since values are opaque bytes, the body is sent without any encoding. The first write of a key is answered with 201 Created. All the responses in this section were produced by the server of chapter 2, with its date header removed.

$ curl -i -X PUT http://127.0.0.1:7001/v1/keys/user-1 --data-binary 'ada'
HTTP/1.1 201 Created
etag: "1"
content-length: 0

The version of an entry is conveyed in the etag header. In HTTP, an entity tag, or ETag, identifies 1 particular state of a resource, which is exactly what a version is. GET returns the value together with its tag.

$ curl -i http://127.0.0.1:7001/v1/keys/user-1
HTTP/1.1 200 OK
content-type: application/octet-stream
etag: "1"
content-length: 3

ada

Conditional writes

Functional requirement 3 is directed at a problem called the lost update, which arises when 2 clients read a value, each modify it independently, and each write it back.

Both writes succeed, and milk is missing from the final value. Nothing failed, and no error was reported to anyone. The write at step 4 was computed from what B read at step 2, which by then was no longer current, and the store, having been told only to set the value, set it.

The remedy is to let a write state which version it was based on. HTTP provides the If-Match header for this purpose: the server performs the write only if the present version of the resource is the version named in the header.

$ curl -i -X PUT http://127.0.0.1:7001/v1/keys/user-1 -H 'If-Match: "1"' --data-binary 'ada lovelace'
HTTP/1.1 204 No Content
etag: "2"

The key was at version 1, the condition held, and version 2 resulted. The same request sent a second time is refused, since version 1 is no longer current.

$ curl -i -X PUT http://127.0.0.1:7001/v1/keys/user-1 -H 'If-Match: "1"' --data-binary 'ada byron'
HTTP/1.1 412 Precondition Failed
content-type: application/problem+json
content-length: 144

{"type":"/problems/condition-failed","title":"The condition on the key's version does not hold","status":412,"detail":"the key is at version 2"}

The status 412 Precondition Failed informs web server B that its copy is out of date. B reads again, obtains tea, milk at version 2, adds the sugar, and writes with If-Match: "2". The procedure of reading, modifying and writing conditionally, repeated until the write succeeds, is the general method by which clients of a store cooperate without locks, and it recurs throughout the later chapters under the name of compare-and-set.

The header If-None-Match: * expresses the complementary condition, that the write is to be performed only if the key does not exist. With it, several clients may all attempt to create the same key, and exactly 1 succeeds.

$ curl -i -X PUT http://127.0.0.1:7001/v1/keys/user-1 -H 'If-None-Match: *' --data-binary 'someone else'
HTTP/1.1 412 Precondition Failed
content-type: application/problem+json
content-length: 144

{"type":"/problems/condition-failed","title":"The condition on the key's version does not hold","status":412,"detail":"the key is at version 2"}

On a GET, If-None-Match with a version asks for the value only if it has changed since that version. If it has not, the server answers 304 Not Modified with no body, which spares the transfer of data that the client already holds.

$ curl -i http://127.0.0.1:7001/v1/keys/user-1 -H 'If-None-Match: "2"'
HTTP/1.1 304 Not Modified
etag: "2"

Listing keys

GET /v1/keys returns the keys that begin with prefix, in sorted order, and no more than limit of them. When more remain, the response carries next_after, and the client obtains the next page by passing that value as after.

$ curl -i 'http://127.0.0.1:7001/v1/keys?prefix=user-&limit=1'
HTTP/1.1 200 OK
content-type: application/json
content-length: 71

{"keys":[{"key":"user-1","version":2,"size":12}],"next_after":"user-1"}

$ curl -i 'http://127.0.0.1:7001/v1/keys?prefix=user-&limit=1&after=user-1'
HTTP/1.1 200 OK
content-type: application/json
content-length: 66

{"keys":[{"key":"user-2","version":3,"size":5}],"next_after":null}

The position in the listing is thus expressed as the last key of the previous page. A page number would not serve, for a reason that applies to every listing of data that may change. If a key is inserted ahead of the reader’s position between 2 requests, every later key moves down by 1 place, and a request for “page 2” returns a key that the reader has already received. A request for the keys after user-1 is unaffected by insertions and deletions elsewhere. Chapter 15 meets the same problem in a news feed and solves it in the same way.

Deleting a key

$ curl -i -X DELETE http://127.0.0.1:7001/v1/keys/user-2
HTTP/1.1 204 No Content

Errors

A failed request is answered with a status code in the 400s or the 500s and a body in the format of RFC 9457. The type field identifies the category of the error and is what a client program should test. The title is a fixed description of the category, and detail, when present, describes the particular occurrence.

$ curl -i http://127.0.0.1:7001/v1/keys/user-2
HTTP/1.1 404 Not Found
content-type: application/problem+json
content-length: 80

{"type":"/problems/key-not-found","title":"The key does not exist","status":404}

The 2 ranges of status codes call for different behaviour on the part of the client. A code in the 400s signifies a fault in the request, and sending the same request again will produce the same error. A code in the 500s signifies that the server was unable to carry out a valid request, which may well succeed if repeated.

Retries

Suppose that a client sends a request and loses its connection before any response arrives. It has no means of knowing whether the operation was performed. This situation, the lost response, is the most characteristic difficulty of distributed systems and reappears in nearly every chapter of this book. The API meets it by making every operation idempotent, so that repeating a request is safe whichever was the case.

The protocol between nodes

The API described so far serves applications. The storage nodes also communicate with one another. A node sends the records it has written to the nodes that hold the other copies, and every node obtains the current ring from the coordinator. This internal traffic has different needs from a public API and uses a protocol of its own.

Protocol Buffers is a format in which the types of messages are declared in a schema file and encoded in a compact binary form. gRPC is a protocol for remote procedure calls that carries such messages over HTTP/2. The schema file is the specification of the internal protocol, and it is written before any implementation.

The Record message corresponds to the Record entity.

Listing 1.1 The Record message. proto/store.proto

// A change to one key, as the leader of a shard wrote it to its log.
message Record {
  uint64 sequence = 1;
  oneof change {
    Put put = 2;
    Delete delete = 3;
  }
}

message Put {
  string key = 1;
  bytes value = 2;
}

message Delete {
  string key = 1;
}

Every field of a message has a number, and it is by the number that the binary encoding identifies the field. Names occur only in the schema. This arrangement is what allows nodes running different versions of the software to communicate, which they must do whenever a cluster is upgraded 1 node after another without stopping. A newer version may add a field under a number not previously used. An older node disregards numbers it does not recognise, and a newer node supplies a default value where a field is missing. The single rule that makes this work is that a field number, once used, keeps its meaning for ever.

A leader sends records to its followers by means of the Append call. The terms leader, follower and term are defined in the section on replication.

Listing 1.2 The replication service. proto/store.proto

// Sent by the leader of a shard to each follower.
service Replication {
  // Appends records to the follower's log. A call with no records is a
  // heartbeat: it tells the follower that the leader is alive.
  rpc Append(AppendRequest) returns (AppendResponse);
}

message AppendRequest {
  // The shard, named by the last position it owns.
  fixed64 shard = 1;
  // The term of the sender. A follower rejects a leader of an older term.
  uint64 term = 2;
  // The sequence of the record that precedes `records` in the leader's log.
  uint64 previous_sequence = 3;
  repeated Record records = 4;
  // Every record up to this sequence is on a majority of the replicas.
  uint64 committed_sequence = 5;
  // Added for leader election. The term of the record at previous_sequence,
  // and the id of the sender.
  uint64 previous_term = 6;
  uint32 leader = 7;
}

message AppendResponse {
  uint64 term = 1;
  // False when the follower's log does not end at `previous_sequence`.
  bool appended = 2;
  // The sequence of the last record in the follower's log.
  uint64 last_sequence = 3;
}

The field previous_sequence is what keeps a follower’s log identical to its leader’s. A follower appends the records it is sent only if its own log at present ends at previous_sequence. If it does not, the follower answers appended = false together with the actual last sequence number of its log, and the leader sends again from that point. No follower’s log can therefore contain a gap or hold records in a different order from the leader’s.

The coordinator stores the ring as a Ring message.

Listing 1.3 The ring, as the coordinator stores it. proto/store.proto

// The ring, as the coordinator stores it. Routers and nodes read it and watch it
// for changes.
message Ring {
  uint64 version = 1;
  repeated Node nodes = 2;
  repeated Shard shards = 3;
}

message Node {
  string id = 1;
  string address = 2;
  // The positions this node owns on the ring.
  repeated fixed64 tokens = 3;
}

// A range of the ring and the nodes that hold its keys.
message Shard {
  // The shard owns the positions from first_token to last_token, both included.
  fixed64 first_token = 1;
  fixed64 last_token = 2;
  uint64 term = 3;
  string leader = 4;
  repeated string followers = 5;
}

Routers and storage nodes read the ring when they start and thereafter watch it. A watch is a request that the coordinator holds open and answers whenever the stored value changes, so that every component learns of a new version of the ring within milliseconds and none of them has to ask repeatedly.

High-level design

The high-level design names the components of the system, states the role of each, and traces the path of a request through them. Figure 1.7 shows the whole. The numbers in the figure are the steps of a write, and they appear again in its legend and in the text below.

The coordinator is a small replicated store for configuration. It runs on 3 machines, keeps a complete copy of its data on each, and remains available as long as a majority of them is running. It may seem odd that a key-value store should depend on a second, smaller store, and the reason is a circularity that cannot otherwise be escaped. The ring is what tells every component where a key is to be found, and the store cannot keep the ring within itself, since the ring would be needed in order to find it. The ring is therefore kept in a separate system at a fixed address that every component knows. Chapter 4 shows how a group of 3 machines agrees on a value, which is what the coordinator does.

A write to the key user-1 passes through the following steps.

0. Before any request arrives, every router holds a copy of the ring, and the coordinator informs the routers of every change to it.
1. An application sends PUT /v1/keys/user-1 to the load balancer.
2. The load balancer forwards the request to a router that passed its most recent health check.
3. The router hashes user-1, looks up the resulting position in its copy of the ring, finds that the leader of the shard is node-10, and forwards the request there.
4. node-10 appends a record to its log and waits for its disk to confirm.
5. node-10 sends the record to both followers, node-1 and node-6.
6. As soon as 1 follower has confirmed, node-10 applies the record and answers 201 Created.
7. The response returns to the application by way of the router and the load balancer.

A read follows steps 1 to 3 and is then answered by node-10 from memory, with steps 4 to 6 omitted.

2 features of this arrangement deserve to be drawn out, since they are deliberate.

The routers are stateless. A component without state can be run in any number of instances, added and removed at will, and replaced after a failure without any procedure of recovery. The design confines all lasting state to the storage nodes and the coordinator, so that those are the only components which require replication and the careful treatment of failure that occupies the rest of this chapter.

And, as step 0 indicates, the coordinator takes no part in the handling of requests. If it becomes unreachable, reads and writes continue undisturbed on the copies of the ring that the routers already hold. Only changes to the ring are suspended: a change of leader cannot be recorded and a node cannot be added until the coordinator returns. A component that every request had to consult would be both a bottleneck and a single point of failure, and keeping the coordinator off the path of requests avoids both.

Sharding

Sharding is the division of the data into parts that are stored on different nodes, each part being a shard. With sharding, the total volume of data can exceed the capacity of any 1 node, and the load of requests is distributed among all of them. The central question is by what function a key is assigned to a node, and the obvious answer turns out to have a serious flaw.

Hashing modulo the number of nodes

The obvious method hashes the key and takes the remainder on division by the number of nodes, node = hash(key) mod N. Provided that the hash function distributes its output evenly, every node receives an equal share of the keys, and any component that knows N can locate any key by a single calculation.

The flaw appears when N changes. A key stays where it is only if its hash leaves the same remainder under the old number of nodes and the new. The following table shows, for a few hash values, what happens when N increases from 4 to 5.

Of any 20 consecutive hash values, only 4 leave the same remainder modulo 4 and modulo 5. The addition of a fifth node therefore relocates 80% of all keys, although the new node has need of only 20% of them, and the other 60% merely change places among the existing nodes to nobody’s advantage. During such a relocation the cluster expends a large part of its capacity on copying data, at exactly the time when a node was added because capacity was short.

Consistent hashing

Consistent hashing is a method of assignment under which a change in the set of nodes relocates only the keys that have to move. The range of the hash function is regarded as a circle, called the ring, on which the largest value is followed by the smallest. Every node is given a position on the ring, called its token. A key is hashed to a position by the same function, and it belongs to the node whose token is the first at or after that position, proceeding clockwise.

In figure 1.8 the ring has only 100 positions, to keep the arithmetic easy. Node B has the token 37, and the token before it is 12, so that B owns the positions 13 to 37. A key that hashes to position 20 belongs to B. A key that hashes to 90 lies beyond the last token, passes the end of the range, continues from 0, and belongs to A at position 12.

The gain of the method is seen when a node is added.

Node E is given the token 50, which lies between the token of B and that of C. E takes over the positions 38 to 50, which formerly belonged to C, and every other position keeps its owner. The addition has relocated 13% of the positions, and every relocated key has gone to E. The removal of a node has the reverse effect: its positions pass to its clockwise neighbour, and no other key is touched.

The last row is the price. Under the first method a component needs to know 1 number, and under the second it needs the whole ring. How every component comes to have the ring, and what happens when its copy is out of date, is taken up below.

Virtual nodes

With a single token for each node, the size of a node’s range depends on where its own token and its predecessor’s happen to fall, and the ranges can differ enormously. Furthermore, when a node is removed its entire range passes to 1 neighbour, which then carries a double load.

Both defects are removed by giving every node many tokens. The tokens of a node are called its virtual nodes. Each node then owns many small ranges scattered around the ring. Its total share is the sum of many random range sizes, which varies far less than any single one of them, and the ranges of a removed node pass to many different neighbours.

The program ringsim measures the effect. It places 100000 keys on a ring of 24 nodes, with 1, 8, 64 and 256 tokens for each node, and reports the smallest and the largest share of keys held by any node. It then adds a node and counts the keys whose owner has changed.

$ cargo run --release -p ringsim
100000 keys on 24 nodes, ideal share 4.17%
     1 tokens per node: smallest share  0.16%, largest share 11.26%
     8 tokens per node: smallest share  1.42%, largest share  7.83%
    64 tokens per node: smallest share  3.35%, largest share  5.07%
   256 tokens per node: smallest share  3.86%, largest share  4.73%
adding node 25 to a ring with 256 tokens per node, ideal movement 4.00%
  on the ring: 4332 keys changed owner (4.33%), 4332 of them to the new node
  hash mod N : 96001 keys changed owner (96.00%)
replicas of user-1: node-10, node-1, node-6
position of user-1: 0x7c7615c11438e5c9

We use 256 tokens for each node. With 24 nodes, the ring contains 6144 tokens, and each range between adjacent tokens holds about 180 MB on average. A larger number of tokens would improve the balance further, at the cost of a larger ring to store and to distribute.

The ring uses a 64-bit hash function, xxHash3, for tokens and keys alike. The token of a virtual node is the hash of the name of its node followed by a number, as in node-7#12. Since the tokens are derived from the names, every router and node that knows the names of all the nodes computes an identical ring, and no table of tokens has to be transmitted.

The tokens are kept in a sorted map. To find the owner of a key, a lookup takes the tokens from the key’s position to the end of the map, followed by those from its beginning, which is clockwise order on the ring. The cost is logarithmic in the number of tokens.

Shards on the ring

Each range between adjacent tokens is a shard. The replicas of a shard are placed on the first 3 distinct nodes encountered going clockwise from its range. The node that owns the range is the preferred leader of the shard, and the other 2 are its followers. For the key user-1, ringsim reports node-10, node-1 and node-6, and these 3 nodes serve as the example in every chapter that follows.

The word “distinct” matters. Consecutive tokens on the ring may belong to the same physical node, since each node has 256 of them, and 2 replicas placed on 1 machine would give no protection against the loss of that machine. Tokens of a node already chosen are therefore passed over.

The ring in a running system

The ring has so far been described as a data structure. It remains to describe how the components of a running cluster obtain it, use it and change it, and it is here that most of the practical difficulty lies.

Where the ring is kept. There is a single authoritative copy, the Ring message held by the coordinator. It contains the name, address and tokens of every node, and the present leader of every shard. Every change increases its version number by 1. Routers and storage nodes hold a copy in memory and keep it current by means of a watch.

How a request is routed. Routing requires no communication with the coordinator or with other routers. A router hashes the key and performs a binary search in its sorted list of tokens, which with 6144 tokens takes 13 comparisons.

What happens when a ring is out of date. A router may forward a request an instant before a new version of the ring reaches it, and the request then arrives at a node that is no longer responsible for the key. If that node were to answer, it might return a stale value or accept a write that belongs elsewhere, and strong consistency would be lost through nothing more than a delayed notification. Routers therefore attach the version of their ring to every forwarded request, in a Ring-Version header. A node that holds a newer ring and is not responsible for the key refuses the request with the status 421 Misdirected Request, which HTTP defines for a request sent to a server that cannot answer for the resource. On receiving it, the router waits for the newer ring and forwards again, and the client observes nothing but a slightly slower response. The principle, that a component acting on possibly stale information should declare which version it is acting on so that the recipient can check, reappears in this book wherever stale knowledge is dangerous.

How a node joins. A new node cannot assume ownership of its ranges at once, since it does not yet hold their data. Ownership is therefore transferred range by range, each after its data has been copied.

How a node leaves. When an operator removes a node, the procedure runs in reverse. The departing node copies each of its ranges to the node that will own it, and the coordinator reassigns a range when its copy is complete. When a node fails without warning, there is no opportunity to copy. Its ranges are still held by the other 2 replicas of each shard, the shards it led elect new leaders, and a replacement node is supplied with the data by the surviving replicas.

Gossip as an alternative. Some systems distribute the ring without any coordinator. Each node holds a copy, and the nodes exchange their copies in pairs, each repeatedly choosing another at random and comparing versions, so that a change spreads through the cluster as a rumour does through a crowd.

A gossip protocol removes the dependence on a coordinator and scales to very large clusters, since no node has to communicate with more than a few others. Its disadvantage is that for some seconds after a change, different nodes hold different versions of the ring and may direct the same key to different nodes. Designs founded on gossip have to tolerate such disagreement, and accordingly tend to be those that offer eventual consistency. We use a coordinator, since a single authoritative ring is far easier to reason about when strong consistency is the aim, and a cluster of 24 nodes has no need of the scalability of gossip.

The cost of sharding

An operation on a single key involves a single shard. An operation on many keys may involve many. Listing keys by prefix, functional requirement 4, is of the second kind, because hashing deliberately destroys the order of keys: keys with a common prefix hash to unrelated positions and are scattered over all the nodes. A router has to send such a request to every node and merge their sorted results, so that a listing takes time proportional to the number of nodes.

This is the general price of hashing, and systems whose work consists largely of range queries choose differently. They divide the key space into contiguous ranges of keys, without hashing, so that neighbouring keys remain on the same node and a range query touches few nodes. In exchange they must actively move the boundaries of the ranges in order to keep the load even, since nothing in an unhashed key space guarantees that keys are evenly spread.

Replication

Replication is the keeping of copies of the same data on several nodes, each copy being a replica. Non-functional requirement 0 lays down that an acknowledged write survives the loss of any 1 machine, from which it follows that a write must be stored on at least 2 machines before it is acknowledged.

Replication with a leader

If every replica accepted writes independently, 2 clients might write different values for the same key to 2 replicas at the same moment, and the replicas would then hold conflicting data, to be reconciled afterwards by some rule that is bound to discard one client’s work. This design avoids such conflicts by appointing 1 replica of every shard as its leader. All writes to the shard go to the leader, which assigns each the next sequence number, so that the writes have a single agreed order, and sends the resulting record to the other replicas, the followers. A follower applies the records in the order of their sequence numbers and thereby arrives at the leader’s state.

Acknowledgement by a majority

The leader acknowledges a write as soon as its record is stored in 2 of the 3 logs, its own and that of 1 follower. It does not wait for the second follower.

The rule has 2 consequences, both of them wanted. The record is on 2 machines at the moment the client receives its acknowledgement, as non-functional requirement 0 demands. And the response time of a write is governed by the faster of the 2 followers, so that a follower which is slow, restarting or unreachable delays nothing, which is what keeps the p99 latency within the bound of non-functional requirement 3.

Any set containing more than half of the replicas is called a majority. With 3 replicas, a majority consists of 2. The whole importance of majorities derives from a single property: any 2 majorities of the same shard have at least 1 node in common.

To see what this property accomplishes, consider what happens when the leader of a shard fails. Its followers elect a new leader, and a candidate requires the votes of a majority. The majority that stored record 42 and the majority that elects the new leader share at least 1 node, and that node holds record 42. By the rule of the election, a node votes only for a candidate whose log is at least as complete as its own. The node holding record 42 therefore refuses its vote to any candidate that lacks it, and such a candidate cannot assemble a majority. Every newly elected leader consequently holds every acknowledged record, and no acknowledged write is lost in a change of leader. Chapter 4 implements this argument and tests it.

The table shows that an even number of replicas tolerates no more failures than the odd number below it, while costing more. We therefore use 3 replicas. A deployment that must survive 2 simultaneous failures uses 5.

Asynchronous replication

In asynchronous replication, the leader acknowledges a write as soon as its own log holds the record and sends it to the followers afterwards. Writes are faster, since they wait for 1 disk and for no network round trip. But if the leader’s machine is lost immediately after an acknowledgement, no other machine has the record, and a write that was acknowledged is gone. Asynchronous replication is therefore incompatible with non-functional requirement 0, and this design uses acknowledgement by a majority. Later chapters adopt the asynchronous form where its risk is acceptable, as in the cache of chapter 9, and reject it where it is not, as in the lock service of chapter 10.

Failure handling

A distributed system is defined as much by what it does when its parts fail as by what it does when they work. This section takes each kind of failure in turn and states how it is detected, what clients observe, and how the system recovers.

Failure of a leader

A leader sends an Append call to each of its followers every 100 ms. When there are no new records, the call is empty and is known as a heartbeat, a periodic message whose arrival shows that its sender is running. A follower that receives no Append for 1 second concludes that its leader has failed.

Every election opens a new term, identified by a number that increases by 1. Terms are how nodes tell the present leader from a former one. In figure 1.16, node-10 still regards itself as the leader of term 4 when it returns. The first message it receives carries term 5. Since that is higher than its own, node-10 gives up its leadership and continues as a follower. Every Append call carries the term of its sender so that the receiver can make this comparison, and a returning leader that has been superseded is thereby prevented from doing any harm.

Network partition

A network partition is a failure of the network that divides the nodes into groups which cannot communicate with each other, while every node continues to run. Seen from inside one group, a partition is indistinguishable from the failure of all the nodes in the other, and no node can discover which has occurred.

In figure 1.17, node-10 is isolated and still regards itself as the leader. It accepts a write and appends a record to its own log, but it cannot send the record to a follower. Since no majority holds the record, node-10 does not acknowledge, and its client receives an error. Meanwhile 2 of the 3 replicas are in the other group, which elects a new leader and continues to process writes.

The rule of the majority thus resolves, without any further mechanism, the danger that a partition presents, namely 2 leaders each accepting writes that the other knows nothing of. At most 1 group can contain a majority of a shard’s replicas, so at most 1 leader can acknowledge writes, and no node is required to determine why it has lost contact with the others.

Summary of failures

The last row marks the limit of the design. With 3 replicas, the store tolerates the loss of any 1 machine and remains consistent under any partition of the network. A shard that has lost a majority of its replicas is unavailable, and has to be so, since the single surviving replica cannot know whether it holds every acknowledged write.

Guarantees and trade-offs

During a network partition, a distributed store cannot offer both consistency and availability. It must either refuse some requests in order to keep all its answers consistent, or accept all requests and allow the 2 groups to diverge. This result is known as the CAP theorem, and it is less a theorem to be proved than an observation that follows directly from the previous section: a node that cannot reach the others must choose between answering from what it has, which may be out of date, and declining to answer. This design gives priority to consistency.

The availability allowed by non-functional requirement 1 can be related to elections by a short calculation. An allowance of 21.6 minutes a month is 1296 seconds. If an election takes 5 seconds and the failed node led 1/24 of all the shards, the election causes 1/24 of all writes to fail for 5 seconds and consumes about 0.2 seconds of the allowance. Several thousand such failures a month would be within it. In practice, therefore, the requirement is not threatened by the failures of individual machines, which this design absorbs, and is limited by events that affect many nodes together, such as a faulty release of the software or the loss of a segment of the network.

The design, deployed

The figure of the high-level design showed 1 box for each kind of component. A running system has many of each, and their number and placement are part of the design. Applications reach the store through a pair of load balancers, of which 1 is active and the other takes over its address if it fails. Behind them are 12 routers, which can be any number, since they are stateless. The 24 storage nodes are divided equally among 3 availability zones, which are separate buildings or halls with their own power and network, and the ring is constructed so that the 3 replicas of every shard fall in 3 different zones. The loss of an entire zone then removes 1 replica of each shard and leaves a majority of every one. The coordinator has 1 machine in each zone for the same reason.

How real systems do it

The design of this chapter is not the only way of building a key-value store, and it is instructive to place it among the systems that are in use.

Its 2 ancestors are quite different from each other. Dynamo, described by Amazon in 2007, introduced to a wide audience the combination of consistent hashing, virtual nodes and replication to the next nodes on the ring that this chapter uses for placement. In every other respect Dynamo chose the opposite course. It has no leaders. Any replica accepts a write, a request succeeds when W of N replicas have answered, conflicting versions are detected with vector clocks and handed back to the application to be reconciled, and membership spreads by gossip. It is the model of eventual and tunable consistency described earlier, and Cassandra and Riak descend from it. The other ancestor is the line of consensus protocols, Paxos and later Raft, in which a leader orders the writes and a majority must store each one. The coordinator of this chapter is a system of that kind, as etcd and ZooKeeper are.

The design here combines the 2: placement by a hash ring, as in Dynamo, with a consensus group for each shard, as in Raft. Systems that are built this way in practice, among them CockroachDB, TiKV and YugabyteDB, differ from our design in 1 important respect. They do not hash keys. They divide the ordered key space into contiguous ranges, each replicated by its own Raft group, and they split a range when it grows beyond a set size and merge ranges that shrink. This keeps neighbouring keys together, so that a listing by prefix reads 1 or 2 ranges where ours must ask every node, and it lets the system move load by moving the boundaries. The price is the machinery for splitting, merging and rebalancing, which a hash ring does not need. Google’s Bigtable and Spanner also partition by range.

Several things that a production store requires are absent from this chapter and from the code that follows. A log that grows for ever must be cut short by taking snapshots of the state. Changing the set of members of a shard while it runs needs a protocol of its own, which Raft calls joint consensus. Reads served by a leader on the strength of a recent heartbeat depend on clocks behaving reasonably, and stricter systems confirm leadership with a majority for each read. And none of this has been tested here in the way that such systems are tested in earnest, by a tool such as Jepsen, which partitions a real cluster at random and checks every response against the definition of linearizability.

Mistakes to avoid

A first look at the low-level design

Every guarantee stated above rests on an assumption that has so far been taken for granted: that a record which a node has reported as stored is still in that node’s log after the node has crashed and restarted. If a node could lose such a record, a record counted as stored on 2 of 3 nodes might in truth be stored on 1, and the whole argument from majorities would be without foundation. A storage node justifies the assumption by observing 2 rules.

1. Every change is appended to a log on disk, and the node waits for the disk to confirm, before the change is applied in memory and before any response is sent.
2. On starting, a node rebuilds its state in memory by reading its log from beginning to end.

Chapter 2 implements a storage node on these 2 rules: a format for records on disk, a log with its procedure of recovery, and an HTTP server that provides the API of this chapter. It includes a simulation that cuts a log short at every possible byte and verifies that recovery returns exactly the records that were complete. The entities of this chapter appear in its code under the same names.

The simulation of the ring

References

• G. DeCandia and others, “Dynamo: Amazon’s Highly Available Key-value Store”, SOSP 2007.
• D. Ongaro and J. Ousterhout, “In Search of an Understandable Consensus Algorithm”, USENIX ATC 2014.
• D. Karger and others, “Consistent Hashing and Random Trees”, STOC 1997.
• F. Chang and others, “Bigtable: A Distributed Storage System for Structured Data”, OSDI 2006.
• J. Corbett and others, “Spanner: Google’s Globally-Distributed Database”, OSDI 2012.
• M. Herlihy and J. Wing, “Linearizability: A Correctness Condition for Concurrent Objects”, ACM TOPLAS, 1990.
• S. Gilbert and N. Lynch, “Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services”, ACM SIGACT News, 2002.
• M. Kleppmann, Designing Data-Intensive Applications, O’Reilly, 2017.

Terms introduced in this chapter

Key-Value Store: Storage Node (Low-Level Design)

The design of chapter 1 distributes a key-value store over 24 machines, keeps every key on 3 of them, and reasons at length about what happens when machines fail. All of that reasoning rests on a single assumption about each individual machine, which chapter 1 stated in its closing section and did not justify: that a write which a node has acknowledged will still be there after the node has crashed and been restarted. If 1 node can quietly lose a write it has confirmed, then a write “stored on 2 of 3 nodes” may be stored on 1, and nothing built on top of that count can be trusted.

This chapter builds the component that makes the assumption true. A storage node is the program that holds the entries of the store on 1 machine, and its whole difficulty is concentrated in the few milliseconds during which a write is in progress and the machine may lose power. We begin by examining what a crash can do to a write at each stage, which leads to the technique of writing every change to a log before doing anything else. We then define the format of the log on disk, write the code that recovers from a damaged log, and build on top of it the store and the HTTP API of chapter 1. The chapter ends by trying, in 3 different ways, to make the node lose data: by damaging its log at every possible byte, by killing it under load, and by killing it under load from another machine.

Storing data durably

A storage node keeps its entries in memory, so that a read requires no access to the disk. Memory is lost when the process stops, and so the node must keep the data on disk as well, in a form from which the contents of memory can be reconstructed.

The first design that comes to mind keeps a file which mirrors the contents of memory and, on every write, overwrites the affected part of that file. In the presence of crashes this is unsafe, and the reason is worth seeing concretely. Suppose that a value of 1 KB is being replaced by another of the same size, and that the power fails when half of the new bytes have reached the disk. The file now contains the first half of the new value followed by the second half of the old. The entry is corrupt, its former value can no longer be recovered, and, worst of all, nothing in the file indicates that anything is wrong.

The fault in this design is that an update in place destroys the old state before the new state is complete, so that there is a moment at which neither exists. The standard remedy is to give up updating in place altogether. The node appends a description of each change to the end of a file and modifies the file in no other way. A file used in this manner is called a log. Because every change is entered in the log before it is applied to any other structure, the technique is known as write-ahead logging, and the file as a write-ahead log.

With a file that is only ever appended to, a crash during a write can affect nothing but the record that was being appended. All the earlier records are intact, since nothing ever writes to that part of the file again.

The node maintains 4 invariants, conditions that hold at every moment, whatever requests arrive and wherever a crash occurs.

1. Every change is in the log, and confirmed by the disk, before its client receives a response.
2. Every change is in the log before it is applied to the map in memory.
3. The map is equal to the result of applying all the records of the log in order.
4. The log is modified only by appending. The bytes of a confirmed record are never written again.

Each invariant purchases something specific. The first is durability itself. The second and third together ensure that the map contains nothing that is absent from the log, so that the loss of the map loses no data. And the fourth confines the damage that a crash can do to the end of the file, where recovery knows to look for it.

Crashes during a write

Given this ordering of steps, the outcome of a crash can be worked out for each stage of a write.

The third case requires comment. The record has been stored durably, and the node crashes before it has sent its response. After the restart the write is present, although its client was never told that it had succeeded. This does not violate the guarantee of durability, which covers acknowledged writes and is silent about writes whose outcome was never reported. From the client’s point of view the outcome of such a write is unknown, which is the lost response that chapter 1 discussed under retries. The client resolves the uncertainty by sending the request again, and can do so safely because PUT is idempotent. This case is produced deliberately, by a test, in the section on killing the server.

The format of a record

When a log is read back, the reader has to determine where each record ends, since records differ in length. It must also be able to recognise a record that is incomplete or damaged, since the end of the file may contain the partial output of a write that a crash interrupted.

Both needs are met by beginning every record with a frame of 8 bytes, which holds the length of the data that follows and a checksum of it. The checksum is a CRC-32, a value of 32 bits computed from the data in such a way that any change to the data will almost certainly change the value. When a record is read, the checksum is computed afresh and compared with the one that was stored.

The following are the first bytes of a log after the 2 writes to user-1 shown in chapter 1.

$ xxd data/kv.log | head -4
00000000: 1400 0000 8dd3 da1d 0100 0000 0000 0000  ................
00000010: 0106 0075 7365 722d 3161 6461 1d00 0000  ...user-1ada....
00000020: cefd 7d53 0200 0000 0000 0000 0106 0075  ..}S...........u
00000030: 7365 722d 3161 6461 206c 6f76 656c 6163  ser-1ada lovelac

The first 4 bytes, 1400 0000, encode the length 20 in little-endian order, in which the least significant byte of a number comes first. The 4 bytes of the checksum follow. Within the payload, the sequence number 1 occupies the first 8 bytes, and is followed by the kind, 01, and the length of the key, 0600. The key and the value can be read in the right-hand column. At byte 28, which is 0x1c, the second record begins with the length 1d00 0000, or 29.

Listing 2.1 The imports of kv/src/record.rs. kv/src/record.rs

use crc32fast::hash;

Listing 2.2 The types for a change, a record, and the result of decoding. kv/src/record.rs

/// A change to the store.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Change {
    Put { key: String, value: Vec<u8> },
    Delete { key: String },
}

/// A change with its position in the log. Sequence numbers start at 1 and
/// grow by 1 with each record.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Record {
    pub sequence: u64,
    pub change: Change,
}

/// The result of reading one record from the front of a slice of bytes.
#[derive(Debug, PartialEq, Eq)]
pub enum Decoded {
    /// A whole record, and the number of bytes it occupies.
    Record { record: Record, length: usize },
    /// The bytes end before the record does.
    Incomplete,
    /// The checksum does not match the payload, or the payload is malformed.
    Corrupt,
}

/// The bytes of a frame: the length and the checksum.
pub const FRAME: usize = 8;

/// The largest payload a record may have: a key, a value of 1 MiB, and the
/// fixed fields.
pub const MAX_PAYLOAD: usize = (1 << 20) + 1024;

const PUT: u8 = 1;
const DELETE: u8 = 2;

Decoded has 3 variants because a reader must tell 3 situations apart: a complete record, data that ends before the record is complete, and data that is complete and invalid. The distinction between the last 2 is of no consequence to recovery, which treats them alike, and is valuable when a log is being examined after a fault, since the one indicates an interrupted write and the other a damaged disk.

Listing 2.3 Encoding a record. kv/src/record.rs

impl Record {
    /// Returns the record as the bytes that are appended to the log.
    pub fn encode(&self) -> Vec<u8> {
        let (kind, key, value) = match &self.change {
            Change::Put { key, value } => (PUT, key, value.as_slice()),
            Change::Delete { key } => (DELETE, key, &[][..]),
        };

        let mut payload = Vec::with_capacity(11 + key.len() + value.len());
        payload.extend_from_slice(&self.sequence.to_le_bytes());
        payload.push(kind);
        payload.extend_from_slice(&(key.len() as u16).to_le_bytes());
        payload.extend_from_slice(key.as_bytes());
        payload.extend_from_slice(value);

        let mut bytes = Vec::with_capacity(FRAME + payload.len());
        bytes.extend_from_slice(&(payload.len() as u32).to_le_bytes());
        bytes.extend_from_slice(&hash(&payload).to_le_bytes());
        bytes.extend_from_slice(&payload);

        bytes
    }
}

Decoding carries out the same steps in order and returns the variant that corresponds to the first step that fails.

Listing 2.4 Decoding a record. kv/src/record.rs

impl Record {
    /// Reads the record at the front of `bytes`.
    pub fn decode(bytes: &[u8]) -> Decoded {
        let Some((length, rest)) = bytes.split_first_chunk::<4>() else {
            return Decoded::Incomplete;
        };
        let Some((checksum, rest)) = rest.split_first_chunk::<4>() else {
            return Decoded::Incomplete;
        };

        let length = u32::from_le_bytes(*length) as usize;

        if length > MAX_PAYLOAD {
            return Decoded::Corrupt;
        }

        let Some(payload) = rest.get(..length) else {
            return Decoded::Incomplete;
        };

        if hash(payload) != u32::from_le_bytes(*checksum) {
            return Decoded::Corrupt;
        }

        match parse(payload) {
            Some(record) => Decoded::Record {
                record,
                length: FRAME + length,
            },
            None => Decoded::Corrupt,
        }
    }
}

/// Splits a payload into its fields. It returns `None` for a payload that is
/// too short for its fields or has an unknown kind.
fn parse(payload: &[u8]) -> Option<Record> {
    let (sequence, rest) = payload.split_first_chunk::<8>()?;
    let (&kind, rest) = rest.split_first()?;
    let (key_length, rest) = rest.split_first_chunk::<2>()?;
    let (key, value) = rest.split_at_checked(u16::from_le_bytes(*key_length) as usize)?;
    let key = String::from_utf8(key.to_vec()).ok()?;

    let change = match (kind, value) {
        (PUT, value) => Change::Put {
            key,
            value: value.to_vec(),
        },
        (DELETE, []) => Change::Delete { key },
        (_unknown, _value) => return None,
    };

    Some(Record {
        sequence: u64::from_le_bytes(*sequence),
        change,
    })
}

split_first_chunk returns the first bytes of a slice as an array of fixed size, together with the remainder, or None if the slice is too short. Every field is extracted by this means. A payload shorter than its fields therefore yields None, and at no point does the code use a value read from the disk as an index without first having validated it. A decoder of data from disk must be written on the assumption that the data may be anything at all, since after a fault it may be.

What a durable write costs

Invariant 1 requires that the disk has confirmed every record. Writing to a file does not, by itself, satisfy this requirement, and the difference between the 2 is the most consequential fact about storage that a programmer has to know.

The system call write copies data into a buffer belonging to the kernel and returns. At some later time, the kernel transfers the buffer to the disk. With the call fsync, a program instructs the kernel to transfer all the buffered data of a file at once, and the call does not return until the disk has reported success. In Rust, File::sync_data makes this call.

The program synccost appends 500 records to a log, first without fsync and then with an fsync after every record.

Listing 2.5 The imports of synccost/src/main.rs. synccost/src/main.rs

use std::env;
use std::error::Error;
use std::fs;
use std::path::Path;
use std::time::{Duration, Instant};

use kv::{Change, Log};

Listing 2.6 Measuring the cost of an append with and without fsync. synccost/src/main.rs

const RECORDS: u32 = 500;

/// Appends `RECORDS` records of 100 bytes and returns the time it took.
fn measure(directory: &Path, sync: bool) -> Result<Duration, Box<dyn Error>> {
    let mut log = Log::open(directory, sync)?.0;
    let started = Instant::now();

    for number in 0..RECORDS {
        log.append(Change::Put {
            key: format!("key-{number:06}"),
            value: vec![b'v'; 100],
        })?;
    }

    Ok(started.elapsed())
}

fn main() -> Result<(), Box<dyn Error>> {
    let root = env::args()
        .nth(1)
        .ok_or("usage: synccost <directory>")?;

    for (name, sync) in [("without fsync", false), ("with fsync", true)] {
        let directory = Path::new(&root).join(name.replace(' ', "-"));
        let elapsed = measure(&directory, sync)?;
        let each = elapsed.as_secs_f64() * 1000.0 / f64::from(RECORDS);

        println!(
            "{name:13}: {RECORDS} records in {:8.1} ms, {each:.3} ms per record, {:6.0} records per second",
            elapsed.as_secs_f64() * 1000.0,
            1000.0 / each
        );

        fs::remove_dir_all(&directory)?;
    }

    Ok(())
}

$ cargo run --release -p synccost -- /var/tmp/synccost
without fsync: 500 records in      1.4 ms, 0.003 ms per record, 366991 records per second
with fsync   : 500 records in   2697.5 ms, 5.395 ms per record,    185 records per second

On an NVMe disk, a durable append took 5.4 ms, about 1900 times as long as an append that merely copies data into the kernel’s buffer. This is the figure from which chapter 1 derived its 185 records per second. Because the cost is so high, storage systems commonly make the behaviour configurable, and kvd has an option --no-sync under which the node responds after write alone. In that mode the node keeps every acknowledged write if its process crashes, and may lose the most recent writes if the machine loses power. Whether that is an acceptable bargain depends on the data, and it is a decision that ought to be made knowingly, since the default behaviour of a great many programs makes it silently.

Design overview

The manifest of the workspace declares the edition of Rust, the minimum version of the compiler, and the dependencies that its crates share.

Listing 2.7 The complete Cargo.toml. Cargo.toml

[workspace]
resolver = "3"
members = ["kv", "kvd", "kvctl", "crashsim", "synccost", "ring", "ringsim", "repl", "replsim", "kvnode", "raft", "raftsim", "kvraft", "probe", "kvrouter", "kvgroup", "kvbench",
    "ratelimit", "idgen", "shortener", "cachesim", "queuesim", "locksim", "jobsim", "notifysim", "feedsim", "objectsim",
    "streamsim", "chatsim", "paysim", "bookingsim", "nearbysim", "typeahead",
    "crawlsim", "metricsim", "syncsim", "editsim", "matchsim", "clicksim",
]

[workspace.package]
edition = "2024"
rust-version = "1.99"
license = "MIT"
publish = false

[workspace.dependencies]
kv = { path = "kv" }
ring = { path = "ring" }
repl = { path = "repl" }
raft = { path = "raft" }
kvd = { path = "kvd" }
axum = "0.8"
clap = { version = "4", features = ["derive"] }
crc32fast = "1"
parking_lot = "0.12"
reqwest = { version = "0.13", default-features = false, features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
thiserror = "2"
prost = "0.14"
tokio = { version = "1", features = ["macros", "net", "rt-multi-thread", "sync", "time"] }
tonic = "0.14"
tonic-prost = "0.14"
tonic-prost-build = "0.14"
ureq = "3"
xxhash-rust = { version = "0.8", features = ["xxh3"] }

The log

Log owns the log file. It offers 2 operations: opening, which includes recovery, and appending a record.

Listing 2.8 The imports of kv/src/log.rs. kv/src/log.rs

use std::fs::{self, File, OpenOptions};
use std::io::{self, Read, Write};
use std::path::Path;

use crate::record::{Change, Decoded, Record};

Listing 2.9 The log, and the result of opening it. kv/src/log.rs

/// The name of the log inside the data directory.
pub const FILE_NAME: &str = "kv.log";

/// An open log, positioned at its end.
pub struct Log {
    file: File,
    /// Whether `append` waits for the disk before it returns.
    sync: bool,
    /// The sequence number of the next record.
    next: u64,
    /// Set by a failed write. A failed log accepts no more records.
    failed: bool,
}

/// What `Log::open` found in the file.
pub struct Recovery {
    /// The complete records, in the order they were written.
    pub records: Vec<Record>,
    /// The bytes after the last complete record, which were removed.
    pub discarded: u64,
}

Recovery

Opening a log reads the entire file and decodes records from the beginning until one fails to decode. Since Record::decode already distinguishes every possible outcome, the loop is short. Recovery treats an incomplete record and a corrupt one alike: in either case the record before it becomes the last record of the log.

Listing 2.10 Decoding records until one is incomplete or corrupt. kv/src/log.rs

/// Reads records from the front of `bytes` until one is incomplete or
/// corrupt. It returns the records and the number of bytes they occupy.
fn scan(bytes: &[u8]) -> (Vec<Record>, usize) {
    let mut records = Vec::new();
    let mut valid = 0;

    while let Decoded::Record { record, length } = Record::decode(&bytes[valid..]) {
        records.push(record);
        valid += length;
    }

    (records, valid)
}

Listing 2.11 Opening the log and cutting off a damaged end. kv/src/log.rs

impl Log {
    /// Opens the log in `directory`, creating both if they do not exist.
    /// Anything after the last complete record is cut off.
    pub fn open(directory: &Path, sync: bool) -> io::Result<(Log, Recovery)> {
        fs::create_dir_all(directory)?;

        let mut file = OpenOptions::new()
            .read(true)
            .append(true)
            .create(true)
            .open(directory.join(FILE_NAME))?;
        let mut bytes = Vec::new();
        file.read_to_end(&mut bytes)?;

        let (records, valid) = scan(&bytes);
        let discarded = (bytes.len() - valid) as u64;

        if discarded > 0 {
            file.set_len(valid as u64)?;
        }

        if sync {
            // The file's length is part of the file. Its name is part of the directory.
            file.sync_all()?;
            File::open(directory)?.sync_all()?;
        }

        let next = records
            .last()
            .map_or(1, |record| record.sequence + 1);

        Ok((
            Log {
                file,
                sync,
                next,
                failed: false,
            },
            Recovery { records, discarded },
        ))
    }
}

The call to set_len removes the damaged end of the file before any new record is appended, and the correctness of the whole log depends on this step. If the damaged bytes were left where they are, the next record would be written after them, a later recovery would stop at the damage, and the new record would be lost although the node had acknowledged it. A recovery procedure that reads correctly and fails to repair has merely postponed the loss.

Both calls to sync_all are necessary, one to make the new length of the file durable and the other for its directory. Synchronising a file makes its contents durable, but the existence of the file is recorded in the directory that contains it, and that is a separate object. A newly created file whose directory has not been synchronised may be absent after a power failure, contents and all. The log therefore synchronises its directory when it is opened.

Appending

Listing 2.12 Appending a record. kv/src/log.rs

impl Log {
    /// Appends a change and returns the record that holds it. When the log
    /// was opened with `sync`, the record is on the disk when this returns.
    pub fn append(&mut self, change: Change) -> io::Result<Record> {
        if self.failed {
            return Err(io::Error::other(
                "the log failed on an earlier write and is closed",
            ));
        }

        let record = Record {
            sequence: self.next,
            change,
        };

        if let Err(error) = self.write(&record.encode()) {
            self.failed = true;

            return Err(error);
        }

        self.next += 1;

        Ok(record)
    }

    fn write(&mut self, bytes: &[u8]) -> io::Result<()> {
        self.file.write_all(bytes)?;

        if self.sync {
            self.file.sync_data()?;
        }

        Ok(())
    }

    /// The sequence number of the last record, or 0 for an empty log.
    pub fn last_sequence(&self) -> u64 {
        self.next - 1
    }
}

The field failed implements a policy known as fail-stop, under which a component that meets an error it cannot handle safely ceases to operate. If write_all or sync_data returns an error, the state of the file is unknown: the record may be absent, partly written, or complete. A log that went on accepting records could append a valid record after a damaged one, where recovery would never reach it.

It might be thought that the node could call fsync again. This is unsafe as well, and for a reason that has surprised the authors of several well-known databases. After a failed fsync, the kernel may already have discarded the data it was unable to write, and a second call may then report success although nothing has reached the disk. A success after a failure therefore proves nothing. The log accordingly rejects all further appends. From that moment its node answers 503 to writes, continues to serve reads from memory, and has to be restarted or replaced by an operator. In the design of chapter 1, the followers of all the affected shards elect new leaders at this point, and the fault of 1 disk is absorbed by the cluster.

The store

Store contains the map and the log, and every modification of either passes through it.

Listing 2.13 The imports of kv/src/store.rs. kv/src/store.rs

use std::collections::BTreeMap;
use std::io;
use std::ops::Bound;
use std::path::Path;

use thiserror::Error;

use crate::log::Log;
use crate::record::{Change, Record};

Listing 2.14 Opening the store by applying the records of the log. kv/src/store.rs

impl Store {
    /// Opens the store in `directory` and applies every record of its log.
    /// It returns the store, the number of records applied, and the number
    /// of bytes that were cut from the end of the log.
    pub fn open(directory: &Path, sync: bool) -> io::Result<(Store, usize, u64)> {
        let (log, recovery) = Log::open(directory, sync)?;
        let applied = recovery.records.len();
        let mut entries = BTreeMap::new();

        for record in recovery.records {
            apply(&mut entries, record);
        }

        Ok((Store { entries, log }, applied, recovery.discarded))
    }
}

/// Makes the change of one record in the map. Recovery and the write path
/// both go through this function, so they cannot disagree.
pub fn apply(entries: &mut BTreeMap<String, Entry>, record: Record) {
    match record.change {
        Change::Put { key, value } => {
            entries.insert(
                key,
                Entry {
                    value,
                    version: record.sequence,
                },
            );
        }
        Change::Delete { key } => {
            entries.remove(&key);
        }
    }
}

Recovery and the path of a write share a single function, apply, for modifying the map. Invariant 3 is a consequence of this arrangement. What a record does to the map is defined in exactly 1 place, which is used both when a record is first written and whenever a log is read back, so that the 2 cannot come to differ.

Listing 2.15 Conditional writes: check, append, apply. kv/src/store.rs

impl Store {
    /// Sets the value of a key. The change is in the log before it is in
    /// the map.
    pub fn put(
        &mut self,
        key: String,
        value: Vec<u8>,
        condition: Condition,
    ) -> Result<Written, StoreError> {
        let current = self.version(&key);
        check(condition, current)?;

        let record = self
            .log
            .append(Change::Put { key, value })?;
        let version = record.sequence;
        apply(&mut self.entries, record);

        Ok(Written {
            version,
            created: current.is_none(),
        })
    }

    /// Removes a key.
    pub fn delete(&mut self, key: &str, condition: Condition) -> Result<(), StoreError> {
        let current = self.version(key);

        if current.is_none() {
            return Err(StoreError::NotFound);
        }

        check(condition, current)?;

        let record = self.log.append(Change::Delete {
            key: key.to_owned(),
        })?;
        apply(&mut self.entries, record);

        Ok(())
    }

    fn version(&self, key: &str) -> Option<u64> {
        self.entries
            .get(key)
            .map(|entry| entry.version)
    }
}

pub fn check(condition: Condition, current: Option<u64>) -> Result<(), StoreError> {
    let holds = match condition {
        Condition::Always => true,
        Condition::Absent => current.is_none(),
        Condition::Version(version) => current == Some(version),
    };

    holds
        .then_some(())
        .ok_or(StoreError::ConditionFailed { current })
}

put and delete perform the same 3 steps in the same order. The condition is evaluated first, against the map, so that a write which is refused produces no record. append then returns when the record is durable, or returns an error, in which case the ? operator leaves the function before apply is reached. The map is modified last. The order of these steps is the order of the invariants, and the code has been arranged so that it cannot be otherwise.

Reads use the map alone.

Listing 2.16 Reading a key, and listing a page of keys. kv/src/store.rs

impl Store {
    pub fn get(&self, key: &str) -> Option<&Entry> {
        self.entries.get(key)
    }

    /// Returns up to `limit` keys that start with `prefix` and sort after
    /// `after`, in order, and whether more keys follow them.
    pub fn list(&self, prefix: &str, after: Option<&str>, limit: usize) -> (Vec<Summary>, bool) {
        page(&self.entries, prefix, after, limit)
    }

    pub fn len(&self) -> usize {
        self.entries.len()
    }

    pub fn is_empty(&self) -> bool {
        self.entries.is_empty()
    }

    /// The sequence number of the last record in the log.
    pub fn last_sequence(&self) -> u64 {
        self.log.last_sequence()
    }
}

/// One page of a listing over a map of entries.
pub fn page(
    entries: &BTreeMap<String, Entry>,
    prefix: &str,
    after: Option<&str>,
    limit: usize,
) -> (Vec<Summary>, bool) {
    let start = match after {
        Some(key) => Bound::Excluded(key),
        None => Bound::Included(prefix),
    };

    let mut matching = entries
        .range::<str, _>((start, Bound::Unbounded))
        .take_while(|entry| entry.0.starts_with(prefix))
        .map(|(key, entry)| Summary {
            key: key.clone(),
            version: entry.version,
            size: entry.value.len(),
        });

    let page: Vec<Summary> = matching
        .by_ref()
        .take(limit)
        .collect();
    let more = matching.next().is_some();

    (page, more)
}

A BTreeMap keeps its keys in sorted order, so that a listing is an iteration from some starting position, which is implemented in a function named page. Its call to range begins at the prefix or immediately after the cursor, and take_while ends the iteration at the first key that lacks the prefix. By looking for 1 item beyond the page, page learns whether a further page exists without having to count the remaining keys.

The HTTP server

kvd presents the store through the API of chapter 1.

Listing 2.17 The imports of kvd/src/main.rs. kvd/src/main.rs

use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;

use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, Query, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_NONE_MATCH};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use kv::{Condition, Store, StoreError, Summary};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, version_header, write_condition};
use parking_lot::Mutex;
use serde::{Deserialize, Serialize};
use tokio::net::TcpListener;
use tokio::task;

Listing 2.18 The options of the command line, and the shared state. kvd/src/main.rs

#[derive(Parser)]
struct Options {
    /// The address to accept connections on.
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    /// The directory that holds the log.
    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// Respond to a write without waiting for the disk.
    #[arg(long)]
    no_sync: bool,
}

/// The store, shared by every request.
type Shared = Arc<Mutex<Store>>;

Errors

Each kind of failure of a request is a variant of ApiError, in the module kvd::api, and a single function converts a variant into a status code and a response body in the format of RFC 9457. Handlers return Result<_, ApiError> and use ?. The conversion from StoreError is the 1 place that maps a failed condition to 412 and a failure of the log to 503.

Listing 2.19 The imports of kvd/src/api.rs. kvd/src/api.rs

use axum::Json;
use axum::http::header::{CONTENT_TYPE, IF_MATCH, IF_NONE_MATCH};
use axum::http::{HeaderMap, HeaderName, StatusCode};
use axum::response::{IntoResponse, Response};
use kv::{Condition, StoreError};
use serde::Serialize;

Listing 2.20 The errors of the API and their responses. kvd/src/api.rs

/// The ways a request can fail. Each one is a status code and a sentence.
pub enum ApiError {
    InvalidKey,
    InvalidHeader(HeaderName),
    NotFound,
    ConditionFailed(Option<u64>),
    Unavailable(String),
    /// The request reached a node that does not serve it. The text names the reason.
    Misdirected(String),
    /// The shard has no leader at the moment. Nothing was changed.
    NoLeader,
}

/// The body of an error response, in the format of RFC 9457.
#[derive(Serialize)]
struct Problem {
    #[serde(rename = "type")]
    kind: &'static str,
    title: &'static str,
    status: u16,
    #[serde(skip_serializing_if = "String::is_empty")]
    detail: String,
}

impl IntoResponse for ApiError {
    fn into_response(self) -> Response {
        let (status, kind, title, detail) = match self {
            ApiError::InvalidKey => (
                StatusCode::BAD_REQUEST,
                "/problems/invalid-key",
                "The key is not valid",
                format!("a key is 1 to {MAX_KEY} characters from A-Z, a-z, 0-9, '.', '_' and '-'"),
            ),
            ApiError::InvalidHeader(name) => (
                StatusCode::BAD_REQUEST,
                "/problems/invalid-header",
                "A request header is not valid",
                format!("{name} must be a version in double quotes, or * where that is allowed"),
            ),
            ApiError::NotFound => (
                StatusCode::NOT_FOUND,
                "/problems/key-not-found",
                "The key does not exist",
                String::new(),
            ),
            ApiError::ConditionFailed(current) => (
                StatusCode::PRECONDITION_FAILED,
                "/problems/condition-failed",
                "The condition on the key's version does not hold",
                current.map_or("the key does not exist".into(), |version| {
                    format!("the key is at version {version}")
                }),
            ),
            ApiError::Misdirected(detail) => (
                StatusCode::MISDIRECTED_REQUEST,
                "/problems/not-leader",
                "This node is not the leader",
                detail,
            ),
            ApiError::NoLeader => (
                StatusCode::SERVICE_UNAVAILABLE,
                "/problems/no-leader",
                "The shard has no leader at the moment",
                String::new(),
            ),
            ApiError::Unavailable(detail) => (
                StatusCode::SERVICE_UNAVAILABLE,
                "/problems/store-unavailable",
                "The node cannot serve the request",
                detail,
            ),
        };

        let problem = Problem {
            kind,
            title,
            status: status.as_u16(),
            detail,
        };

        (
            status,
            [(CONTENT_TYPE, "application/problem+json")],
            Json(problem),
        )
            .into_response()
    }
}

impl From<StoreError> for ApiError {
    fn from(error: StoreError) -> Self {
        match error {
            StoreError::NotFound => ApiError::NotFound,
            StoreError::ConditionFailed { current } => ApiError::ConditionFailed(current),
            StoreError::Log(error) => ApiError::Unavailable(error.to_string()),
        }
    }
}

Conditions

Listing 2.21 Parsing the If-Match and If-None-Match headers. kvd/src/api.rs

/// A version as it appears in `ETag`, `If-Match` and `If-None-Match`.
pub fn etag(version: u64) -> String {
    format!("\"{version}\"")
}

pub fn valid_key(key: &str) -> Result<(), ApiError> {
    let allowed = |c: char| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-');

    let valid = (1..=MAX_KEY).contains(&key.len()) && key.chars().all(allowed);

    valid
        .then_some(())
        .ok_or(ApiError::InvalidKey)
}

/// Reads a header that holds a quoted version or `*`.
pub fn version_header(
    headers: &HeaderMap,
    name: HeaderName,
) -> Result<Option<Condition>, ApiError> {
    let Some(value) = headers.get(&name) else {
        return Ok(None);
    };

    let Ok(text) = value.to_str() else {
        return Err(ApiError::InvalidHeader(name));
    };

    if text == "*" {
        return Ok(Some(Condition::Absent));
    }

    let version = text
        .strip_prefix("\"")
        .and_then(|rest| rest.strip_suffix("\""))
        .and_then(|digits| digits.parse().ok());

    version
        .map(Condition::Version)
        .map(Some)
        .ok_or(ApiError::InvalidHeader(name))
}

/// The condition a write carries: `If-Match: "7"` requires version 7, and
/// `If-None-Match: *` requires that the key does not exist.
pub fn write_condition(headers: &HeaderMap) -> Result<Condition, ApiError> {
    match (
        version_header(headers, IF_MATCH)?,
        version_header(headers, IF_NONE_MATCH)?,
    ) {
        (Some(Condition::Version(version)), None) => Ok(Condition::Version(version)),
        (None, Some(Condition::Absent)) => Ok(Condition::Absent),
        (None, None) => Ok(Condition::Always),
        (Some(..), ..) => Err(ApiError::InvalidHeader(IF_MATCH)),
        (None, Some(..)) => Err(ApiError::InvalidHeader(IF_NONE_MATCH)),
    }
}

write_condition accepts the combinations of headers that are meaningful for a write and answers 400 to all others, such as If-Match: *. The server thus rejects a condition that it cannot interpret. The alternative, to ignore it, would mean performing an unconditional write on behalf of a client that had asked for a conditional one, which is exactly the lost update that the condition was sent to prevent.

Blocking calls in an asynchronous server

The server uses an asynchronous runtime, in which a small pool of threads serves all the connections and switches among them whenever one has to wait for the network. To block such a thread, as an fsync of 5 ms does, would delay every connection that the thread serves. spawn_blocking runs a closure on a separate pool of threads intended for operations that block, and the handler awaits the result without occupying a thread of the runtime.

Listing 2.22 Running a write on a thread that is permitted to block. kvd/src/main.rs

/// Runs a write on a thread that may block. A write waits for the disk, and
/// a thread that waits must not be one of the threads that serve requests.
async fn write<T, F>(store: Shared, work: F) -> Result<T, ApiError>
where
    F: FnOnce(&mut Store) -> Result<T, StoreError> + Send + 'static,
    T: Send + 'static,
{
    let result = task::spawn_blocking(move || work(&mut store.lock()))
        .await
        .map_err(|error| ApiError::Unavailable(error.to_string()))?;

    Ok(result?)
}

Listing 2.23 The handlers for PUT and DELETE. kvd/src/main.rs

async fn put_key(
    State(store): State<Shared>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    let written = write(store, move |store| store.put(key, body.to_vec(), condition)).await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(store): State<Shared>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    write(store, move |store| store.delete(&key, condition)).await?;

    Ok(StatusCode::NO_CONTENT)
}

PUT answers 201 Created when it creates a key and 204 No Content when it replaces a value. Both responses carry the new version in the ETag header.

Reads

Listing 2.24 The handler for GET on a single key. kvd/src/main.rs

async fn get_key(
    State(store): State<Shared>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let cached = version_header(&headers, IF_NONE_MATCH)?;
    let entry = store
        .lock()
        .get(&key)
        .cloned()
        .ok_or(ApiError::NotFound)?;
    let tag = [(ETAG, etag(entry.version))];

    if cached == Some(Condition::Version(entry.version)) {
        return Ok((StatusCode::NOT_MODIFIED, tag).into_response());
    }

    Ok((
        tag,
        [(CONTENT_TYPE, "application/octet-stream")],
        entry.value,
    )
        .into_response())
}

A read holds the lock of the store only while an entry is being copied, and performs no access to the disk. It can therefore run directly on a thread of the runtime.

Listing 2.25 The handlers for listing keys and for the health check. kvd/src/main.rs

#[derive(Deserialize)]
struct ListQuery {
    #[serde(default)]
    prefix: String,
    after: Option<String>,
    limit: Option<usize>,
}

#[derive(Serialize)]
struct Item {
    key: String,
    version: u64,
    size: usize,
}

#[derive(Serialize)]
struct Page {
    keys: Vec<Item>,
    /// The value of `after` that returns the next page, when there is one.
    next_after: Option<String>,
}

async fn list_keys(State(store): State<Shared>, Query(query): Query<ListQuery>) -> Json<Page> {
    let limit = query
        .limit
        .unwrap_or(100)
        .clamp(1, 1000);
    let (page, more) = store
        .lock()
        .list(&query.prefix, query.after.as_deref(), limit);
    let next_after = page
        .last()
        .filter(|_last| more)
        .map(|last| last.key.clone());
    let keys = page
        .into_iter()
        .map(|Summary { key, version, size }| Item { key, version, size })
        .collect();

    Json(Page { keys, next_after })
}

#[derive(Serialize)]
struct Health {
    status: &'static str,
    keys: usize,
    last_sequence: u64,
}

async fn health(State(store): State<Shared>) -> Json<Health> {
    let store = store.lock();

    Json(Health {
        status: "ok",
        keys: store.len(),
        last_sequence: store.last_sequence(),
    })
}

Listing 2.26 Opening the store and starting the server. kvd/src/main.rs

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let (store, applied, discarded) = Store::open(&options.data, !options.no_sync)?;

    println!(
        "recovered {} keys from {applied} records, cut {discarded} bytes from the end of the log",
        store.len()
    );

    let routes = Router::new()
        .route("/v1/keys", get(list_keys))
        .route(
            "/v1/keys/{key}",
            get(get_key)
                .put(put_key)
                .delete(delete_key),
        )
        .route("/healthz", get(health))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(Arc::new(Mutex::new(store)));

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Running the node

$ cd code
$ cargo build --release
$ ./target/release/kvd --listen 127.0.0.1:7001 --data /var/tmp/kvd
recovered 0 keys from 0 records, cut 0 bytes from the end of the log
listening on 127.0.0.1:7001

All the responses shown in chapter 1 came from this program. A further 3 complete the set: a key containing a character outside the permitted set, a method that a resource does not support, and a health check.

$ curl -i -X PUT 'http://127.0.0.1:7001/v1/keys/bad%20key' --data-binary 'x'
HTTP/1.1 400 Bad Request
content-type: application/problem+json
content-length: 153

{"type":"/problems/invalid-key","title":"The key is not valid","status":400,"detail":"a key is 1 to 256 characters from A-Z, a-z, 0-9, '.', '_' and '-'"}

$ curl -i -X POST http://127.0.0.1:7001/v1/keys/user-1 --data-binary 'x'
HTTP/1.1 405 Method Not Allowed
allow: GET,HEAD,PUT,DELETE
content-length: 0

Both the 405 and its allow header are produced by axum, whose router knows which methods are registered for each path.

$ curl -i http://127.0.0.1:7001/healthz
HTTP/1.1 200 OK
content-type: application/json
content-length: 42

{"status":"ok","keys":2,"last_sequence":5}

Simulating crashes

The log is intended to recover exactly its complete records, wherever a crash occurs. A claim of this kind is not established by a few examples, and crashsim verifies it exhaustively, for a log of 5 records, by damaging copies of the log in every possible position.

Listing 2.27 The imports of crashsim/src/main.rs. crashsim/src/main.rs

use std::env;
use std::error::Error;
use std::fs;
use std::path::Path;
use std::process;

use kv::log::FILE_NAME;
use kv::{Change, Log, Record};

Listing 2.28 Building the test log. crashsim/src/main.rs

/// The changes of the test log.
fn changes() -> Vec<Change> {
    let put = |key: &str, value: &str| Change::Put {
        key: key.into(),
        value: value.into(),
    };

    vec![
        put("user-1", "ada"),
        put("user-2", "grace"),
        put("user-1", "ada lovelace"),
        Change::Delete {
            key: "user-2".into(),
        },
        put("user-3", "barbara"),
    ]
}

/// Writes the changes to a log in `directory`. It returns the bytes of the
/// log and the offset at which each record ends.
fn build(directory: &Path) -> Result<(Vec<u8>, Vec<usize>), Box<dyn Error>> {
    let mut log = Log::open(directory, false)?.0;
    let mut ends = Vec::new();

    for change in changes() {
        let record = log.append(change)?;
        ends.push(ends.last().copied().unwrap_or(0) + record.encode().len());
    }

    Ok((fs::read(directory.join(FILE_NAME))?, ends))
}

Listing 2.29 Opening a damaged copy, and the expected result of recovery. crashsim/src/main.rs

/// Replaces the log in `directory` with `bytes` and opens it.
fn recover(directory: &Path, bytes: &[u8]) -> Result<Vec<Record>, Box<dyn Error>> {
    fs::write(directory.join(FILE_NAME), bytes)?;
    let recovery = Log::open(directory, false)?.1;

    Ok(recovery.records)
}

/// Whether `records` are exactly the first `count` changes, in order.
fn is_prefix(records: &[Record], count: usize) -> bool {
    let expected = changes();

    records.len() == count
        && records
            .iter()
            .zip(&expected)
            .enumerate()
            .all(|(index, (record, change))| {
                record.sequence == index as u64 + 1 && &record.change == change
            })
}

A crash during a write leaves a file that is shorter than was intended. cuts creates every such file, 1 for each possible length, and checks the result of recovering each.

Listing 2.30 Cutting the log short at every byte. crashsim/src/main.rs

/// Cuts the log at every length and checks that recovery returns the records
/// that end at or before the cut.
fn cuts(directory: &Path, log: &[u8], ends: &[usize]) -> Result<(), Box<dyn Error>> {
    for length in 0..=log.len() {
        let complete = ends
            .iter()
            .filter(|&&end| end <= length)
            .count();
        let records = recover(directory, &log[..length])?;

        if !is_prefix(&records, complete) {
            return Err(format!(
                "cut at {length}: expected {complete} records, recovered {}",
                records.len()
            )
            .into());
        }
    }

    println!(
        "cut at every length from 0 to {}: each recovery returned the complete records",
        log.len()
    );

    let mut first = 0;

    for (count, &end) in ends.iter().enumerate() {
        println!("  lengths {first:3} to {:3}: {count} records", end - 1);
        first = end;
    }

    println!("  length  {first:3}       : {} records", ends.len());

    Ok(())
}

Storage devices are also capable of returning data that differs from what was written. flips inverts 1 bit in each byte of the log, in a separate run for every byte. With a bit inverted, the affected record fails its checksum, and recovery is expected to return all the records before it.

Listing 2.31 Inverting 1 bit in every byte. crashsim/src/main.rs

/// Flips 1 bit in each byte in turn and checks that recovery returns the
/// records before the damaged one.
fn flips(directory: &Path, log: &[u8], ends: &[usize]) -> Result<(), Box<dyn Error>> {
    for position in 0..log.len() {
        let mut damaged = log.to_vec();
        damaged[position] ^= 0x10;

        let before = ends
            .iter()
            .filter(|&&end| end <= position)
            .count();
        let records = recover(directory, &damaged)?;

        if !is_prefix(&records, before) {
            return Err(format!(
                "flip at {position}: expected {before} records, recovered {}",
                records.len()
            )
            .into());
        }
    }

    println!(
        "1 bit flipped in each of {} bytes: each recovery returned the records before the damage",
        log.len()
    );

    Ok(())
}

Listing 2.32 Running the simulation. crashsim/src/main.rs

fn run(directory: &Path) -> Result<(), Box<dyn Error>> {
    let (log, ends) = build(directory)?;
    let boundaries: Vec<String> = ends
        .iter()
        .map(usize::to_string)
        .collect();

    println!(
        "a log of {} records and {} bytes, with records ending at {}",
        ends.len(),
        log.len(),
        boundaries.join(", ")
    );

    cuts(directory, &log, &ends)?;
    flips(directory, &log, &ends)?;

    let records = recover(directory, &log[..ends[1] + 5])?;
    let mut reopened = Log::open(directory, false)?.0;
    let next = reopened.append(Change::Delete {
        key: "user-1".into(),
    })?;

    println!(
        "after a cut at {}: {} records recovered, and the next record has sequence {}",
        ends[1] + 5,
        records.len(),
        next.sequence
    );

    Ok(())
}

fn main() -> Result<(), Box<dyn Error>> {
    let directory = env::temp_dir().join(format!("crashsim-{}", process::id()));
    let result = run(&directory);

    fs::remove_dir_all(&directory)?;

    result
}

$ cargo run -p crashsim
a log of 5 records and 152 bytes, with records ending at 28, 58, 95, 120, 152
cut at every length from 0 to 152: each recovery returned the complete records
  lengths   0 to  27: 0 records
  lengths  28 to  57: 1 records
  lengths  58 to  94: 2 records
  lengths  95 to 119: 3 records
  lengths 120 to 151: 4 records
  length  152       : 5 records
1 bit flipped in each of 152 bytes: each recovery returned the records before the damage
after a cut at 63: 2 records recovered, and the next record has sequence 3

The test with inverted bits also discloses a limitation. When a bit is inverted in record 2, recovery stops there and discards records 3 to 5, although they are intact. For damage at the end of the file, which is what an interrupted write produces, this behaviour is correct. Damage in the middle of the file is a different matter: it means that the disk has lost data which was acknowledged, and a single node has no means of repairing such a loss. In the design of chapter 1, the record is recovered from the other replicas, and this is the first of several points at which a limitation of 1 node is the reason for something in the larger design.

Killing the server under load

The simulation exercises the log in isolation. The next test covers a complete node: the HTTP server, the store, the log and the disk. The command kvctl load writes keys in sequence and records each key in a local file once the server has answered with success.

Listing 2.33 The imports of kvctl/src/main.rs. kvctl/src/main.rs

use std::error::Error;
use std::fs::{self, File};
use std::io::{BufWriter, Write};
use std::path::PathBuf;

use clap::{Parser, Subcommand};
use ureq::Agent;

Listing 2.34 Writing keys, and recording each acknowledged key. kvctl/src/main.rs

/// Writes `count` keys. A key is recorded in `acked` only after the server
/// has answered with a success status.
fn load(agent: &Agent, server: &str, count: u32, acked: &PathBuf) -> Result<(), Box<dyn Error>> {
    let mut file = BufWriter::new(File::create(acked)?);
    let mut written = 0;

    for number in 1..=count {
        let key = format!("key-{number:06}");

        if let Err(error) = agent
            .put(format!("{server}/v1/keys/{key}"))
            .send(format!("value of {key}"))
        {
            println!("write {number} failed: {error}");

            break;
        }

        writeln!(file, "{key}")?;
        written += 1;
    }

    file.flush()?;
    println!("{written} writes acknowledged");

    Ok(())
}

Listing 2.35 Reading back every acknowledged key. kvctl/src/main.rs

/// Reads every acknowledged key and compares the value with the one that
/// was written.
fn verify(agent: &Agent, server: &str, acked: &PathBuf) -> Result<(), Box<dyn Error>> {
    let keys = fs::read_to_string(acked)?;
    let (mut found, mut missing) = (0, 0);

    for key in keys.lines() {
        let value = match agent
            .get(format!("{server}/v1/keys/{key}"))
            .call()
        {
            Ok(mut response) => response
                .body_mut()
                .read_to_string()?,
            Err(ureq::Error::StatusCode(404)) => String::new(),
            Err(error) => return Err(error.into()),
        };

        if value == format!("value of {key}") {
            found += 1;
        } else {
            missing += 1;
        }
    }

    println!(
        "{} acknowledged, {found} found, {missing} missing",
        found + missing
    );

    Ok(())
}

While the load is running, the server is terminated with signal 9, SIGKILL, which stops a process at once and allows it to complete nothing that is in progress.

$ ./target/release/kvd --data /var/tmp/kvd &
recovered 0 keys from 0 records, cut 0 bytes from the end of the log
listening on 127.0.0.1:7001
$ ./target/release/kvctl load --server http://127.0.0.1:7001 --count 5000 --acked acked.txt &
$ sleep 3; pkill -9 -x kvd
write 495 failed: io: Peer disconnected
494 writes acknowledged
$ ./target/release/kvd --data /var/tmp/kvd &
recovered 495 keys from 495 records, cut 0 bytes from the end of the log
listening on 127.0.0.1:7001
$ ./target/release/kvctl verify --server http://127.0.0.1:7001 --acked acked.txt
494 acknowledged, 494 found, 0 missing

Write 495 is an instance of the third case in figure 2.2. Its record was stored durably, and the process was killed before any response was sent. From the client’s side the write failed, although the store contains it, and a client that needs a definite outcome sends its PUT again.

The limits of this test

The same test, run with --no-sync, also recovers every acknowledged write.

$ ./target/release/kvd --data /var/tmp/kvd --no-sync &
$ ./target/release/kvctl load --server http://127.0.0.1:7001 --count 200000 --acked acked.txt &
$ sleep 1.5; pkill -9 -x kvd
write 8595 failed: io: Connection reset by peer (os error 104)
8594 writes acknowledged
$ ./target/release/kvd --data /var/tmp/kvd --no-sync &
recovered 8594 keys from 8594 records, cut 0 bytes from the end of the log
$ ./target/release/kvctl verify --server http://127.0.0.1:7001 --acked acked.txt
8594 acknowledged, 8594 found, 0 missing

It would be a serious error to conclude from this that fsync is unnecessary. When a process is killed, its own memory is lost, but the data it had passed to write is in the kernel’s buffers. The kernel continues to run, and writes that data to the disk shortly afterwards. Only a failure that stops the kernel, such as a loss of power, loses data that was written without fsync. A test that kills a process therefore succeeds with and without fsync, and provides no evidence whatever about durability under a loss of power. Many systems have been declared durable on the strength of exactly this test.

Running the test on 2 machines

The server and the client are separate programs that communicate over HTTP, and can therefore run on different machines without any change. In the following session, the server runs on a machine named node-b at the address 10.0.0.2, and the client on node-a. A binary can be copied between machines of the same architecture and operating system, and is otherwise built on each.

node-a$ scp target/release/kvd node-b:/tmp/kvd
node-b$ /tmp/kvd --listen 0.0.0.0:7001 --data /tmp/kvd-data &
recovered 0 keys from 0 records, cut 0 bytes from the end of the log
listening on 0.0.0.0:7001

node-a$ ./target/release/kvctl load --server http://10.0.0.2:7001 --count 5000 --acked acked.txt &
node-b$ pkill -9 -x kvd
write 1108 failed: io: Connection reset by peer (os error 104)
1107 writes acknowledged

node-b$ /tmp/kvd --listen 0.0.0.0:7001 --data /tmp/kvd-data &
recovered 1107 keys from 1107 records, cut 0 bytes from the end of the log
node-a$ ./target/release/kvctl verify --server http://10.0.0.2:7001 --acked acked.txt
1107 acknowledged, 1107 found, 0 missing
node-a$ curl http://10.0.0.2:7001/healthz
{"status":"ok","keys":1107,"last_sequence":1107}

The client’s file of acknowledged keys is held on a machine that no fault affected, and is on that account an independent record of what the server acknowledged. Tests of faults follow this structure quite generally. A component outside the system under test records what was acknowledged, a fault is injected, and the record is compared with the state of the system after it has recovered. Chapters 3 and 4 test replication and elections in the same way.

The test can be run in several configurations.

The limits of a single node

Within these limits, the node retains every acknowledged write when its process is killed, when its log is cut short at any byte, and when a bit is corrupted in its last record. That is the assumption on which chapter 1 was built, and chapter 3 now builds replication upon it.

The complete programs

Listing 2.36 The complete kv/src/lib.rs. kv/src/lib.rs

//! A key-value store for one node that keeps every acknowledged write across
//! a crash. A write is appended to a log on disk before it is applied in
//! memory, and the log is read back when the store is opened.

pub mod log;
pub mod record;
pub mod store;

pub use log::{Log, Recovery};
pub use record::{Change, Decoded, Record};
pub use store::{Condition, Entry, Store, StoreError, Summary, Write, Written, apply, check, page};

Listing 2.37 The complete kv/src/record.rs. kv/src/record.rs

//! The format of one record of the log.
//!
//! A record is a frame of 8 bytes followed by a payload:
//!
//! ```text
//! length: u32 | checksum: u32 | sequence: u64 | kind: u8 | key length: u16 | key | value
//! ```
//!
//! `length` counts the bytes of the payload and `checksum` is the CRC-32 of
//! the payload. All integers are little-endian.

use crc32fast::hash;

/// A change to the store.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Change {
    Put { key: String, value: Vec<u8> },
    Delete { key: String },
}

/// A change with its position in the log. Sequence numbers start at 1 and
/// grow by 1 with each record.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Record {
    pub sequence: u64,
    pub change: Change,
}

/// The result of reading one record from the front of a slice of bytes.
#[derive(Debug, PartialEq, Eq)]
pub enum Decoded {
    /// A whole record, and the number of bytes it occupies.
    Record { record: Record, length: usize },
    /// The bytes end before the record does.
    Incomplete,
    /// The checksum does not match the payload, or the payload is malformed.
    Corrupt,
}

/// The bytes of a frame: the length and the checksum.
pub const FRAME: usize = 8;

/// The largest payload a record may have: a key, a value of 1 MiB, and the
/// fixed fields.
pub const MAX_PAYLOAD: usize = (1 << 20) + 1024;

const PUT: u8 = 1;
const DELETE: u8 = 2;

impl Record {
    /// Returns the record as the bytes that are appended to the log.
    pub fn encode(&self) -> Vec<u8> {
        let (kind, key, value) = match &self.change {
            Change::Put { key, value } => (PUT, key, value.as_slice()),
            Change::Delete { key } => (DELETE, key, &[][..]),
        };

        let mut payload = Vec::with_capacity(11 + key.len() + value.len());
        payload.extend_from_slice(&self.sequence.to_le_bytes());
        payload.push(kind);
        payload.extend_from_slice(&(key.len() as u16).to_le_bytes());
        payload.extend_from_slice(key.as_bytes());
        payload.extend_from_slice(value);

        let mut bytes = Vec::with_capacity(FRAME + payload.len());
        bytes.extend_from_slice(&(payload.len() as u32).to_le_bytes());
        bytes.extend_from_slice(&hash(&payload).to_le_bytes());
        bytes.extend_from_slice(&payload);

        bytes
    }
}

impl Record {
    /// Reads the record at the front of `bytes`.
    pub fn decode(bytes: &[u8]) -> Decoded {
        let Some((length, rest)) = bytes.split_first_chunk::<4>() else {
            return Decoded::Incomplete;
        };
        let Some((checksum, rest)) = rest.split_first_chunk::<4>() else {
            return Decoded::Incomplete;
        };

        let length = u32::from_le_bytes(*length) as usize;

        if length > MAX_PAYLOAD {
            return Decoded::Corrupt;
        }

        let Some(payload) = rest.get(..length) else {
            return Decoded::Incomplete;
        };

        if hash(payload) != u32::from_le_bytes(*checksum) {
            return Decoded::Corrupt;
        }

        match parse(payload) {
            Some(record) => Decoded::Record {
                record,
                length: FRAME + length,
            },
            None => Decoded::Corrupt,
        }
    }
}

/// Splits a payload into its fields. It returns `None` for a payload that is
/// too short for its fields or has an unknown kind.
fn parse(payload: &[u8]) -> Option<Record> {
    let (sequence, rest) = payload.split_first_chunk::<8>()?;
    let (&kind, rest) = rest.split_first()?;
    let (key_length, rest) = rest.split_first_chunk::<2>()?;
    let (key, value) = rest.split_at_checked(u16::from_le_bytes(*key_length) as usize)?;
    let key = String::from_utf8(key.to_vec()).ok()?;

    let change = match (kind, value) {
        (PUT, value) => Change::Put {
            key,
            value: value.to_vec(),
        },
        (DELETE, []) => Change::Delete { key },
        (_unknown, _value) => return None,
    };

    Some(Record {
        sequence: u64::from_le_bytes(*sequence),
        change,
    })
}

Listing 2.38 The complete kv/src/log.rs. kv/src/log.rs

//! The write-ahead log: a file that only grows, to which every change is
//! appended before it is applied.

use std::fs::{self, File, OpenOptions};
use std::io::{self, Read, Write};
use std::path::Path;

use crate::record::{Change, Decoded, Record};

/// The name of the log inside the data directory.
pub const FILE_NAME: &str = "kv.log";

/// An open log, positioned at its end.
pub struct Log {
    file: File,
    /// Whether `append` waits for the disk before it returns.
    sync: bool,
    /// The sequence number of the next record.
    next: u64,
    /// Set by a failed write. A failed log accepts no more records.
    failed: bool,
}

/// What `Log::open` found in the file.
pub struct Recovery {
    /// The complete records, in the order they were written.
    pub records: Vec<Record>,
    /// The bytes after the last complete record, which were removed.
    pub discarded: u64,
}

/// Reads records from the front of `bytes` until one is incomplete or
/// corrupt. It returns the records and the number of bytes they occupy.
fn scan(bytes: &[u8]) -> (Vec<Record>, usize) {
    let mut records = Vec::new();
    let mut valid = 0;

    while let Decoded::Record { record, length } = Record::decode(&bytes[valid..]) {
        records.push(record);
        valid += length;
    }

    (records, valid)
}

impl Log {
    /// Opens the log in `directory`, creating both if they do not exist.
    /// Anything after the last complete record is cut off.
    pub fn open(directory: &Path, sync: bool) -> io::Result<(Log, Recovery)> {
        fs::create_dir_all(directory)?;

        let mut file = OpenOptions::new()
            .read(true)
            .append(true)
            .create(true)
            .open(directory.join(FILE_NAME))?;
        let mut bytes = Vec::new();
        file.read_to_end(&mut bytes)?;

        let (records, valid) = scan(&bytes);
        let discarded = (bytes.len() - valid) as u64;

        if discarded > 0 {
            file.set_len(valid as u64)?;
        }

        if sync {
            // The file's length is part of the file. Its name is part of the directory.
            file.sync_all()?;
            File::open(directory)?.sync_all()?;
        }

        let next = records
            .last()
            .map_or(1, |record| record.sequence + 1);

        Ok((
            Log {
                file,
                sync,
                next,
                failed: false,
            },
            Recovery { records, discarded },
        ))
    }
}

impl Log {
    /// Appends a change and returns the record that holds it. When the log
    /// was opened with `sync`, the record is on the disk when this returns.
    pub fn append(&mut self, change: Change) -> io::Result<Record> {
        if self.failed {
            return Err(io::Error::other(
                "the log failed on an earlier write and is closed",
            ));
        }

        let record = Record {
            sequence: self.next,
            change,
        };

        if let Err(error) = self.write(&record.encode()) {
            self.failed = true;

            return Err(error);
        }

        self.next += 1;

        Ok(record)
    }

    fn write(&mut self, bytes: &[u8]) -> io::Result<()> {
        self.file.write_all(bytes)?;

        if self.sync {
            self.file.sync_data()?;
        }

        Ok(())
    }

    /// The sequence number of the last record, or 0 for an empty log.
    pub fn last_sequence(&self) -> u64 {
        self.next - 1
    }
}

impl Log {
    /// Appends records that another node numbered, in one write and with one
    /// wait for the disk. The first record must follow the last record of
    /// this log.
    pub fn append_records(&mut self, records: &[Record]) -> io::Result<()> {
        if self.failed {
            return Err(io::Error::other(
                "the log failed on an earlier write and is closed",
            ));
        }

        let misplaced = (self.next..)
            .zip(records)
            .find(|(expected, record)| record.sequence != *expected);

        if let Some((expected, record)) = misplaced {
            return Err(io::Error::other(format!(
                "record {} cannot follow record {}",
                record.sequence,
                expected - 1
            )));
        }

        let bytes: Vec<u8> = records
            .iter()
            .flat_map(Record::encode)
            .collect();

        if let Err(error) = self.write(&bytes) {
            self.failed = true;

            return Err(error);
        }

        self.next += records.len() as u64;

        Ok(())
    }
}

impl Log {
    /// Removes the records after record `keep`, which ends at byte `length`
    /// of the file. A follower uses this to discard records that were never
    /// committed and that differ from the leader's.
    pub fn truncate(&mut self, keep: u64, length: u64) -> io::Result<()> {
        self.file.set_len(length)?;

        if self.sync {
            self.file.sync_all()?;
        }

        self.next = keep + 1;

        Ok(())
    }
}

Listing 2.39 The complete kv/src/store.rs. kv/src/store.rs

//! The store: the current value of every key in memory, and the log that
//! rebuilds it.

use std::collections::BTreeMap;
use std::io;
use std::ops::Bound;
use std::path::Path;

use thiserror::Error;

use crate::log::Log;
use crate::record::{Change, Record};

/// The value of a key and the sequence number of the record that wrote it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Entry {
    pub value: Vec<u8>,
    pub version: u64,
}

/// What a write requires of the key's current state.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Condition {
    /// The write is unconditional.
    Always,
    /// The key must not exist.
    Absent,
    /// The key must exist at this version.
    Version(u64),
}

/// The result of a `put`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Written {
    pub version: u64,
    /// Whether the key did not exist before.
    pub created: bool,
}

/// One line of a listing.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Summary {
    pub key: String,
    pub version: u64,
    pub size: usize,
}

#[derive(Debug, Error)]
pub enum StoreError {
    #[error("the key does not exist")]
    NotFound,
    #[error("the condition on the key's version does not hold")]
    ConditionFailed { current: Option<u64> },
    #[error("the log could not be written: {0}")]
    Log(#[from] io::Error),
}

pub struct Store {
    entries: BTreeMap<String, Entry>,
    log: Log,
}

impl Store {
    /// Opens the store in `directory` and applies every record of its log.
    /// It returns the store, the number of records applied, and the number
    /// of bytes that were cut from the end of the log.
    pub fn open(directory: &Path, sync: bool) -> io::Result<(Store, usize, u64)> {
        let (log, recovery) = Log::open(directory, sync)?;
        let applied = recovery.records.len();
        let mut entries = BTreeMap::new();

        for record in recovery.records {
            apply(&mut entries, record);
        }

        Ok((Store { entries, log }, applied, recovery.discarded))
    }
}

/// Makes the change of one record in the map. Recovery and the write path
/// both go through this function, so they cannot disagree.
pub fn apply(entries: &mut BTreeMap<String, Entry>, record: Record) {
    match record.change {
        Change::Put { key, value } => {
            entries.insert(
                key,
                Entry {
                    value,
                    version: record.sequence,
                },
            );
        }
        Change::Delete { key } => {
            entries.remove(&key);
        }
    }
}

impl Store {
    /// Sets the value of a key. The change is in the log before it is in
    /// the map.
    pub fn put(
        &mut self,
        key: String,
        value: Vec<u8>,
        condition: Condition,
    ) -> Result<Written, StoreError> {
        let current = self.version(&key);
        check(condition, current)?;

        let record = self
            .log
            .append(Change::Put { key, value })?;
        let version = record.sequence;
        apply(&mut self.entries, record);

        Ok(Written {
            version,
            created: current.is_none(),
        })
    }

    /// Removes a key.
    pub fn delete(&mut self, key: &str, condition: Condition) -> Result<(), StoreError> {
        let current = self.version(key);

        if current.is_none() {
            return Err(StoreError::NotFound);
        }

        check(condition, current)?;

        let record = self.log.append(Change::Delete {
            key: key.to_owned(),
        })?;
        apply(&mut self.entries, record);

        Ok(())
    }

    fn version(&self, key: &str) -> Option<u64> {
        self.entries
            .get(key)
            .map(|entry| entry.version)
    }
}

pub fn check(condition: Condition, current: Option<u64>) -> Result<(), StoreError> {
    let holds = match condition {
        Condition::Always => true,
        Condition::Absent => current.is_none(),
        Condition::Version(version) => current == Some(version),
    };

    holds
        .then_some(())
        .ok_or(StoreError::ConditionFailed { current })
}

/// One write of a batch: the change, and the condition it carries.
pub struct Write {
    pub change: Change,
    pub condition: Condition,
}

impl Store {
    /// Performs several writes with a single wait for the disk. Each write
    /// is checked as if the writes before it in the batch had already
    /// happened. The result of each is returned in the same order.
    pub fn commit(&mut self, writes: Vec<Write>) -> Vec<Result<Written, StoreError>> {
        // The versions that keys will have once the accepted writes of this
        // batch are applied. `None` is a key that a write of the batch deletes.
        let mut pending: BTreeMap<String, Option<u64>> = BTreeMap::new();
        let mut records = Vec::new();
        let mut results = Vec::new();
        let first = self.log.last_sequence() + 1;

        for write in writes {
            let key = match &write.change {
                Change::Put { key, .. } | Change::Delete { key } => key.clone(),
            };
            let current = match pending.get(&key) {
                Some(version) => *version,
                None => self.version(&key),
            };
            let deletes = matches!(write.change, Change::Delete { .. });

            let checked = if deletes && current.is_none() {
                Err(StoreError::NotFound)
            } else {
                check(write.condition, current)
            };

            if let Err(error) = checked {
                results.push(Err(error));

                continue;
            }

            let sequence = first + records.len() as u64;
            let version = if deletes { None } else { Some(sequence) };

            pending.insert(key, version);
            results.push(Ok(Written {
                version: sequence,
                created: current.is_none(),
            }));
            records.push(Record {
                sequence,
                change: write.change,
            });
        }

        // One write and one wait for the disk, for the whole batch.
        if let Err(error) = self.log.append_records(&records) {
            let failure = error.to_string();

            return results
                .into_iter()
                .map(|result| result.and(Err(StoreError::Log(io::Error::other(failure.clone())))))
                .collect();
        }

        for record in records {
            apply(&mut self.entries, record);
        }

        results
    }
}

impl Store {
    pub fn get(&self, key: &str) -> Option<&Entry> {
        self.entries.get(key)
    }

    /// Returns up to `limit` keys that start with `prefix` and sort after
    /// `after`, in order, and whether more keys follow them.
    pub fn list(&self, prefix: &str, after: Option<&str>, limit: usize) -> (Vec<Summary>, bool) {
        page(&self.entries, prefix, after, limit)
    }

    pub fn len(&self) -> usize {
        self.entries.len()
    }

    pub fn is_empty(&self) -> bool {
        self.entries.is_empty()
    }

    /// The sequence number of the last record in the log.
    pub fn last_sequence(&self) -> u64 {
        self.log.last_sequence()
    }
}

/// One page of a listing over a map of entries.
pub fn page(
    entries: &BTreeMap<String, Entry>,
    prefix: &str,
    after: Option<&str>,
    limit: usize,
) -> (Vec<Summary>, bool) {
    let start = match after {
        Some(key) => Bound::Excluded(key),
        None => Bound::Included(prefix),
    };

    let mut matching = entries
        .range::<str, _>((start, Bound::Unbounded))
        .take_while(|entry| entry.0.starts_with(prefix))
        .map(|(key, entry)| Summary {
            key: key.clone(),
            version: entry.version,
            size: entry.value.len(),
        });

    let page: Vec<Summary> = matching
        .by_ref()
        .take(limit)
        .collect();
    let more = matching.next().is_some();

    (page, more)
}

Listing 2.40 The complete kvd/src/lib.rs. kvd/src/lib.rs

//! The HTTP API of a storage node.

pub mod api;

Listing 2.41 The complete kvd/src/api.rs. kvd/src/api.rs

//! The parts of the HTTP API that do not depend on how a node stores its
//! data: the errors and their responses, and the reading of keys and
//! conditional headers.

use axum::Json;
use axum::http::header::{CONTENT_TYPE, IF_MATCH, IF_NONE_MATCH};
use axum::http::{HeaderMap, HeaderName, StatusCode};
use axum::response::{IntoResponse, Response};
use kv::{Condition, StoreError};
use serde::Serialize;

/// The largest value a key may hold.
pub const MAX_VALUE: usize = 1 << 20;

pub const MAX_KEY: usize = 256;

/// The ways a request can fail. Each one is a status code and a sentence.
pub enum ApiError {
    InvalidKey,
    InvalidHeader(HeaderName),
    NotFound,
    ConditionFailed(Option<u64>),
    Unavailable(String),
    /// The request reached a node that does not serve it. The text names the reason.
    Misdirected(String),
    /// The shard has no leader at the moment. Nothing was changed.
    NoLeader,
}

/// The body of an error response, in the format of RFC 9457.
#[derive(Serialize)]
struct Problem {
    #[serde(rename = "type")]
    kind: &'static str,
    title: &'static str,
    status: u16,
    #[serde(skip_serializing_if = "String::is_empty")]
    detail: String,
}

impl IntoResponse for ApiError {
    fn into_response(self) -> Response {
        let (status, kind, title, detail) = match self {
            ApiError::InvalidKey => (
                StatusCode::BAD_REQUEST,
                "/problems/invalid-key",
                "The key is not valid",
                format!("a key is 1 to {MAX_KEY} characters from A-Z, a-z, 0-9, '.', '_' and '-'"),
            ),
            ApiError::InvalidHeader(name) => (
                StatusCode::BAD_REQUEST,
                "/problems/invalid-header",
                "A request header is not valid",
                format!("{name} must be a version in double quotes, or * where that is allowed"),
            ),
            ApiError::NotFound => (
                StatusCode::NOT_FOUND,
                "/problems/key-not-found",
                "The key does not exist",
                String::new(),
            ),
            ApiError::ConditionFailed(current) => (
                StatusCode::PRECONDITION_FAILED,
                "/problems/condition-failed",
                "The condition on the key's version does not hold",
                current.map_or("the key does not exist".into(), |version| {
                    format!("the key is at version {version}")
                }),
            ),
            ApiError::Misdirected(detail) => (
                StatusCode::MISDIRECTED_REQUEST,
                "/problems/not-leader",
                "This node is not the leader",
                detail,
            ),
            ApiError::NoLeader => (
                StatusCode::SERVICE_UNAVAILABLE,
                "/problems/no-leader",
                "The shard has no leader at the moment",
                String::new(),
            ),
            ApiError::Unavailable(detail) => (
                StatusCode::SERVICE_UNAVAILABLE,
                "/problems/store-unavailable",
                "The node cannot serve the request",
                detail,
            ),
        };

        let problem = Problem {
            kind,
            title,
            status: status.as_u16(),
            detail,
        };

        (
            status,
            [(CONTENT_TYPE, "application/problem+json")],
            Json(problem),
        )
            .into_response()
    }
}

impl From<StoreError> for ApiError {
    fn from(error: StoreError) -> Self {
        match error {
            StoreError::NotFound => ApiError::NotFound,
            StoreError::ConditionFailed { current } => ApiError::ConditionFailed(current),
            StoreError::Log(error) => ApiError::Unavailable(error.to_string()),
        }
    }
}

/// A version as it appears in `ETag`, `If-Match` and `If-None-Match`.
pub fn etag(version: u64) -> String {
    format!("\"{version}\"")
}

pub fn valid_key(key: &str) -> Result<(), ApiError> {
    let allowed = |c: char| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-');

    let valid = (1..=MAX_KEY).contains(&key.len()) && key.chars().all(allowed);

    valid
        .then_some(())
        .ok_or(ApiError::InvalidKey)
}

/// Reads a header that holds a quoted version or `*`.
pub fn version_header(
    headers: &HeaderMap,
    name: HeaderName,
) -> Result<Option<Condition>, ApiError> {
    let Some(value) = headers.get(&name) else {
        return Ok(None);
    };

    let Ok(text) = value.to_str() else {
        return Err(ApiError::InvalidHeader(name));
    };

    if text == "*" {
        return Ok(Some(Condition::Absent));
    }

    let version = text
        .strip_prefix("\"")
        .and_then(|rest| rest.strip_suffix("\""))
        .and_then(|digits| digits.parse().ok());

    version
        .map(Condition::Version)
        .map(Some)
        .ok_or(ApiError::InvalidHeader(name))
}

/// The condition a write carries: `If-Match: "7"` requires version 7, and
/// `If-None-Match: *` requires that the key does not exist.
pub fn write_condition(headers: &HeaderMap) -> Result<Condition, ApiError> {
    match (
        version_header(headers, IF_MATCH)?,
        version_header(headers, IF_NONE_MATCH)?,
    ) {
        (Some(Condition::Version(version)), None) => Ok(Condition::Version(version)),
        (None, Some(Condition::Absent)) => Ok(Condition::Absent),
        (None, None) => Ok(Condition::Always),
        (Some(..), ..) => Err(ApiError::InvalidHeader(IF_MATCH)),
        (None, Some(..)) => Err(ApiError::InvalidHeader(IF_NONE_MATCH)),
    }
}

Listing 2.42 The complete kvd/src/main.rs. kvd/src/main.rs

//! Command kvd serves a key-value store over HTTP. Every write is in the log
//! on disk before the response is sent.
//!
//! Run it from code/ with:
//!
//!     cargo run -p kvd -- --listen 0.0.0.0:7001 --data /var/tmp/kvd

use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;

use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, Query, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_NONE_MATCH};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use kv::{Condition, Store, StoreError, Summary};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, version_header, write_condition};
use parking_lot::Mutex;
use serde::{Deserialize, Serialize};
use tokio::net::TcpListener;
use tokio::task;

#[derive(Parser)]
struct Options {
    /// The address to accept connections on.
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    /// The directory that holds the log.
    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// Respond to a write without waiting for the disk.
    #[arg(long)]
    no_sync: bool,
}

/// The store, shared by every request.
type Shared = Arc<Mutex<Store>>;


/// Runs a write on a thread that may block. A write waits for the disk, and
/// a thread that waits must not be one of the threads that serve requests.
async fn write<T, F>(store: Shared, work: F) -> Result<T, ApiError>
where
    F: FnOnce(&mut Store) -> Result<T, StoreError> + Send + 'static,
    T: Send + 'static,
{
    let result = task::spawn_blocking(move || work(&mut store.lock()))
        .await
        .map_err(|error| ApiError::Unavailable(error.to_string()))?;

    Ok(result?)
}

async fn put_key(
    State(store): State<Shared>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    let written = write(store, move |store| store.put(key, body.to_vec(), condition)).await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(store): State<Shared>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    write(store, move |store| store.delete(&key, condition)).await?;

    Ok(StatusCode::NO_CONTENT)
}

async fn get_key(
    State(store): State<Shared>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let cached = version_header(&headers, IF_NONE_MATCH)?;
    let entry = store
        .lock()
        .get(&key)
        .cloned()
        .ok_or(ApiError::NotFound)?;
    let tag = [(ETAG, etag(entry.version))];

    if cached == Some(Condition::Version(entry.version)) {
        return Ok((StatusCode::NOT_MODIFIED, tag).into_response());
    }

    Ok((
        tag,
        [(CONTENT_TYPE, "application/octet-stream")],
        entry.value,
    )
        .into_response())
}

#[derive(Deserialize)]
struct ListQuery {
    #[serde(default)]
    prefix: String,
    after: Option<String>,
    limit: Option<usize>,
}

#[derive(Serialize)]
struct Item {
    key: String,
    version: u64,
    size: usize,
}

#[derive(Serialize)]
struct Page {
    keys: Vec<Item>,
    /// The value of `after` that returns the next page, when there is one.
    next_after: Option<String>,
}

async fn list_keys(State(store): State<Shared>, Query(query): Query<ListQuery>) -> Json<Page> {
    let limit = query
        .limit
        .unwrap_or(100)
        .clamp(1, 1000);
    let (page, more) = store
        .lock()
        .list(&query.prefix, query.after.as_deref(), limit);
    let next_after = page
        .last()
        .filter(|_last| more)
        .map(|last| last.key.clone());
    let keys = page
        .into_iter()
        .map(|Summary { key, version, size }| Item { key, version, size })
        .collect();

    Json(Page { keys, next_after })
}

#[derive(Serialize)]
struct Health {
    status: &'static str,
    keys: usize,
    last_sequence: u64,
}

async fn health(State(store): State<Shared>) -> Json<Health> {
    let store = store.lock();

    Json(Health {
        status: "ok",
        keys: store.len(),
        last_sequence: store.last_sequence(),
    })
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let (store, applied, discarded) = Store::open(&options.data, !options.no_sync)?;

    println!(
        "recovered {} keys from {applied} records, cut {discarded} bytes from the end of the log",
        store.len()
    );

    let routes = Router::new()
        .route("/v1/keys", get(list_keys))
        .route(
            "/v1/keys/{key}",
            get(get_key)
                .put(put_key)
                .delete(delete_key),
        )
        .route("/healthz", get(health))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(Arc::new(Mutex::new(store)));

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Listing 2.43 The complete kvctl/src/main.rs. kvctl/src/main.rs

//! Command kvctl tests a kvd server from another process or another machine.
//! `load` writes keys and records each write the server acknowledged, and
//! `verify` reads every recorded key back.
//!
//! Run it from code/ with:
//!
//!     cargo run -p kvctl -- load --server http://127.0.0.1:7001 --count 1000 --acked acked.txt
//!     cargo run -p kvctl -- verify --server http://127.0.0.1:7001 --acked acked.txt

use std::error::Error;
use std::fs::{self, File};
use std::io::{BufWriter, Write};
use std::path::PathBuf;

use clap::{Parser, Subcommand};
use ureq::Agent;

#[derive(Parser)]
struct Options {
    #[command(subcommand)]
    command: Command,
}

#[derive(Subcommand)]
enum Command {
    /// Write keys until the count is reached or a write fails.
    Load {
        #[arg(long)]
        server: String,
        #[arg(long, default_value_t = 1000)]
        count: u32,
        /// The file that receives the key of every acknowledged write.
        #[arg(long)]
        acked: PathBuf,
    },
    /// Read back every key in the file of acknowledged writes.
    Verify {
        #[arg(long)]
        server: String,
        #[arg(long)]
        acked: PathBuf,
    },
}

/// Writes `count` keys. A key is recorded in `acked` only after the server
/// has answered with a success status.
fn load(agent: &Agent, server: &str, count: u32, acked: &PathBuf) -> Result<(), Box<dyn Error>> {
    let mut file = BufWriter::new(File::create(acked)?);
    let mut written = 0;

    for number in 1..=count {
        let key = format!("key-{number:06}");

        if let Err(error) = agent
            .put(format!("{server}/v1/keys/{key}"))
            .send(format!("value of {key}"))
        {
            println!("write {number} failed: {error}");

            break;
        }

        writeln!(file, "{key}")?;
        written += 1;
    }

    file.flush()?;
    println!("{written} writes acknowledged");

    Ok(())
}

/// Reads every acknowledged key and compares the value with the one that
/// was written.
fn verify(agent: &Agent, server: &str, acked: &PathBuf) -> Result<(), Box<dyn Error>> {
    let keys = fs::read_to_string(acked)?;
    let (mut found, mut missing) = (0, 0);

    for key in keys.lines() {
        let value = match agent
            .get(format!("{server}/v1/keys/{key}"))
            .call()
        {
            Ok(mut response) => response
                .body_mut()
                .read_to_string()?,
            Err(ureq::Error::StatusCode(404)) => String::new(),
            Err(error) => return Err(error.into()),
        };

        if value == format!("value of {key}") {
            found += 1;
        } else {
            missing += 1;
        }
    }

    println!(
        "{} acknowledged, {found} found, {missing} missing",
        found + missing
    );

    Ok(())
}

fn main() -> Result<(), Box<dyn Error>> {
    let agent = Agent::new_with_defaults();

    match Options::parse().command {
        Command::Load {
            server,
            count,
            acked,
        } => load(&agent, &server, count, &acked),
        Command::Verify { server, acked } => verify(&agent, &server, &acked),
    }
}

Listing 2.44 The complete crashsim/src/main.rs. crashsim/src/main.rs

//! Command crashsim checks what the log recovers after a crash in the middle
//! of a write. It writes a log of 5 records, then damages a copy of it in
//! every way a crash or a bad disk can: it cuts the copy at every length,
//! and it flips 1 bit in every byte. After each, it opens the copy and
//! compares the records that come back with the records that were complete.
//!
//! Run it from code/ with:
//!
//!     cargo run -p crashsim

use std::env;
use std::error::Error;
use std::fs;
use std::path::Path;
use std::process;

use kv::log::FILE_NAME;
use kv::{Change, Log, Record};

/// The changes of the test log.
fn changes() -> Vec<Change> {
    let put = |key: &str, value: &str| Change::Put {
        key: key.into(),
        value: value.into(),
    };

    vec![
        put("user-1", "ada"),
        put("user-2", "grace"),
        put("user-1", "ada lovelace"),
        Change::Delete {
            key: "user-2".into(),
        },
        put("user-3", "barbara"),
    ]
}

/// Writes the changes to a log in `directory`. It returns the bytes of the
/// log and the offset at which each record ends.
fn build(directory: &Path) -> Result<(Vec<u8>, Vec<usize>), Box<dyn Error>> {
    let mut log = Log::open(directory, false)?.0;
    let mut ends = Vec::new();

    for change in changes() {
        let record = log.append(change)?;
        ends.push(ends.last().copied().unwrap_or(0) + record.encode().len());
    }

    Ok((fs::read(directory.join(FILE_NAME))?, ends))
}

/// Replaces the log in `directory` with `bytes` and opens it.
fn recover(directory: &Path, bytes: &[u8]) -> Result<Vec<Record>, Box<dyn Error>> {
    fs::write(directory.join(FILE_NAME), bytes)?;
    let recovery = Log::open(directory, false)?.1;

    Ok(recovery.records)
}

/// Whether `records` are exactly the first `count` changes, in order.
fn is_prefix(records: &[Record], count: usize) -> bool {
    let expected = changes();

    records.len() == count
        && records
            .iter()
            .zip(&expected)
            .enumerate()
            .all(|(index, (record, change))| {
                record.sequence == index as u64 + 1 && &record.change == change
            })
}

/// Cuts the log at every length and checks that recovery returns the records
/// that end at or before the cut.
fn cuts(directory: &Path, log: &[u8], ends: &[usize]) -> Result<(), Box<dyn Error>> {
    for length in 0..=log.len() {
        let complete = ends
            .iter()
            .filter(|&&end| end <= length)
            .count();
        let records = recover(directory, &log[..length])?;

        if !is_prefix(&records, complete) {
            return Err(format!(
                "cut at {length}: expected {complete} records, recovered {}",
                records.len()
            )
            .into());
        }
    }

    println!(
        "cut at every length from 0 to {}: each recovery returned the complete records",
        log.len()
    );

    let mut first = 0;

    for (count, &end) in ends.iter().enumerate() {
        println!("  lengths {first:3} to {:3}: {count} records", end - 1);
        first = end;
    }

    println!("  length  {first:3}       : {} records", ends.len());

    Ok(())
}

/// Flips 1 bit in each byte in turn and checks that recovery returns the
/// records before the damaged one.
fn flips(directory: &Path, log: &[u8], ends: &[usize]) -> Result<(), Box<dyn Error>> {
    for position in 0..log.len() {
        let mut damaged = log.to_vec();
        damaged[position] ^= 0x10;

        let before = ends
            .iter()
            .filter(|&&end| end <= position)
            .count();
        let records = recover(directory, &damaged)?;

        if !is_prefix(&records, before) {
            return Err(format!(
                "flip at {position}: expected {before} records, recovered {}",
                records.len()
            )
            .into());
        }
    }

    println!(
        "1 bit flipped in each of {} bytes: each recovery returned the records before the damage",
        log.len()
    );

    Ok(())
}

fn run(directory: &Path) -> Result<(), Box<dyn Error>> {
    let (log, ends) = build(directory)?;
    let boundaries: Vec<String> = ends
        .iter()
        .map(usize::to_string)
        .collect();

    println!(
        "a log of {} records and {} bytes, with records ending at {}",
        ends.len(),
        log.len(),
        boundaries.join(", ")
    );

    cuts(directory, &log, &ends)?;
    flips(directory, &log, &ends)?;

    let records = recover(directory, &log[..ends[1] + 5])?;
    let mut reopened = Log::open(directory, false)?.0;
    let next = reopened.append(Change::Delete {
        key: "user-1".into(),
    })?;

    println!(
        "after a cut at {}: {} records recovered, and the next record has sequence {}",
        ends[1] + 5,
        records.len(),
        next.sequence
    );

    Ok(())
}

fn main() -> Result<(), Box<dyn Error>> {
    let directory = env::temp_dir().join(format!("crashsim-{}", process::id()));
    let result = run(&directory);

    fs::remove_dir_all(&directory)?;

    result
}

Listing 2.45 The complete synccost/src/main.rs. synccost/src/main.rs

//! Command synccost measures what it costs to wait for the disk. It appends
//! the same records to 2 logs, one that calls fsync after each record and
//! one that does not, and prints the time per record.
//!
//! Run it from code/ with:
//!
//!     cargo run --release -p synccost -- /var/tmp/synccost

use std::env;
use std::error::Error;
use std::fs;
use std::path::Path;
use std::time::{Duration, Instant};

use kv::{Change, Log};

const RECORDS: u32 = 500;

/// Appends `RECORDS` records of 100 bytes and returns the time it took.
fn measure(directory: &Path, sync: bool) -> Result<Duration, Box<dyn Error>> {
    let mut log = Log::open(directory, sync)?.0;
    let started = Instant::now();

    for number in 0..RECORDS {
        log.append(Change::Put {
            key: format!("key-{number:06}"),
            value: vec![b'v'; 100],
        })?;
    }

    Ok(started.elapsed())
}

fn main() -> Result<(), Box<dyn Error>> {
    let root = env::args()
        .nth(1)
        .ok_or("usage: synccost <directory>")?;

    for (name, sync) in [("without fsync", false), ("with fsync", true)] {
        let directory = Path::new(&root).join(name.replace(' ', "-"));
        let elapsed = measure(&directory, sync)?;
        let each = elapsed.as_secs_f64() * 1000.0 / f64::from(RECORDS);

        println!(
            "{name:13}: {RECORDS} records in {:8.1} ms, {each:.3} ms per record, {:6.0} records per second",
            elapsed.as_secs_f64() * 1000.0,
            1000.0 / each
        );

        fs::remove_dir_all(&directory)?;
    }

    Ok(())
}

Further reading

• T. Pillai and others, “All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications”, OSDI 2014.
• A. Rebello and others, “Can Applications Recover from fsync Failures?”, USENIX ATC 2020.
• C. Mohan and others, “ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging”, ACM Transactions on Database Systems, 1992.
• A. Petrov, Database Internals, O’Reilly, 2019.

Key-Value Store: Replication (Low-Level Design)

The storage node of chapter 2 keeps every acknowledged write when its process crashes. Against the loss of the machine itself, or of its disk, it offers nothing: a log on 1 disk is gone when that disk is gone. Non-functional requirement 0 of chapter 1 demands that an acknowledged write survive the loss of any 1 machine, and it follows that the record of every write has to be stored on at least 2 machines before the client is answered.

To copy a file from 1 machine to another is easy. What makes replication a subject is that the copying takes place over a network which loses messages, delays them and delivers their answers late or not at all, between machines that stop and restart at arbitrary moments, and that through all of this the copies must remain identical, in content and in order, record for record. A follower that has missed a message must not apply the next one as though nothing were missing. A leader that has lost an answer must not conclude that the follower lacks the record, nor that it has it. And a machine that was the leader an hour ago and has since been replaced must be prevented from writing as though it still were.

This chapter implements the replication that chapter 1 specified: a leader that orders the writes of a shard, and 2 followers that store the same records in the same order. It turns out that a single small rule on the follower’s side handles every one of the situations listed above. We define the messages and the state of a replica, derive the rules for followers and leaders, and implement them as a library that performs no network operations whatever, so that a simulation can subject it to every possible combination of lost messages. We then build kvnode, a storage node that exchanges the messages over gRPC, and run 3 nodes while stopping followers, removing the majority, and replacing a lost leader.

Overview

Every shard has 3 replicas. All writes go to the leader, and the other 2 replicas are followers. Figure 3.1 shows the nodes of this chapter and the route of a write among them.

The implementation maintains 4 invariants.

1. The log of every follower is a prefix of its leader’s log: the same records in the same order, and possibly fewer of them.
2. A record is acknowledged to its client only after at least 2 of the 3 logs hold it.
3. A replica applies a record to its entries only after the record is known to be held in at least 2 of the 3 logs.
4. A replica rejects every message from a leader whose term is lower than its own.

The second invariant is non-functional requirement 0 in the form of a rule. The third ensures that a read returns only data which would survive the loss of a machine. The fourth prevents a former leader from modifying a shard after it has been replaced.

In this chapter the leader of a shard is chosen by an operator, who starts 1 node in that role. Automatic election, as chapter 1 described it, is built upon the mechanism of this chapter and is the subject of chapter 4.

The messages

The protocol of chapter 1 defines a single call, Append, which a leader sends to each of its followers.

Listing 3.1 The replication service in the schema. proto/store.proto

// Sent by the leader of a shard to each follower.
service Replication {
  // Appends records to the follower's log. A call with no records is a
  // heartbeat: it tells the follower that the leader is alive.
  rpc Append(AppendRequest) returns (AppendResponse);
}

message AppendRequest {
  // The shard, named by the last position it owns.
  fixed64 shard = 1;
  // The term of the sender. A follower rejects a leader of an older term.
  uint64 term = 2;
  // The sequence of the record that precedes `records` in the leader's log.
  uint64 previous_sequence = 3;
  repeated Record records = 4;
  // Every record up to this sequence is on a majority of the replicas.
  uint64 committed_sequence = 5;
  // Added for leader election. The term of the record at previous_sequence,
  // and the id of the sender.
  uint64 previous_term = 6;
  uint32 leader = 7;
}

message AppendResponse {
  uint64 term = 1;
  // False when the follower's log does not end at `previous_sequence`.
  bool appended = 2;
  // The sequence of the last record in the follower's log.
  uint64 last_sequence = 3;
}

The library uses types of its own for these messages, so that it has no dependence on gRPC, and the conversion between the 2 representations is performed in the node.

Listing 3.2 The imports of repl/src/lib.rs. repl/src/lib.rs

use std::collections::BTreeMap;
use std::io;
use std::path::Path;

use kv::{Change, Condition, Entry, Log, Record, StoreError, Summary, apply, check, page};

Listing 3.3 The message and its answer, as the library uses them. repl/src/lib.rs

/// The message a leader sends to a follower.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Append {
    /// The term of the sender.
    pub term: u64,
    /// The sequence of the record that precedes `records` in the leader's log.
    pub previous_sequence: u64,
    pub records: Vec<Record>,
    /// Every record up to this sequence is on a majority of the replicas.
    pub committed_sequence: u64,
}

/// The follower's answer.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Appended {
    pub term: u64,
    /// False when the follower's log does not end at `previous_sequence`,
    /// or when the sender's term is older than the follower's.
    pub appended: bool,
    /// The sequence of the last record in the follower's log.
    pub last_sequence: u64,
}

The last field is the one on which the whole protocol turns. A follower never merely says yes or no. In every answer it reports where its log now ends, and this single number is enough for a leader to correct any mistaken belief it may hold about that follower.

The state of a replica

A replica consists of a log, of the entries that result from applying a part of that log, and of 3 numbers: the sequence of its last record, the point up to which records have been applied, and its term.

A record is committed when a majority of the replicas has stored it. A committed record survives the loss of any 1 machine, since any 2 of the 3 replicas include at least 1 that holds it. A replica keeps uncommitted records in its log without applying them. The entries therefore reflect committed records only, and no read can return a value that might yet be lost.

Figure 3.3 shows where each part of this state resides and which operation carries data from 1 part to another.

In the code, records is a vector that holds record n at index n - 1, so that its length is the sequence of its last record. commit proceeds from committed to the sequence it is given and passes each record to apply, the function of chapter 2 that inserts or removes an entry. Since commit is the only caller of apply, there is no path by which an uncommitted record can reach the entries.

Listing 3.4 The replica, and the application of committed records. repl/src/lib.rs

pub struct Replica {
    log: Log,
    /// Every record of the log, in order. Record `n` is at index `n - 1`.
    records: Vec<Record>,
    /// The result of applying the records up to `committed`.
    entries: BTreeMap<String, Entry>,
    /// The sequence of the last record that is applied to `entries`.
    committed: u64,
    term: u64,
}

impl Replica {
    /// Opens the replica in `directory`. Nothing is applied yet: a replica
    /// that has restarted does not know which of its records are committed.
    pub fn open(directory: &Path, sync: bool, term: u64) -> io::Result<Replica> {
        let (log, recovery) = Log::open(directory, sync)?;

        Ok(Replica {
            log,
            records: recovery.records,
            entries: BTreeMap::new(),
            committed: 0,
            term,
        })
    }

    pub fn term(&self) -> u64 {
        self.term
    }

    pub fn last_sequence(&self) -> u64 {
        self.records.len() as u64
    }

    pub fn committed(&self) -> u64 {
        self.committed
    }

    /// One page of the committed keys.
    pub fn list(&self, prefix: &str, after: Option<&str>, limit: usize) -> (Vec<Summary>, bool) {
        page(&self.entries, prefix, after, limit)
    }

    /// Every record of the log, in order.
    pub fn records(&self) -> &[Record] {
        &self.records
    }

    /// The committed value of a key.
    pub fn get(&self, key: &str) -> Option<&Entry> {
        self.entries.get(key)
    }

    /// Applies the records up to `sequence` that are not applied yet.
    pub fn commit(&mut self, sequence: u64) {
        let until = sequence.min(self.last_sequence());

        while self.committed < until {
            apply(
                &mut self.entries,
                self.records[self.committed as usize].clone(),
            );
            self.committed += 1;
        }
    }
}

Replica::open sets committed to 0 and leaves the entries empty, although the log may contain records. This may seem wasteful, and it is in fact necessary. After a restart, a replica cannot tell from its own log which of its records had reached a majority, since its last few records may exist on its own disk and nowhere else. A follower learns the committed sequence from its leader, and a leader learns it from the answers of its followers.

The follower

A follower applies the same procedure to every message it receives.

handle_append is a direct transcription of that figure. Each check that fails returns by way of a closure named refuse, which builds an answer from the follower’s present term and the end of its log. When both checks pass, append_records writes the whole batch with a single wait for the disk, and commit then applies whatever the leader has declared committed. Since commit stops at the end of the log, a committed sequence that lies beyond what this follower holds does no harm.

Listing 3.5 The follower's handling of a message. repl/src/lib.rs

impl Replica {
    /// Processes a message from a leader.
    pub fn handle_append(&mut self, request: Append) -> io::Result<Appended> {
        let refuse = |replica: &Replica| Appended {
            term: replica.term,
            appended: false,
            last_sequence: replica.last_sequence(),
        };

        if request.term < self.term {
            return Ok(refuse(self));
        }

        self.term = request.term;

        if request.previous_sequence != self.last_sequence() {
            return Ok(refuse(self));
        }

        if !request.records.is_empty() {
            self.log
                .append_records(&request.records)?;
            self.records.extend(request.records);
        }

        self.commit(request.committed_sequence);

        Ok(Appended {
            term: self.term,
            appended: true,
            last_sequence: self.last_sequence(),
        })
    }
}

The second row of the table is what maintains invariant 1, and it deserves to be looked at closely, since the whole reliability of replication derives from it. A follower appends records only at the exact position where the leader’s message begins. Its log can therefore never acquire a gap, a duplicate or a record out of place, whatever has happened to earlier messages. When the positions differ, the follower attempts no repair of any kind. It reports where its log ends, and the leader sends a message that begins there.

This 1 exchange covers every circumstance in which a follower is behind or the leader’s information is out of date: a follower that was stopped and has restarted, a message that was lost, an answer that was lost, and a new follower with an empty log. None of these cases has code of its own. They are all the same case, in which the leader’s belief about the end of the follower’s log is wrong and the follower’s answer puts it right.

The records of a message are appended with Log::append_records, which writes all of them and waits for the disk once.

Listing 3.6 The imports of kv/src/log.rs. kv/src/log.rs

use std::fs::{self, File, OpenOptions};
use std::io::{self, Read, Write};
use std::path::Path;

use crate::record::{Change, Decoded, Record};

Listing 3.7 Appending a batch of records that another node has numbered. kv/src/log.rs

impl Log {
    /// Appends records that another node numbered, in one write and with one
    /// wait for the disk. The first record must follow the last record of
    /// this log.
    pub fn append_records(&mut self, records: &[Record]) -> io::Result<()> {
        if self.failed {
            return Err(io::Error::other(
                "the log failed on an earlier write and is closed",
            ));
        }

        let misplaced = (self.next..)
            .zip(records)
            .find(|(expected, record)| record.sequence != *expected);

        if let Some((expected, record)) = misplaced {
            return Err(io::Error::other(format!(
                "record {} cannot follow record {}",
                record.sequence,
                expected - 1
            )));
        }

        let bytes: Vec<u8> = records
            .iter()
            .flat_map(Record::encode)
            .collect();

        if let Err(error) = self.write(&bytes) {
            self.failed = true;

            return Err(error);
        }

        self.next += records.len() as u64;

        Ok(())
    }
}

The leader

A leader has 3 responsibilities: to accept writes, to send each follower whatever records it lacks, and to determine which records are committed.

Proposing a record

propose checks the condition of a write against the entries and appends the record to the leader’s own log. The record is not yet applied, since at this point only 1 replica holds it.

A leader accepts a new write only when is_settled is true, which is to say when every record in its log is committed. The reason lies in conditional writes. A condition is checked against the entries, and the entries do not include uncommitted records. If the leader were to accept a second write while an earlier record was still uncommitted, the condition of the second would be evaluated against a state that lacked its predecessor, and a conditional write could succeed when it ought to have failed. Requiring a settled log guarantees that every condition is evaluated against all earlier writes. The price is that the writes of a shard are processed strictly in sequence. Production systems evaluate conditions against a speculative state and process writes in batches, which is considerably more intricate and is not attempted here, although chapter 6 recovers much of the lost throughput by another means.

Within propose, the present version of the key is read from the entries, a Delete of a key that does not exist is rejected, and check evaluates the condition. Only then is the record appended. On return, the record is in the leader’s log and in records, and the entries are unchanged.

Listing 3.8 Checking the condition of a write and appending its record. repl/src/lib.rs

impl Replica {
    /// Whether every record of the log is committed. A leader accepts a new
    /// write only in this state, so that the condition of the write is
    /// checked against the result of all earlier writes.
    pub fn is_settled(&self) -> bool {
        self.committed == self.last_sequence()
    }

    /// Checks the condition of a write and appends its record to the
    /// leader's log. The record is not applied: it is not committed yet.
    pub fn propose(&mut self, change: Change, condition: Condition) -> Result<u64, StoreError> {
        let key = match &change {
            Change::Put { key, .. } | Change::Delete { key } => key,
        };
        let current = self
            .entries
            .get(key)
            .map(|entry| entry.version);

        if matches!(change, Change::Delete { .. }) && current.is_none() {
            return Err(StoreError::NotFound);
        }

        check(condition, current)?;

        let record = self.log.append(change)?;
        let sequence = record.sequence;
        self.records.push(record);

        Ok(sequence)
    }
}

Building a message for a follower

For every follower, the leader remembers the last sequence that the follower reported. append_for takes that number and returns a message containing every later record. For a follower that is up to date, the slice of records is empty and the message is a heartbeat. A reported sequence that lies beyond the end of the leader’s own log is limited to that end, a situation that can arise when a leader has itself been replaced and restarted with a shorter log.

Listing 3.9 The message that brings 1 follower up to date. repl/src/lib.rs

impl Replica {
    /// The message that brings a follower whose log ends at `follower_last`
    /// up to the end of this log. With nothing to send it is a heartbeat.
    pub fn append_for(&self, follower_last: u64) -> Append {
        let from = follower_last.min(self.last_sequence()) as usize;

        Append {
            term: self.term,
            previous_sequence: from as u64,
            records: self.records[from..].to_vec(),
            committed_sequence: self.committed,
        }
    }
}

The commit rule

The leader knows where its own log ends and what last sequence each follower has reported. If these numbers are sorted, the one in the middle is the highest sequence that a majority has stored.

majority_sequence implements the rule for any number of replicas, by taking the element at index n / 2 of the values sorted in descending order. For 3 replicas that is the second value, and for 5 the third.

Listing 3.10 The commit rule. repl/src/lib.rs

/// The highest sequence that is stored on a majority, given the last
/// sequence of every replica, the leader included.
pub fn majority_sequence(mut last_sequences: Vec<u64>) -> u64 {
    last_sequences.sort_unstable_by(|a, b| b.cmp(a));

    last_sequences
        .get(last_sequences.len() / 2)
        .copied()
        .unwrap_or(0)
}

Simulating lost messages

Because the library performs no network operations, a test can carry the messages between replicas itself and decide the fate of every exchange. This is the reason for writing the rules as a library in the first place. Faults in a real network occur rarely and unrepeatably, and a protocol that has only been tested on a real network has scarcely been tested against faults at all.

replsim creates a leader and 2 followers in a single process. For each exchange it chooses 1 of 3 outcomes: the message is delivered and answered, the message is lost, or the message is processed and its answer is lost.

Listing 3.11 The imports of replsim/src/main.rs. replsim/src/main.rs

use std::env;
use std::error::Error;
use std::fs;
use std::path::{Path, PathBuf};
use std::process;

use kv::{Change, Condition};
use repl::{Replica, majority_sequence};

Listing 3.12 A cluster in 1 process, with control over every exchange. replsim/src/main.rs

/// What happens to one exchange between the leader and a follower.
#[derive(Clone, Copy, PartialEq, Eq)]
enum Fate {
    Delivered,
    /// The follower does not receive the message.
    RequestLost,
    /// The follower processes the message, and its answer does not arrive.
    AnswerLost,
}

/// A leader, 2 followers, and what the leader knows about each follower.
struct Cluster {
    leader: Replica,
    followers: Vec<Replica>,
    /// The last sequence each follower has reported.
    reported: Vec<u64>,
}

impl Cluster {
    fn open(root: &Path) -> Result<Cluster, Box<dyn Error>> {
        let open = |name: &str| Replica::open(&root.join(name), false, 1);

        Ok(Cluster {
            leader: open("leader")?,
            followers: vec![open("follower-1")?, open("follower-2")?],
            reported: vec![0, 0],
        })
    }

    /// One exchange with a follower: the leader sends what the follower
    /// lacks, as far as it knows, and commits what a majority now holds.
    fn exchange(&mut self, follower: usize, fate: Fate) -> Result<(), Box<dyn Error>> {
        let message = self
            .leader
            .append_for(self.reported[follower]);

        if fate == Fate::RequestLost {
            return Ok(());
        }

        let answer = self.followers[follower].handle_append(message)?;

        if fate == Fate::AnswerLost {
            return Ok(());
        }

        self.reported[follower] = answer.last_sequence;

        let stored = vec![
            self.leader.last_sequence(),
            self.reported[0],
            self.reported[1],
        ];
        self.leader
            .commit(majority_sequence(stored));

        Ok(())
    }

    /// A write: the leader appends the record and makes one exchange with
    /// each follower. It returns whether the write was acknowledged, or
    /// `None` when the leader could not accept it.
    fn write(
        &mut self,
        key: &str,
        value: &str,
        fates: [Fate; 2],
    ) -> Result<Option<bool>, Box<dyn Error>> {
        if !self.leader.is_settled() {
            return Ok(None);
        }

        let change = Change::Put {
            key: key.into(),
            value: value.into(),
        };
        let sequence = self
            .leader
            .propose(change, Condition::Always)?;

        self.exchange(0, fates[0])?;
        self.exchange(1, fates[1])?;

        Ok(Some(self.leader.committed() >= sequence))
    }

    /// The last sequence of the leader and of each follower.
    fn lasts(&self) -> [u64; 3] {
        [
            self.leader.last_sequence(),
            self.followers[0].last_sequence(),
            self.followers[1].last_sequence(),
        ]
    }
}

The first part of the simulation runs the cases discussed above.

Listing 3.13 The cases of the chapter, run in order. replsim/src/main.rs

const BOTH: [Fate; 2] = [Fate::Delivered, Fate::Delivered];

fn cases(root: &Path) -> Result<(), Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;
    let show = |cluster: &Cluster| {
        format!(
            "logs end at {:?}, committed {}",
            cluster.lasts(),
            cluster.leader.committed()
        )
    };

    for key in ["a", "b", "c"] {
        cluster.write(key, "1", BOTH)?;
    }

    println!("3 writes, every message delivered: {}", show(&cluster));

    for key in ["d", "e", "f"] {
        cluster.write(key, "1", [Fate::RequestLost, Fate::Delivered])?;
    }

    println!(
        "3 writes while follower 1 is unreachable: {}",
        show(&cluster)
    );

    cluster.exchange(0, Fate::Delivered)?;
    println!("follower 1 returns, 1 exchange: {}", show(&cluster));

    for key in ["g", "h", "i"] {
        cluster.write(key, "1", [Fate::AnswerLost, Fate::Delivered])?;
    }

    println!(
        "3 writes while the answers of follower 1 are lost: {}",
        show(&cluster)
    );
    println!(
        "  the leader believes that the log of follower 1 ends at {}",
        cluster.reported[0]
    );

    cluster.exchange(0, Fate::Delivered)?;
    println!(
        "  next exchange: the leader sends records 7 to 9, follower 1 refuses and reports {}",
        cluster.reported[0]
    );

    cluster.exchange(0, Fate::Delivered)?;
    println!("  the exchange after it: {}", show(&cluster));

    let acknowledged =
        cluster.write("j", "1", [Fate::RequestLost, Fate::RequestLost])? == Some(true);
    println!(
        "1 write while both followers are unreachable: acknowledged {acknowledged}, {}",
        show(&cluster)
    );
    println!(
        "  the leader reads j as {:?}",
        cluster
            .leader
            .get("j")
            .map(|entry| entry.version)
    );

    let accepted = cluster
        .write("k", "1", [Fate::RequestLost, Fate::RequestLost])?
        .is_some();
    println!("  the leader accepts another write: {accepted}");

    cluster.exchange(1, Fate::Delivered)?;
    println!("follower 2 returns: {}", show(&cluster));
    println!(
        "  the leader reads j as {:?}",
        cluster
            .leader
            .get("j")
            .map(|entry| entry.version)
    );

    let mut newer = Replica::open(&root.join("newer"), false, 2)?;
    let answer = newer.handle_append(cluster.leader.append_for(0))?;
    println!(
        "a replica at term 2 receives a message of term 1: appended {}, its term {}",
        answer.appended, answer.term
    );

    Ok(())
}

The second part runs 3 writes under every combination of outcomes. Each write involves 2 exchanges, and each exchange is assigned 2 bits of a pattern, which gives 4096 patterns. After each pattern, replsim verifies invariants 1 and 2, then delivers all messages and verifies that the 3 logs become equal.

Listing 3.14 Every combination of lost messages over 3 writes. replsim/src/main.rs

const WRITES: usize = 3;

/// The fate of exchange number `index` under a pattern: 2 bits for each.
fn fate(pattern: u32, index: usize) -> Fate {
    match (pattern >> (2 * index)) & 3 {
        0 | 1 => Fate::Delivered,
        2 => Fate::RequestLost,
        3.. => Fate::AnswerLost,
    }
}

/// Runs 3 writes under one pattern of losses and checks the replicas. It
/// returns the number of writes that were acknowledged.
fn run(root: &Path, pattern: u32) -> Result<usize, Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;
    let mut acknowledged = Vec::new();

    for number in 0..WRITES {
        let fates = [fate(pattern, 2 * number), fate(pattern, 2 * number + 1)];

        if cluster.write(&format!("key-{number}"), "1", fates)? == Some(true) {
            acknowledged.push(cluster.leader.last_sequence());
        }
    }

    for &sequence in &acknowledged {
        let holders = cluster
            .lasts()
            .iter()
            .filter(|&&last| last >= sequence)
            .count();

        if holders < 2 {
            return Err(format!(
                "pattern {pattern}: record {sequence} was acknowledged and is in {holders} log"
            )
            .into());
        }
    }

    for follower in &cluster.followers {
        if !cluster
            .leader
            .records()
            .starts_with(follower.records())
        {
            return Err(format!(
                "pattern {pattern}: a follower's log is not the beginning of the leader's"
            )
            .into());
        }
    }

    for _ in 0..2 {
        cluster.exchange(0, Fate::Delivered)?;
        cluster.exchange(1, Fate::Delivered)?;
    }

    let [leader, first, second] = cluster.lasts();

    if first != leader || second != leader || !cluster.leader.is_settled() {
        return Err(format!(
            "pattern {pattern}: the logs differ after every message was delivered"
        )
        .into());
    }

    Ok(acknowledged.len())
}

fn exhaustive(root: &Path) -> Result<(), Box<dyn Error>> {
    let patterns = 1u32 << (4 * WRITES);
    let mut by_acknowledged = [0; WRITES + 1];

    for pattern in 0..patterns {
        let directory = root.join(format!("pattern-{pattern}"));
        by_acknowledged[run(&directory, pattern)?] += 1;
        fs::remove_dir_all(&directory)?;
    }

    println!("{patterns} patterns of lost messages over {WRITES} writes:");
    println!("  every acknowledged record was in at least 2 of the 3 logs");
    println!("  every follower's log was the beginning of the leader's log");
    println!("  after the losses stopped, the 3 logs were equal and every record was committed");

    for (count, patterns) in by_acknowledged.iter().enumerate() {
        println!("  {count} of {WRITES} writes acknowledged: {patterns} patterns");
    }

    Ok(())
}

Listing 3.15 Running the simulation. replsim/src/main.rs

fn directory() -> PathBuf {
    env::temp_dir().join(format!("replsim-{}", process::id()))
}

fn main() -> Result<(), Box<dyn Error>> {
    let root = directory();
    let result = cases(&root.join("cases")).and_then(|()| exhaustive(&root));

    fs::remove_dir_all(&root)?;

    result
}

$ cargo run --release -p replsim
3 writes, every message delivered: logs end at [3, 3, 3], committed 3
3 writes while follower 1 is unreachable: logs end at [6, 3, 6], committed 6
follower 1 returns, 1 exchange: logs end at [6, 6, 6], committed 6
3 writes while the answers of follower 1 are lost: logs end at [9, 7, 9], committed 9
  the leader believes that the log of follower 1 ends at 6
  next exchange: the leader sends records 7 to 9, follower 1 refuses and reports 7
  the exchange after it: logs end at [9, 9, 9], committed 9
1 write while both followers are unreachable: acknowledged false, logs end at [10, 9, 9], committed 9
  the leader reads j as None
  the leader accepts another write: false
follower 2 returns: logs end at [10, 9, 10], committed 10
  the leader reads j as Some(10)
a replica at term 2 receives a message of term 1: appended false, its term 2
4096 patterns of lost messages over 3 writes:
  every acknowledged record was in at least 2 of the 3 logs
  every follower's log was the beginning of the leader's log
  after the losses stopped, the 3 logs were equal and every record was committed
  0 of 3 writes acknowledged: 1024 patterns
  1 of 3 writes acknowledged: 1024 patterns
  2 of 3 writes acknowledged: 704 patterns
  3 of 3 writes acknowledged: 1344 patterns

The seventh row shows something that every client of a replicated store has to take into account. A write that was answered with an error may nevertheless take effect later. The record was in the leader’s log, and it was committed when a follower became reachable again. For a client, therefore, an error from a leader signifies an unknown outcome, exactly as a lost connection does, and is to be resolved by reading the key or by repeating an idempotent request. An error does not mean that nothing happened.

Design overview

The node

kvnode combines a replica with a gRPC service for replication and the HTTP API of chapter 1.

Listing 3.16 The imports of kvnode/src/main.rs. kvnode/src/main.rs

use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;

use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, Query, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_NONE_MATCH};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::{Parser, ValueEnum};
use kv::{Change, Condition};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, version_header, write_condition};
use repl::Replica;
use serde::{Deserialize, Serialize};
use tokio::net::TcpListener;
use tonic::transport::Server;

use crate::node::Node;
use crate::rpc::{Follower, ReplicationServer};

Listing 3.17 The options of the command line. kvnode/src/main.rs

#[derive(Clone, Copy, PartialEq, Eq, ValueEnum)]
enum Role {
    Leader,
    Follower,
}

#[derive(Parser)]
struct Options {
    #[arg(long, value_enum)]
    role: Role,

    /// The address of the HTTP API.
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    /// The address on which a follower accepts records from the leader.
    #[arg(long, default_value = "127.0.0.1:7101")]
    replication: SocketAddr,

    /// The replication address of a follower, such as http://127.0.0.1:7102.
    /// Given once for each follower, to the leader.
    #[arg(long = "follower")]
    followers: Vec<String>,

    /// The term this node starts in. A new leader is given a higher term
    /// than the leader it replaces.
    #[arg(long, default_value_t = 1)]
    term: u64,

    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// Respond without waiting for the disk.
    #[arg(long)]
    no_sync: bool,
}

The gRPC service

The Rust types, the client code and the server code for Append are generated from the schema when the program is built. Building requires protoc, the compiler for Protocol Buffers, which is available as the package protobuf-compiler on Debian and Ubuntu.

Listing 3.18 The complete kvnode/build.rs. kvnode/build.rs

//! Generates the Rust types and the gRPC client and server from the schema.
//! It needs the Protocol Buffers compiler, `protoc`, on the PATH.

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_prost_build::compile_protos("../proto/store.proto")?;

    Ok(())
}

Listing 3.19 The imports of kvnode/src/rpc.rs. kvnode/src/rpc.rs

use std::sync::Arc;

use kv::{Change, Record};
use repl::{Append, Appended};
use tokio::task;
use tonic::{Request, Response, Status};

use crate::node::Node;

Listing 3.20 Including the generated code. kvnode/src/rpc.rs

/// The types that `build.rs` generates from `proto/store.proto`. The schema
/// also defines the ring, which this program does not use.
#[allow(dead_code)]
pub mod proto {
    tonic::include_proto!("store.v1");
}

pub use proto::replication_client::ReplicationClient;
pub use proto::replication_server::{Replication, ReplicationServer};

The generated types are converted to the types of kv and repl at the boundary of the service. A Record message that has no change field is rejected with the gRPC status InvalidArgument.

Listing 3.21 Conversion between the generated types and the library's types. kvnode/src/rpc.rs

impl From<&Record> for proto::Record {
    fn from(record: &Record) -> Self {
        let change = match record.change.clone() {
            Change::Put { key, value } => proto::record::Change::Put(proto::Put { key, value }),
            Change::Delete { key } => proto::record::Change::Delete(proto::Delete { key }),
        };

        proto::Record {
            sequence: record.sequence,
            change: Some(change),
        }
    }
}

impl TryFrom<proto::Record> for Record {
    type Error = Status;

    fn try_from(record: proto::Record) -> Result<Self, Status> {
        let change = match record.change {
            Some(proto::record::Change::Put(put)) => Change::Put {
                key: put.key,
                value: put.value,
            },
            Some(proto::record::Change::Delete(delete)) => Change::Delete { key: delete.key },
            None => return Err(Status::invalid_argument("a record without a change")),
        };

        Ok(Record {
            sequence: record.sequence,
            change,
        })
    }
}

impl From<Append> for proto::AppendRequest {
    fn from(message: Append) -> Self {
        proto::AppendRequest {
            shard: 0,
            term: message.term,
            previous_sequence: message.previous_sequence,
            records: message
                .records
                .iter()
                .map(proto::Record::from)
                .collect(),
            committed_sequence: message.committed_sequence,
            // The fields that leader election added are not used by this program.
            previous_term: 0,
            leader: 0,
        }
    }
}

impl From<proto::AppendResponse> for Appended {
    fn from(answer: proto::AppendResponse) -> Self {
        Appended {
            term: answer.term,
            appended: answer.appended,
            last_sequence: answer.last_sequence,
        }
    }
}

Followers run the server side of the service. The handler passes each message to Replica::handle_append. Since that method waits for the disk, the call is wrapped in block_in_place, which informs the asynchronous runtime that the current thread is about to block.

Listing 3.22 The server side of Append. kvnode/src/rpc.rs

/// The server side of `Append`, which a follower runs.
pub struct Follower {
    pub node: Arc<Node>,
}

#[tonic::async_trait]
impl Replication for Follower {
    async fn append(
        &self,
        request: Request<proto::AppendRequest>,
    ) -> Result<Response<proto::AppendResponse>, Status> {
        let request = request.into_inner();
        let records = request
            .records
            .into_iter()
            .map(Record::try_from)
            .collect::<Result<Vec<_>, _>>()?;

        let message = Append {
            term: request.term,
            previous_sequence: request.previous_sequence,
            records,
            committed_sequence: request.committed_sequence,
        };

        // Appending waits for the disk, so it runs where blocking is allowed.
        let answer = task::block_in_place(|| {
            self.node
                .replica
                .lock()
                .handle_append(message)
        })
        .map_err(|error| Status::unavailable(error.to_string()))?;

        Ok(Response::new(proto::AppendResponse {
            term: answer.term,
            appended: answer.appended,
            last_sequence: answer.last_sequence,
        }))
    }
}

The leader’s tasks

Figure 3.7 shows the parts of a leader and the order in which a write passes through them. Each box corresponds to a function or a field in the listings that follow.

Listing 3.23 The imports of kvnode/src/node.rs. kvnode/src/node.rs

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, Instant};

use kv::{Change, Condition, Entry, Summary, Written};
use kvd::api::ApiError;
use parking_lot::Mutex;
use repl::{Appended, Replica, majority_sequence};
use tokio::sync::{Notify, watch};
use tokio::task;
use tokio::time::{sleep, timeout};
use tonic::transport::{Channel, Endpoint};

use crate::rpc::{ReplicationClient, proto};

Listing 3.24 The state of a node. kvnode/src/node.rs

/// How often a leader sends a message to a follower that has nothing new to receive.
const HEARTBEAT: Duration = Duration::from_millis(100);

/// How long a leader waits for the answer to one message.
const ANSWER: Duration = Duration::from_millis(500);

/// How long a write waits for its record to reach a majority.
const COMMIT: Duration = Duration::from_secs(1);

/// What the leader knows about one follower.
pub struct Peer {
    pub address: String,
    /// The last sequence the follower has reported.
    reported: AtomicU64,
    /// When the follower last answered.
    heard: Mutex<Option<Instant>>,
}

pub struct Node {
    pub replica: Mutex<Replica>,
    pub leader: bool,
    pub peers: Vec<Peer>,
    /// The sequence the log ended at when the node started. A leader serves
    /// reads once this much is committed.
    ready: u64,
    /// Held by the write that is in progress, so that writes take turns.
    turn: tokio::sync::Mutex<()>,
    /// The committed sequence, for writes that wait for it to advance.
    committed: watch::Sender<u64>,
    /// Signalled when the leader has appended a record.
    appended: Notify,
    /// Set when a follower answers with a newer term: another leader exists.
    deposed: AtomicBool,
}

impl Node {
    pub fn new(replica: Replica, leader: bool, followers: Vec<String>) -> Arc<Node> {
        let peer = |address| Peer {
            address,
            reported: AtomicU64::new(0),
            heard: Mutex::new(None),
        };

        Arc::new(Node {
            ready: replica.last_sequence(),
            replica: Mutex::new(replica),
            leader,
            peers: followers
                .into_iter()
                .map(peer)
                .collect(),
            turn: tokio::sync::Mutex::new(()),
            committed: watch::Sender::new(0),
            appended: Notify::new(),
            deposed: AtomicBool::new(false),
        })
    }
}

A leader starts 1 task for each follower. The task repeats a single exchange: it builds a message that begins where the follower’s log is believed to end, sends it, and processes the answer. When the follower is up to date, the task waits until the leader appends a record or until the heartbeat interval has elapsed. When the answer shows that records are still missing, the task continues at once.

follow runs for the lifetime of the node, and its loop has 3 phases. It first locks the replica for no longer than it takes to build a message. It then releases the lock and waits for the answer, for at most 500 ms. Finally it passes the answer to record. If record reports that the follower is still behind, the loop continues immediately. Otherwise the task sleeps until appended is signalled or the heartbeat interval elapses, whichever occurs first.

Listing 3.25 The task that keeps a follower up to date. kvnode/src/node.rs

impl Node {
    /// Starts one task for each follower. A task sends the follower what it
    /// lacks, again and again, for as long as the node runs.
    pub fn lead(self: &Arc<Node>) -> Result<(), tonic::transport::Error> {
        for index in 0..self.peers.len() {
            let channel = Endpoint::from_shared(self.peers[index].address.clone())?
                .connect_timeout(ANSWER)
                .connect_lazy();

            tokio::spawn(Arc::clone(self).follow(index, ReplicationClient::new(channel)));
        }

        Ok(())
    }

    async fn follow(self: Arc<Node>, index: usize, mut client: ReplicationClient<Channel>) {
        let peer = &self.peers[index];

        loop {
            let message = self.replica.lock().append_for(
                peer.reported
                    .load(Ordering::Relaxed),
            );
            let request = proto::AppendRequest::from(message);

            let answer = timeout(ANSWER, client.append(request)).await;

            // An answer that did not come counts as a peer that is up to date:
            // the next round is a heartbeat, which finds out more.
            let behind = answer
                .ok()
                .and_then(Result::ok)
                .is_some_and(|answer| self.record(peer, answer.into_inner().into()));

            if !behind {
                // Wait for a new record, or send a heartbeat when none comes.
                tokio::select! {
                    () = self.appended.notified() => {}
                    () = sleep(HEARTBEAT) => {}
                }
            }
        }
    }
}

A point of design here has a large effect on how the node behaves under faults. No task holds the lock of the replica while it waits for the network. It takes the lock to build a message and again to process an answer, and releases it in between. An unreachable follower therefore delays its own task and nothing else, neither the task of the other follower nor any request from a client. Had the lock been held across the network call, a single slow follower would have stopped the whole shard, which would have defeated the purpose of waiting only for a majority.

record is called with every answer. It first compares the term of the answer with the leader’s own. A higher term shows that a newer leader exists, and the node ceases to accept writes. Otherwise it stores the sequence reported, applies the commit rule to the 3 last sequences, and publishes the new committed sequence through a watch channel, where the waiting writes observe it.

Listing 3.26 Processing an answer, and committing. kvnode/src/node.rs

impl Node {
    /// Takes note of a follower's answer and commits what a majority now
    /// holds. It returns whether the follower still lacks records.
    fn record(&self, peer: &Peer, answer: Appended) -> bool {
        let mut replica = self.replica.lock();

        if answer.term > replica.term() {
            self.deposed
                .store(true, Ordering::Relaxed);

            return false;
        }

        peer.reported
            .store(answer.last_sequence, Ordering::Relaxed);
        *peer.heard.lock() = Some(Instant::now());

        let mut stored: Vec<u64> = self
            .peers
            .iter()
            .map(|peer| {
                peer.reported
                    .load(Ordering::Relaxed)
            })
            .collect();
        stored.push(replica.last_sequence());

        replica.commit(majority_sequence(stored));
        self.committed
            .send_replace(replica.committed());

        answer.last_sequence < replica.last_sequence()
    }
}

Writes

Listing 3.27 The write path of the leader. kvnode/src/node.rs

impl Node {
    /// A write on the leader: wait for a turn, append the record, and wait
    /// until a majority of the replicas hold it.
    pub async fn write(&self, change: Change, condition: Condition) -> Result<Written, ApiError> {
        self.must_lead()?;

        let turn = self.turn.lock().await;
        let last = self.replica.lock().last_sequence();
        self.wait_for(last).await?;

        let proposed = task::block_in_place(|| {
            let mut replica = self.replica.lock();
            let created = match &change {
                Change::Put { key, .. } | Change::Delete { key } => replica.get(key).is_none(),
            };

            replica
                .propose(change, condition)
                .map(|version| Written { version, created })
        })?;

        self.appended.notify_waiters();
        self.wait_for(proposed.version)
            .await?;
        drop(turn);

        Ok(proposed)
    }

    /// Waits until the record with this sequence is committed.
    async fn wait_for(&self, sequence: u64) -> Result<(), ApiError> {
        let mut committed = self.committed.subscribe();

        let reached = committed.wait_for(|&committed| committed >= sequence);
        let in_time = timeout(COMMIT, reached)
            .await
            .is_ok_and(|reached| reached.is_ok());

        if in_time {
            Ok(())
        } else {
            Err(ApiError::Unavailable(format!(
                "record {sequence} is not on a majority of the replicas"
            )))
        }
    }

    fn must_lead(&self) -> Result<(), ApiError> {
        if !self.leader {
            return Err(ApiError::Misdirected("this node is a follower".into()));
        }

        if self.deposed.load(Ordering::Relaxed) {
            return Err(ApiError::Misdirected(
                "a leader of a newer term exists".into(),
            ));
        }

        Ok(())
    }
}

The handlers send no messages themselves. A handler appends a record, wakes both follower tasks, and waits for the committed sequence to reach the sequence of its record. A watch channel holds the most recent committed sequence, and wait_for returns as soon as that value satisfies its condition, including the case in which it already does.

If no majority stores the record within 1 second, the handler answers 503. The record remains in the leader’s log. The follower tasks go on sending it, and it is committed when a follower becomes reachable, as the simulation showed.

Reads

Listing 3.28 The read path of the leader. kvnode/src/node.rs

impl Node {
    /// A leader serves reads while a majority of the replicas, itself
    /// included, has answered within the last second, and once everything
    /// its log held at start is committed.
    fn must_serve_reads(&self) -> Result<(), ApiError> {
        self.must_lead()?;

        let recently = |peer: &&Peer| {
            peer.heard
                .lock()
                .is_some_and(|at| at.elapsed() < COMMIT)
        };
        let reachable = 1 + self
            .peers
            .iter()
            .filter(recently)
            .count();

        if 2 * reachable <= self.peers.len() + 1 {
            return Err(ApiError::Unavailable(
                "this leader cannot reach a majority of the replicas".into(),
            ));
        }

        if self.replica.lock().committed() < self.ready {
            return Err(ApiError::Unavailable(
                "the records from before the restart are not committed yet".into(),
            ));
        }

        Ok(())
    }

    pub fn get(&self, key: &str) -> Result<Entry, ApiError> {
        self.must_serve_reads()?;

        self.replica
            .lock()
            .get(key)
            .cloned()
            .ok_or(ApiError::NotFound)
    }

    pub fn list(
        &self,
        prefix: &str,
        after: Option<&str>,
        limit: usize,
    ) -> Result<(Vec<Summary>, bool), ApiError> {
        self.must_serve_reads()?;

        Ok(self
            .replica
            .lock()
            .list(prefix, after, limit))
    }

    /// The last sequence each follower has reported.
    pub fn reported(&self) -> Vec<u64> {
        self.peers
            .iter()
            .map(|peer| {
                peer.reported
                    .load(Ordering::Relaxed)
            })
            .collect()
    }
}

A leader that has been separated from its followers by a network partition cannot know whether another leader has been appointed in the meantime, and its entries may therefore be out of date. must_serve_reads accordingly requires that a majority of the replicas, counting the leader itself, has answered within the last second. The check limits to 1 second the time for which a former leader could return outdated data, on the condition that no new leader is appointed sooner than that after the old one becomes unreachable. Stricter implementations confirm their leadership with a majority for every read, at the cost of a network round trip for each.

A second check concerns a leader that has been restarted. Its log may contain committed records that have not yet been applied, since committed begins at 0. Such a leader serves reads once every record that its log held at startup has been committed again.

The HTTP API

The handlers reuse the error type and the header parsing of chapter 2, from kvd::api. They differ from the handlers of chapter 2 in passing writes to Node::write and reads to Node::get.

Listing 3.29 The handlers for PUT, DELETE and GET. kvnode/src/main.rs

async fn put_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    let written = node
        .write(
            Change::Put {
                key,
                value: body.to_vec(),
            },
            condition,
        )
        .await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    node.write(Change::Delete { key }, condition)
        .await?;

    Ok(StatusCode::NO_CONTENT)
}

async fn get_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let cached = version_header(&headers, IF_NONE_MATCH)?;
    let entry = node.get(&key)?;
    let tag = [(ETAG, etag(entry.version))];

    if cached == Some(Condition::Version(entry.version)) {
        return Ok((StatusCode::NOT_MODIFIED, tag).into_response());
    }

    Ok((
        tag,
        [(CONTENT_TYPE, "application/octet-stream")],
        entry.value,
    )
        .into_response())
}

Listing 3.30 Listing keys, and the health of the node. kvnode/src/main.rs

#[derive(Deserialize)]
struct ListQuery {
    #[serde(default)]
    prefix: String,
    after: Option<String>,
    limit: Option<usize>,
}

#[derive(Serialize)]
struct Item {
    key: String,
    version: u64,
    size: usize,
}

#[derive(Serialize)]
struct Page {
    keys: Vec<Item>,
    next_after: Option<String>,
}

async fn list_keys(
    State(node): State<Arc<Node>>,
    Query(query): Query<ListQuery>,
) -> Result<Json<Page>, ApiError> {
    let limit = query
        .limit
        .unwrap_or(100)
        .clamp(1, 1000);
    let (page, more) = node.list(&query.prefix, query.after.as_deref(), limit)?;
    let next_after = if more {
        page.last()
            .map(|last| last.key.clone())
    } else {
        None
    };
    let keys = page
        .into_iter()
        .map(|item| Item {
            key: item.key,
            version: item.version,
            size: item.size,
        })
        .collect();

    Ok(Json(Page { keys, next_after }))
}

#[derive(Serialize)]
struct Health {
    role: &'static str,
    term: u64,
    last_sequence: u64,
    committed: u64,
    /// The last sequence each follower has reported to this leader.
    followers: Vec<u64>,
}

async fn health(State(node): State<Arc<Node>>) -> Json<Health> {
    let replica = node.replica.lock();

    Json(Health {
        role: if node.leader { "leader" } else { "follower" },
        term: replica.term(),
        last_sequence: replica.last_sequence(),
        committed: replica.committed(),
        followers: node.reported(),
    })
}

Listing 3.31 Starting a leader or a follower. kvnode/src/main.rs

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let replica = Replica::open(&options.data, !options.no_sync, options.term)?;

    println!(
        "term {}, log ends at {}",
        replica.term(),
        replica.last_sequence()
    );

    let node = Node::new(replica, options.role == Role::Leader, options.followers);

    match options.role {
        Role::Leader => node.lead()?,
        Role::Follower => {
            let service = ReplicationServer::new(Follower {
                node: Arc::clone(&node),
            });

            tokio::spawn(
                Server::builder()
                    .add_service(service)
                    .serve(options.replication),
            );
            println!("accepting records on {}", options.replication);
        }
    }

    let routes = Router::new()
        .route("/v1/keys", get(list_keys))
        .route(
            "/v1/keys/{key}",
            get(get_key)
                .put(put_key)
                .delete(delete_key),
        )
        .route("/healthz", get(health))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(node);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Running 3 nodes

The followers are started first, each with an address for HTTP and another for replication. The leader is then started with the replication addresses of both followers.

$ ./target/release/kvnode --role follower --listen 127.0.0.1:7002 --replication 127.0.0.1:7102 --data /var/tmp/n2 &
term 1, log ends at 0
accepting records on 127.0.0.1:7102
listening on 127.0.0.1:7002
$ ./target/release/kvnode --role follower --listen 127.0.0.1:7003 --replication 127.0.0.1:7103 --data /var/tmp/n3 &
$ ./target/release/kvnode --role leader --listen 127.0.0.1:7001 --data /var/tmp/n1 \
      --follower http://127.0.0.1:7102 --follower http://127.0.0.1:7103 &
term 1, log ends at 0
listening on 127.0.0.1:7001

A write to the leader is stored by all 3 nodes, and a request to a follower is refused with the status that chapter 1 specifies for a request sent to the wrong node.

$ curl -i -X PUT http://127.0.0.1:7001/v1/keys/user-1 --data-binary 'ada'
HTTP/1.1 201 Created
etag: "1"
content-length: 0

$ curl -i -X PUT http://127.0.0.1:7002/v1/keys/user-1 --data-binary 'x'
HTTP/1.1 421 Misdirected Request
content-type: application/problem+json
content-length: 117

{"type":"/problems/not-leader","title":"This node is not the leader","status":421,"detail":"this node is a follower"}
$ for port in 7001 7002 7003; do curl http://127.0.0.1:$port/healthz; echo; done
{"role":"leader","term":1,"last_sequence":1,"committed":1,"followers":[1,1]}
{"role":"follower","term":1,"last_sequence":1,"committed":1,"followers":[]}
{"role":"follower","term":1,"last_sequence":1,"committed":1,"followers":[]}
$ cmp /var/tmp/n1/kv.log /var/tmp/n2/kv.log && cmp /var/tmp/n1/kv.log /var/tmp/n3/kv.log && echo identical
identical

The health response of the leader lists the last sequence reported by each follower. The 3 log files are identical byte for byte, which is invariant 1 in the strongest form it could take.

Experiments

All the nodes in these experiments run on 1 machine and share its disk. On separate machines, each node has a disk of its own and the nodes do not compete for it. Nothing else changes: each node is started with the addresses of its peers in place of 127.0.0.1.

The cost of replication

$ time ./target/release/kvctl load --server http://127.0.0.1:7001 --count 300 --acked acked.txt
300 writes acknowledged
3.89 s, 77 writes per second

The replicated shard completes 77 writes per second, against 185 for the single node of chapter 2 on the same disk. Each write now waits for 2 disks in succession, the leader’s and then a follower’s, and for a network round trip. Chapter 1 allowed for this in its capacity estimate by counting 3 stored records for every write.

A follower stops

A follower is killed while a load is running.

$ ./target/release/kvctl load --server http://127.0.0.1:7001 --count 400 --acked acked.txt &
$ kill -9 <pid of the node on port 7002>
$ curl http://127.0.0.1:7001/healthz
{"role":"leader","term":1,"last_sequence":435,"committed":434,"followers":[417,434]}
400 writes acknowledged
$ curl http://127.0.0.1:7001/healthz
{"role":"leader","term":1,"last_sequence":700,"committed":700,"followers":[417,700]}

$ ./target/release/kvnode --role follower --listen 127.0.0.1:7002 --replication 127.0.0.1:7102 --data /var/tmp/n2 &
term 1, log ends at 417
$ curl http://127.0.0.1:7001/healthz
{"role":"leader","term":1,"last_sequence":700,"committed":700,"followers":[700,700]}
$ cmp /var/tmp/n1/kv.log /var/tmp/n2/kv.log && echo identical
identical

Apart from restarting the process, no action by an operator was required.

The majority is lost

Both followers are killed.

$ kill -9 <pid of the node on port 7002> <pid of the node on port 7003>
$ curl -o /dev/null -w '%{http_code} in %{time_total} s\n' -X PUT http://127.0.0.1:7001/v1/keys/alone --data-binary 'x'
503 in 1.011346 s
$ curl -X PUT http://127.0.0.1:7001/v1/keys/alone2 --data-binary 'x'
{"type":"/problems/store-unavailable","title":"The node cannot serve the request","status":503,"detail":"record 701 is not on a majority of the replicas"}
$ curl http://127.0.0.1:7001/v1/keys/key-000001
{"type":"/problems/store-unavailable","title":"The node cannot serve the request","status":503,"detail":"this leader cannot reach a majority of the replicas"}
$ curl http://127.0.0.1:7001/healthz
{"role":"leader","term":1,"last_sequence":701,"committed":700,"followers":[700,700]}

$ ./target/release/kvnode --role follower --listen 127.0.0.1:7003 --replication 127.0.0.1:7103 --data /var/tmp/n3 &
$ curl http://127.0.0.1:7001/healthz
{"role":"leader","term":1,"last_sequence":701,"committed":701,"followers":[700,701]}
$ curl -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7001/v1/keys/alone
200
$ curl -o /dev/null -w '%{http_code}\n' -X PUT http://127.0.0.1:7001/v1/keys/after --data-binary 'x'
201

This is exactly the behaviour that chapter 1 specified for the minority side of a partition. The leader remains consistent and becomes unavailable, and it would be easy, and wrong, to regard its refusal to serve as a defect. A leader that went on answering in this condition would be answering from data that it cannot know to be current.

The leader is lost

In this experiment a leader is killed under load and its data directory is moved away, which simulates the loss of a machine together with its disk.

$ ./target/release/kvctl load --server http://127.0.0.1:7001 --count 2000 --acked acked.txt &
$ kill -9 <pid of the leader>
write 229 failed: io: Peer disconnected
228 writes acknowledged
$ mv /var/tmp/n1 /var/tmp/n1.lost
$ for port in 7002 7003; do curl http://127.0.0.1:$port/healthz; echo; done
{"role":"follower","term":1,"last_sequence":228,"committed":227,"followers":[]}
{"role":"follower","term":1,"last_sequence":228,"committed":227,"followers":[]}

$ kill -9 <pid of the node on port 7003>
$ ./target/release/kvnode --role follower --listen 127.0.0.1:7001 --replication 127.0.0.1:7101 --data /var/tmp/n1 &
$ ./target/release/kvnode --role leader --term 2 --listen 127.0.0.1:7003 --data /var/tmp/n3 \
      --follower http://127.0.0.1:7102 --follower http://127.0.0.1:7101 &
term 2, log ends at 228
listening on 127.0.0.1:7003
$ for port in 7003 7002 7001; do curl http://127.0.0.1:$port/healthz; echo; done
{"role":"leader","term":2,"last_sequence":228,"committed":228,"followers":[228,228]}
{"role":"follower","term":2,"last_sequence":228,"committed":228,"followers":[]}
{"role":"follower","term":2,"last_sequence":228,"committed":228,"followers":[]}
$ ./target/release/kvctl verify --server http://127.0.0.1:7003 --acked acked.txt
228 acknowledged, 228 found, 0 missing

The second row illustrates why the committed sequence is not stored on disk. What a follower knows to be committed always lags behind its leader, and after a leader has been lost, the committed sequence is established afresh from the logs: a new leader commits a record as soon as a majority has reported it.

For the promotion, the follower with the longer log must be chosen. By invariant 2, every acknowledged record is on at least 1 of the 2 followers, though not necessarily on both. If their logs differ in length, the longer contains every acknowledged record, and the shorter may lack some of the most recent. To promote the shorter would be to discard acknowledged writes, and this choice, made here by a person reading 2 numbers, is the one that chapter 4 has to teach the nodes to make for themselves.

The former leader returns

Suppose that a former leader was not lost after all and begins to send messages again, as it might when a network partition ends. A leader with term 1 is started against the 2 followers.

$ ./target/release/kvnode --role leader --term 1 --listen 127.0.0.1:7009 --data /var/tmp/n9 \
      --follower http://127.0.0.1:7102 --follower http://127.0.0.1:7101 &
$ curl -i -X PUT http://127.0.0.1:7009/v1/keys/stale --data-binary 'x'
HTTP/1.1 421 Misdirected Request
content-type: application/problem+json
content-length: 125

{"type":"/problems/not-leader","title":"This node is not the leader","status":421,"detail":"a leader of a newer term exists"}
$ curl http://127.0.0.1:7002/healthz
{"role":"follower","term":2,"last_sequence":228,"committed":228,"followers":[]}
$ curl -o /dev/null -w '%{http_code}\n' -X PUT http://127.0.0.1:7003/v1/keys/fresh --data-binary 'x'
201

Both followers are at term 2 and refuse messages of term 1. Each of their answers carries the higher term, so that the old leader records that it has been replaced and rejects writes with 421. The shard meanwhile continues to accept writes through its leader of term 2. Terms are what make a change of leader safe, since a leader that has been replaced cannot add a record to any follower.

Limitations

The first 2 limitations are removed in chapter 4, and the third in chapter 6.

The complete programs

The log and the store are listed in chapter 2. Those listings include Log::append_records as well as apply, check and page, which this chapter uses.

Listing 3.32 The complete repl/src/lib.rs. repl/src/lib.rs

//! Replication of a log from a leader to its followers.
//!
//! A `Replica` is one copy of a shard: a log, and the entries that result
//! from the records that are known to be on a majority of the copies. The
//! same type serves as leader and as follower. This module does no I/O on
//! the network. It builds the messages and processes them, and the caller
//! carries them between replicas.

use std::collections::BTreeMap;
use std::io;
use std::path::Path;

use kv::{Change, Condition, Entry, Log, Record, StoreError, Summary, apply, check, page};

/// The message a leader sends to a follower.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Append {
    /// The term of the sender.
    pub term: u64,
    /// The sequence of the record that precedes `records` in the leader's log.
    pub previous_sequence: u64,
    pub records: Vec<Record>,
    /// Every record up to this sequence is on a majority of the replicas.
    pub committed_sequence: u64,
}

/// The follower's answer.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Appended {
    pub term: u64,
    /// False when the follower's log does not end at `previous_sequence`,
    /// or when the sender's term is older than the follower's.
    pub appended: bool,
    /// The sequence of the last record in the follower's log.
    pub last_sequence: u64,
}

pub struct Replica {
    log: Log,
    /// Every record of the log, in order. Record `n` is at index `n - 1`.
    records: Vec<Record>,
    /// The result of applying the records up to `committed`.
    entries: BTreeMap<String, Entry>,
    /// The sequence of the last record that is applied to `entries`.
    committed: u64,
    term: u64,
}

impl Replica {
    /// Opens the replica in `directory`. Nothing is applied yet: a replica
    /// that has restarted does not know which of its records are committed.
    pub fn open(directory: &Path, sync: bool, term: u64) -> io::Result<Replica> {
        let (log, recovery) = Log::open(directory, sync)?;

        Ok(Replica {
            log,
            records: recovery.records,
            entries: BTreeMap::new(),
            committed: 0,
            term,
        })
    }

    pub fn term(&self) -> u64 {
        self.term
    }

    pub fn last_sequence(&self) -> u64 {
        self.records.len() as u64
    }

    pub fn committed(&self) -> u64 {
        self.committed
    }

    /// One page of the committed keys.
    pub fn list(&self, prefix: &str, after: Option<&str>, limit: usize) -> (Vec<Summary>, bool) {
        page(&self.entries, prefix, after, limit)
    }

    /// Every record of the log, in order.
    pub fn records(&self) -> &[Record] {
        &self.records
    }

    /// The committed value of a key.
    pub fn get(&self, key: &str) -> Option<&Entry> {
        self.entries.get(key)
    }

    /// Applies the records up to `sequence` that are not applied yet.
    pub fn commit(&mut self, sequence: u64) {
        let until = sequence.min(self.last_sequence());

        while self.committed < until {
            apply(
                &mut self.entries,
                self.records[self.committed as usize].clone(),
            );
            self.committed += 1;
        }
    }
}

impl Replica {
    /// Whether every record of the log is committed. A leader accepts a new
    /// write only in this state, so that the condition of the write is
    /// checked against the result of all earlier writes.
    pub fn is_settled(&self) -> bool {
        self.committed == self.last_sequence()
    }

    /// Checks the condition of a write and appends its record to the
    /// leader's log. The record is not applied: it is not committed yet.
    pub fn propose(&mut self, change: Change, condition: Condition) -> Result<u64, StoreError> {
        let key = match &change {
            Change::Put { key, .. } | Change::Delete { key } => key,
        };
        let current = self
            .entries
            .get(key)
            .map(|entry| entry.version);

        if matches!(change, Change::Delete { .. }) && current.is_none() {
            return Err(StoreError::NotFound);
        }

        check(condition, current)?;

        let record = self.log.append(change)?;
        let sequence = record.sequence;
        self.records.push(record);

        Ok(sequence)
    }
}

impl Replica {
    /// The message that brings a follower whose log ends at `follower_last`
    /// up to the end of this log. With nothing to send it is a heartbeat.
    pub fn append_for(&self, follower_last: u64) -> Append {
        let from = follower_last.min(self.last_sequence()) as usize;

        Append {
            term: self.term,
            previous_sequence: from as u64,
            records: self.records[from..].to_vec(),
            committed_sequence: self.committed,
        }
    }
}


/// The highest sequence that is stored on a majority, given the last
/// sequence of every replica, the leader included.
pub fn majority_sequence(mut last_sequences: Vec<u64>) -> u64 {
    last_sequences.sort_unstable_by(|a, b| b.cmp(a));

    last_sequences
        .get(last_sequences.len() / 2)
        .copied()
        .unwrap_or(0)
}

impl Replica {
    /// Processes a message from a leader.
    pub fn handle_append(&mut self, request: Append) -> io::Result<Appended> {
        let refuse = |replica: &Replica| Appended {
            term: replica.term,
            appended: false,
            last_sequence: replica.last_sequence(),
        };

        if request.term < self.term {
            return Ok(refuse(self));
        }

        self.term = request.term;

        if request.previous_sequence != self.last_sequence() {
            return Ok(refuse(self));
        }

        if !request.records.is_empty() {
            self.log
                .append_records(&request.records)?;
            self.records.extend(request.records);
        }

        self.commit(request.committed_sequence);

        Ok(Appended {
            term: self.term,
            appended: true,
            last_sequence: self.last_sequence(),
        })
    }
}

Listing 3.33 The complete replsim/src/main.rs. replsim/src/main.rs

//! Command replsim runs a leader and 2 followers in one process and carries
//! the messages between them by hand, so that it can lose any message it
//! chooses. It first walks through the cases of the chapter, and then tries
//! every combination of lost requests and lost answers over 3 writes and
//! checks what the replicas hold after each.
//!
//! Run it from code/ with:
//!
//!     cargo run --release -p replsim

use std::env;
use std::error::Error;
use std::fs;
use std::path::{Path, PathBuf};
use std::process;

use kv::{Change, Condition};
use repl::{Replica, majority_sequence};

/// What happens to one exchange between the leader and a follower.
#[derive(Clone, Copy, PartialEq, Eq)]
enum Fate {
    Delivered,
    /// The follower does not receive the message.
    RequestLost,
    /// The follower processes the message, and its answer does not arrive.
    AnswerLost,
}

/// A leader, 2 followers, and what the leader knows about each follower.
struct Cluster {
    leader: Replica,
    followers: Vec<Replica>,
    /// The last sequence each follower has reported.
    reported: Vec<u64>,
}

impl Cluster {
    fn open(root: &Path) -> Result<Cluster, Box<dyn Error>> {
        let open = |name: &str| Replica::open(&root.join(name), false, 1);

        Ok(Cluster {
            leader: open("leader")?,
            followers: vec![open("follower-1")?, open("follower-2")?],
            reported: vec![0, 0],
        })
    }

    /// One exchange with a follower: the leader sends what the follower
    /// lacks, as far as it knows, and commits what a majority now holds.
    fn exchange(&mut self, follower: usize, fate: Fate) -> Result<(), Box<dyn Error>> {
        let message = self
            .leader
            .append_for(self.reported[follower]);

        if fate == Fate::RequestLost {
            return Ok(());
        }

        let answer = self.followers[follower].handle_append(message)?;

        if fate == Fate::AnswerLost {
            return Ok(());
        }

        self.reported[follower] = answer.last_sequence;

        let stored = vec![
            self.leader.last_sequence(),
            self.reported[0],
            self.reported[1],
        ];
        self.leader
            .commit(majority_sequence(stored));

        Ok(())
    }

    /// A write: the leader appends the record and makes one exchange with
    /// each follower. It returns whether the write was acknowledged, or
    /// `None` when the leader could not accept it.
    fn write(
        &mut self,
        key: &str,
        value: &str,
        fates: [Fate; 2],
    ) -> Result<Option<bool>, Box<dyn Error>> {
        if !self.leader.is_settled() {
            return Ok(None);
        }

        let change = Change::Put {
            key: key.into(),
            value: value.into(),
        };
        let sequence = self
            .leader
            .propose(change, Condition::Always)?;

        self.exchange(0, fates[0])?;
        self.exchange(1, fates[1])?;

        Ok(Some(self.leader.committed() >= sequence))
    }

    /// The last sequence of the leader and of each follower.
    fn lasts(&self) -> [u64; 3] {
        [
            self.leader.last_sequence(),
            self.followers[0].last_sequence(),
            self.followers[1].last_sequence(),
        ]
    }
}

const BOTH: [Fate; 2] = [Fate::Delivered, Fate::Delivered];

fn cases(root: &Path) -> Result<(), Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;
    let show = |cluster: &Cluster| {
        format!(
            "logs end at {:?}, committed {}",
            cluster.lasts(),
            cluster.leader.committed()
        )
    };

    for key in ["a", "b", "c"] {
        cluster.write(key, "1", BOTH)?;
    }

    println!("3 writes, every message delivered: {}", show(&cluster));

    for key in ["d", "e", "f"] {
        cluster.write(key, "1", [Fate::RequestLost, Fate::Delivered])?;
    }

    println!(
        "3 writes while follower 1 is unreachable: {}",
        show(&cluster)
    );

    cluster.exchange(0, Fate::Delivered)?;
    println!("follower 1 returns, 1 exchange: {}", show(&cluster));

    for key in ["g", "h", "i"] {
        cluster.write(key, "1", [Fate::AnswerLost, Fate::Delivered])?;
    }

    println!(
        "3 writes while the answers of follower 1 are lost: {}",
        show(&cluster)
    );
    println!(
        "  the leader believes that the log of follower 1 ends at {}",
        cluster.reported[0]
    );

    cluster.exchange(0, Fate::Delivered)?;
    println!(
        "  next exchange: the leader sends records 7 to 9, follower 1 refuses and reports {}",
        cluster.reported[0]
    );

    cluster.exchange(0, Fate::Delivered)?;
    println!("  the exchange after it: {}", show(&cluster));

    let acknowledged =
        cluster.write("j", "1", [Fate::RequestLost, Fate::RequestLost])? == Some(true);
    println!(
        "1 write while both followers are unreachable: acknowledged {acknowledged}, {}",
        show(&cluster)
    );
    println!(
        "  the leader reads j as {:?}",
        cluster
            .leader
            .get("j")
            .map(|entry| entry.version)
    );

    let accepted = cluster
        .write("k", "1", [Fate::RequestLost, Fate::RequestLost])?
        .is_some();
    println!("  the leader accepts another write: {accepted}");

    cluster.exchange(1, Fate::Delivered)?;
    println!("follower 2 returns: {}", show(&cluster));
    println!(
        "  the leader reads j as {:?}",
        cluster
            .leader
            .get("j")
            .map(|entry| entry.version)
    );

    let mut newer = Replica::open(&root.join("newer"), false, 2)?;
    let answer = newer.handle_append(cluster.leader.append_for(0))?;
    println!(
        "a replica at term 2 receives a message of term 1: appended {}, its term {}",
        answer.appended, answer.term
    );

    Ok(())
}

const WRITES: usize = 3;

/// The fate of exchange number `index` under a pattern: 2 bits for each.
fn fate(pattern: u32, index: usize) -> Fate {
    match (pattern >> (2 * index)) & 3 {
        0 | 1 => Fate::Delivered,
        2 => Fate::RequestLost,
        3.. => Fate::AnswerLost,
    }
}

/// Runs 3 writes under one pattern of losses and checks the replicas. It
/// returns the number of writes that were acknowledged.
fn run(root: &Path, pattern: u32) -> Result<usize, Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;
    let mut acknowledged = Vec::new();

    for number in 0..WRITES {
        let fates = [fate(pattern, 2 * number), fate(pattern, 2 * number + 1)];

        if cluster.write(&format!("key-{number}"), "1", fates)? == Some(true) {
            acknowledged.push(cluster.leader.last_sequence());
        }
    }

    for &sequence in &acknowledged {
        let holders = cluster
            .lasts()
            .iter()
            .filter(|&&last| last >= sequence)
            .count();

        if holders < 2 {
            return Err(format!(
                "pattern {pattern}: record {sequence} was acknowledged and is in {holders} log"
            )
            .into());
        }
    }

    for follower in &cluster.followers {
        if !cluster
            .leader
            .records()
            .starts_with(follower.records())
        {
            return Err(format!(
                "pattern {pattern}: a follower's log is not the beginning of the leader's"
            )
            .into());
        }
    }

    for _ in 0..2 {
        cluster.exchange(0, Fate::Delivered)?;
        cluster.exchange(1, Fate::Delivered)?;
    }

    let [leader, first, second] = cluster.lasts();

    if first != leader || second != leader || !cluster.leader.is_settled() {
        return Err(format!(
            "pattern {pattern}: the logs differ after every message was delivered"
        )
        .into());
    }

    Ok(acknowledged.len())
}

fn exhaustive(root: &Path) -> Result<(), Box<dyn Error>> {
    let patterns = 1u32 << (4 * WRITES);
    let mut by_acknowledged = [0; WRITES + 1];

    for pattern in 0..patterns {
        let directory = root.join(format!("pattern-{pattern}"));
        by_acknowledged[run(&directory, pattern)?] += 1;
        fs::remove_dir_all(&directory)?;
    }

    println!("{patterns} patterns of lost messages over {WRITES} writes:");
    println!("  every acknowledged record was in at least 2 of the 3 logs");
    println!("  every follower's log was the beginning of the leader's log");
    println!("  after the losses stopped, the 3 logs were equal and every record was committed");

    for (count, patterns) in by_acknowledged.iter().enumerate() {
        println!("  {count} of {WRITES} writes acknowledged: {patterns} patterns");
    }

    Ok(())
}

fn directory() -> PathBuf {
    env::temp_dir().join(format!("replsim-{}", process::id()))
}

fn main() -> Result<(), Box<dyn Error>> {
    let root = directory();
    let result = cases(&root.join("cases")).and_then(|()| exhaustive(&root));

    fs::remove_dir_all(&root)?;

    result
}

Listing 3.34 The complete kvnode/Cargo.toml. kvnode/Cargo.toml

[package]
name = "kvnode"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true
license.workspace = true
publish.workspace = true

[dependencies]
kv.workspace = true
kvd.workspace = true
repl.workspace = true
axum.workspace = true
clap.workspace = true
parking_lot.workspace = true
prost.workspace = true
serde.workspace = true
tokio.workspace = true
tonic.workspace = true
tonic-prost.workspace = true

[build-dependencies]
tonic-prost-build.workspace = true

Listing 3.35 The complete kvnode/src/rpc.rs. kvnode/src/rpc.rs

//! The replication service: the generated gRPC types, the conversion between
//! them and the types of the `repl` crate, and the server side that a
//! follower runs.

use std::sync::Arc;

use kv::{Change, Record};
use repl::{Append, Appended};
use tokio::task;
use tonic::{Request, Response, Status};

use crate::node::Node;

/// The types that `build.rs` generates from `proto/store.proto`. The schema
/// also defines the ring, which this program does not use.
#[allow(dead_code)]
pub mod proto {
    tonic::include_proto!("store.v1");
}

pub use proto::replication_client::ReplicationClient;
pub use proto::replication_server::{Replication, ReplicationServer};

impl From<&Record> for proto::Record {
    fn from(record: &Record) -> Self {
        let change = match record.change.clone() {
            Change::Put { key, value } => proto::record::Change::Put(proto::Put { key, value }),
            Change::Delete { key } => proto::record::Change::Delete(proto::Delete { key }),
        };

        proto::Record {
            sequence: record.sequence,
            change: Some(change),
        }
    }
}

impl TryFrom<proto::Record> for Record {
    type Error = Status;

    fn try_from(record: proto::Record) -> Result<Self, Status> {
        let change = match record.change {
            Some(proto::record::Change::Put(put)) => Change::Put {
                key: put.key,
                value: put.value,
            },
            Some(proto::record::Change::Delete(delete)) => Change::Delete { key: delete.key },
            None => return Err(Status::invalid_argument("a record without a change")),
        };

        Ok(Record {
            sequence: record.sequence,
            change,
        })
    }
}

impl From<Append> for proto::AppendRequest {
    fn from(message: Append) -> Self {
        proto::AppendRequest {
            shard: 0,
            term: message.term,
            previous_sequence: message.previous_sequence,
            records: message
                .records
                .iter()
                .map(proto::Record::from)
                .collect(),
            committed_sequence: message.committed_sequence,
            // The fields that leader election added are not used by this program.
            previous_term: 0,
            leader: 0,
        }
    }
}

impl From<proto::AppendResponse> for Appended {
    fn from(answer: proto::AppendResponse) -> Self {
        Appended {
            term: answer.term,
            appended: answer.appended,
            last_sequence: answer.last_sequence,
        }
    }
}

/// The server side of `Append`, which a follower runs.
pub struct Follower {
    pub node: Arc<Node>,
}

#[tonic::async_trait]
impl Replication for Follower {
    async fn append(
        &self,
        request: Request<proto::AppendRequest>,
    ) -> Result<Response<proto::AppendResponse>, Status> {
        let request = request.into_inner();
        let records = request
            .records
            .into_iter()
            .map(Record::try_from)
            .collect::<Result<Vec<_>, _>>()?;

        let message = Append {
            term: request.term,
            previous_sequence: request.previous_sequence,
            records,
            committed_sequence: request.committed_sequence,
        };

        // Appending waits for the disk, so it runs where blocking is allowed.
        let answer = task::block_in_place(|| {
            self.node
                .replica
                .lock()
                .handle_append(message)
        })
        .map_err(|error| Status::unavailable(error.to_string()))?;

        Ok(Response::new(proto::AppendResponse {
            term: answer.term,
            appended: answer.appended,
            last_sequence: answer.last_sequence,
        }))
    }
}

Listing 3.36 The complete kvnode/src/node.rs. kvnode/src/node.rs

//! A node of a replicated shard: a replica, and for a leader the tasks that
//! keep its followers up to date.

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, Instant};

use kv::{Change, Condition, Entry, Summary, Written};
use kvd::api::ApiError;
use parking_lot::Mutex;
use repl::{Appended, Replica, majority_sequence};
use tokio::sync::{Notify, watch};
use tokio::task;
use tokio::time::{sleep, timeout};
use tonic::transport::{Channel, Endpoint};

use crate::rpc::{ReplicationClient, proto};

/// How often a leader sends a message to a follower that has nothing new to receive.
const HEARTBEAT: Duration = Duration::from_millis(100);

/// How long a leader waits for the answer to one message.
const ANSWER: Duration = Duration::from_millis(500);

/// How long a write waits for its record to reach a majority.
const COMMIT: Duration = Duration::from_secs(1);

/// What the leader knows about one follower.
pub struct Peer {
    pub address: String,
    /// The last sequence the follower has reported.
    reported: AtomicU64,
    /// When the follower last answered.
    heard: Mutex<Option<Instant>>,
}

pub struct Node {
    pub replica: Mutex<Replica>,
    pub leader: bool,
    pub peers: Vec<Peer>,
    /// The sequence the log ended at when the node started. A leader serves
    /// reads once this much is committed.
    ready: u64,
    /// Held by the write that is in progress, so that writes take turns.
    turn: tokio::sync::Mutex<()>,
    /// The committed sequence, for writes that wait for it to advance.
    committed: watch::Sender<u64>,
    /// Signalled when the leader has appended a record.
    appended: Notify,
    /// Set when a follower answers with a newer term: another leader exists.
    deposed: AtomicBool,
}

impl Node {
    pub fn new(replica: Replica, leader: bool, followers: Vec<String>) -> Arc<Node> {
        let peer = |address| Peer {
            address,
            reported: AtomicU64::new(0),
            heard: Mutex::new(None),
        };

        Arc::new(Node {
            ready: replica.last_sequence(),
            replica: Mutex::new(replica),
            leader,
            peers: followers
                .into_iter()
                .map(peer)
                .collect(),
            turn: tokio::sync::Mutex::new(()),
            committed: watch::Sender::new(0),
            appended: Notify::new(),
            deposed: AtomicBool::new(false),
        })
    }
}

impl Node {
    /// Starts one task for each follower. A task sends the follower what it
    /// lacks, again and again, for as long as the node runs.
    pub fn lead(self: &Arc<Node>) -> Result<(), tonic::transport::Error> {
        for index in 0..self.peers.len() {
            let channel = Endpoint::from_shared(self.peers[index].address.clone())?
                .connect_timeout(ANSWER)
                .connect_lazy();

            tokio::spawn(Arc::clone(self).follow(index, ReplicationClient::new(channel)));
        }

        Ok(())
    }

    async fn follow(self: Arc<Node>, index: usize, mut client: ReplicationClient<Channel>) {
        let peer = &self.peers[index];

        loop {
            let message = self.replica.lock().append_for(
                peer.reported
                    .load(Ordering::Relaxed),
            );
            let request = proto::AppendRequest::from(message);

            let answer = timeout(ANSWER, client.append(request)).await;

            // An answer that did not come counts as a peer that is up to date:
            // the next round is a heartbeat, which finds out more.
            let behind = answer
                .ok()
                .and_then(Result::ok)
                .is_some_and(|answer| self.record(peer, answer.into_inner().into()));

            if !behind {
                // Wait for a new record, or send a heartbeat when none comes.
                tokio::select! {
                    () = self.appended.notified() => {}
                    () = sleep(HEARTBEAT) => {}
                }
            }
        }
    }
}

impl Node {
    /// Takes note of a follower's answer and commits what a majority now
    /// holds. It returns whether the follower still lacks records.
    fn record(&self, peer: &Peer, answer: Appended) -> bool {
        let mut replica = self.replica.lock();

        if answer.term > replica.term() {
            self.deposed
                .store(true, Ordering::Relaxed);

            return false;
        }

        peer.reported
            .store(answer.last_sequence, Ordering::Relaxed);
        *peer.heard.lock() = Some(Instant::now());

        let mut stored: Vec<u64> = self
            .peers
            .iter()
            .map(|peer| {
                peer.reported
                    .load(Ordering::Relaxed)
            })
            .collect();
        stored.push(replica.last_sequence());

        replica.commit(majority_sequence(stored));
        self.committed
            .send_replace(replica.committed());

        answer.last_sequence < replica.last_sequence()
    }
}

impl Node {
    /// A write on the leader: wait for a turn, append the record, and wait
    /// until a majority of the replicas hold it.
    pub async fn write(&self, change: Change, condition: Condition) -> Result<Written, ApiError> {
        self.must_lead()?;

        let turn = self.turn.lock().await;
        let last = self.replica.lock().last_sequence();
        self.wait_for(last).await?;

        let proposed = task::block_in_place(|| {
            let mut replica = self.replica.lock();
            let created = match &change {
                Change::Put { key, .. } | Change::Delete { key } => replica.get(key).is_none(),
            };

            replica
                .propose(change, condition)
                .map(|version| Written { version, created })
        })?;

        self.appended.notify_waiters();
        self.wait_for(proposed.version)
            .await?;
        drop(turn);

        Ok(proposed)
    }

    /// Waits until the record with this sequence is committed.
    async fn wait_for(&self, sequence: u64) -> Result<(), ApiError> {
        let mut committed = self.committed.subscribe();

        let reached = committed.wait_for(|&committed| committed >= sequence);
        let in_time = timeout(COMMIT, reached)
            .await
            .is_ok_and(|reached| reached.is_ok());

        if in_time {
            Ok(())
        } else {
            Err(ApiError::Unavailable(format!(
                "record {sequence} is not on a majority of the replicas"
            )))
        }
    }

    fn must_lead(&self) -> Result<(), ApiError> {
        if !self.leader {
            return Err(ApiError::Misdirected("this node is a follower".into()));
        }

        if self.deposed.load(Ordering::Relaxed) {
            return Err(ApiError::Misdirected(
                "a leader of a newer term exists".into(),
            ));
        }

        Ok(())
    }
}

impl Node {
    /// A leader serves reads while a majority of the replicas, itself
    /// included, has answered within the last second, and once everything
    /// its log held at start is committed.
    fn must_serve_reads(&self) -> Result<(), ApiError> {
        self.must_lead()?;

        let recently = |peer: &&Peer| {
            peer.heard
                .lock()
                .is_some_and(|at| at.elapsed() < COMMIT)
        };
        let reachable = 1 + self
            .peers
            .iter()
            .filter(recently)
            .count();

        if 2 * reachable <= self.peers.len() + 1 {
            return Err(ApiError::Unavailable(
                "this leader cannot reach a majority of the replicas".into(),
            ));
        }

        if self.replica.lock().committed() < self.ready {
            return Err(ApiError::Unavailable(
                "the records from before the restart are not committed yet".into(),
            ));
        }

        Ok(())
    }

    pub fn get(&self, key: &str) -> Result<Entry, ApiError> {
        self.must_serve_reads()?;

        self.replica
            .lock()
            .get(key)
            .cloned()
            .ok_or(ApiError::NotFound)
    }

    pub fn list(
        &self,
        prefix: &str,
        after: Option<&str>,
        limit: usize,
    ) -> Result<(Vec<Summary>, bool), ApiError> {
        self.must_serve_reads()?;

        Ok(self
            .replica
            .lock()
            .list(prefix, after, limit))
    }

    /// The last sequence each follower has reported.
    pub fn reported(&self) -> Vec<u64> {
        self.peers
            .iter()
            .map(|peer| {
                peer.reported
                    .load(Ordering::Relaxed)
            })
            .collect()
    }
}

Listing 3.37 The complete kvnode/src/main.rs. kvnode/src/main.rs

//! Command kvnode is a storage node that replicates its log. One node of a
//! shard is started as the leader, with the addresses of its followers, and
//! the others as followers. The leader serves the HTTP API and acknowledges
//! a write when a majority of the nodes hold its record.
//!
//! Run 3 nodes from code/ with:
//!
//!     cargo run -p kvnode -- --role follower --listen 127.0.0.1:7002 --replication 127.0.0.1:7102 --data /var/tmp/n2
//!     cargo run -p kvnode -- --role follower --listen 127.0.0.1:7003 --replication 127.0.0.1:7103 --data /var/tmp/n3
//!     cargo run -p kvnode -- --role leader --listen 127.0.0.1:7001 --data /var/tmp/n1 \
//!         --follower http://127.0.0.1:7102 --follower http://127.0.0.1:7103

mod node;
mod rpc;

use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;

use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, Query, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_NONE_MATCH};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::{Parser, ValueEnum};
use kv::{Change, Condition};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, version_header, write_condition};
use repl::Replica;
use serde::{Deserialize, Serialize};
use tokio::net::TcpListener;
use tonic::transport::Server;

use crate::node::Node;
use crate::rpc::{Follower, ReplicationServer};

#[derive(Clone, Copy, PartialEq, Eq, ValueEnum)]
enum Role {
    Leader,
    Follower,
}

#[derive(Parser)]
struct Options {
    #[arg(long, value_enum)]
    role: Role,

    /// The address of the HTTP API.
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    /// The address on which a follower accepts records from the leader.
    #[arg(long, default_value = "127.0.0.1:7101")]
    replication: SocketAddr,

    /// The replication address of a follower, such as http://127.0.0.1:7102.
    /// Given once for each follower, to the leader.
    #[arg(long = "follower")]
    followers: Vec<String>,

    /// The term this node starts in. A new leader is given a higher term
    /// than the leader it replaces.
    #[arg(long, default_value_t = 1)]
    term: u64,

    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// Respond without waiting for the disk.
    #[arg(long)]
    no_sync: bool,
}

async fn put_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    let written = node
        .write(
            Change::Put {
                key,
                value: body.to_vec(),
            },
            condition,
        )
        .await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    node.write(Change::Delete { key }, condition)
        .await?;

    Ok(StatusCode::NO_CONTENT)
}

async fn get_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let cached = version_header(&headers, IF_NONE_MATCH)?;
    let entry = node.get(&key)?;
    let tag = [(ETAG, etag(entry.version))];

    if cached == Some(Condition::Version(entry.version)) {
        return Ok((StatusCode::NOT_MODIFIED, tag).into_response());
    }

    Ok((
        tag,
        [(CONTENT_TYPE, "application/octet-stream")],
        entry.value,
    )
        .into_response())
}

#[derive(Deserialize)]
struct ListQuery {
    #[serde(default)]
    prefix: String,
    after: Option<String>,
    limit: Option<usize>,
}

#[derive(Serialize)]
struct Item {
    key: String,
    version: u64,
    size: usize,
}

#[derive(Serialize)]
struct Page {
    keys: Vec<Item>,
    next_after: Option<String>,
}

async fn list_keys(
    State(node): State<Arc<Node>>,
    Query(query): Query<ListQuery>,
) -> Result<Json<Page>, ApiError> {
    let limit = query
        .limit
        .unwrap_or(100)
        .clamp(1, 1000);
    let (page, more) = node.list(&query.prefix, query.after.as_deref(), limit)?;
    let next_after = if more {
        page.last()
            .map(|last| last.key.clone())
    } else {
        None
    };
    let keys = page
        .into_iter()
        .map(|item| Item {
            key: item.key,
            version: item.version,
            size: item.size,
        })
        .collect();

    Ok(Json(Page { keys, next_after }))
}

#[derive(Serialize)]
struct Health {
    role: &'static str,
    term: u64,
    last_sequence: u64,
    committed: u64,
    /// The last sequence each follower has reported to this leader.
    followers: Vec<u64>,
}

async fn health(State(node): State<Arc<Node>>) -> Json<Health> {
    let replica = node.replica.lock();

    Json(Health {
        role: if node.leader { "leader" } else { "follower" },
        term: replica.term(),
        last_sequence: replica.last_sequence(),
        committed: replica.committed(),
        followers: node.reported(),
    })
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let replica = Replica::open(&options.data, !options.no_sync, options.term)?;

    println!(
        "term {}, log ends at {}",
        replica.term(),
        replica.last_sequence()
    );

    let node = Node::new(replica, options.role == Role::Leader, options.followers);

    match options.role {
        Role::Leader => node.lead()?,
        Role::Follower => {
            let service = ReplicationServer::new(Follower {
                node: Arc::clone(&node),
            });

            tokio::spawn(
                Server::builder()
                    .add_service(service)
                    .serve(options.replication),
            );
            println!("accepting records on {}", options.replication);
        }
    }

    let routes = Router::new()
        .route("/v1/keys", get(list_keys))
        .route(
            "/v1/keys/{key}",
            get(get_key)
                .put(put_key)
                .delete(delete_key),
        )
        .route("/healthz", get(health))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(node);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Further reading

• B. Oki and B. Liskov, “Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems”, PODC 1988.
• D. Ongaro and J. Ousterhout, “In Search of an Understandable Consensus Algorithm”, USENIX ATC 2014.
• R. van Renesse and F. Schneider, “Chain Replication for Supporting High Throughput and Availability”, OSDI 2004.

Key-Value Store: Leader Election (Low-Level Design)

In chapter 3 the leader of each shard was appointed by an operator, and when a leader was lost its shard could accept no writes until somebody promoted a follower by hand. Non-functional requirement 1 of chapter 1 allows 21.6 minutes of failed requests in a month, and no procedure that depends on a person noticing a failure, understanding it and typing commands can be relied upon to stay within such a limit. The members of a shard have to choose their own leader.

Stated so, the task sounds modest, and it is in fact the hardest problem in this book. The members must agree on a leader although they can communicate only by messages that may be lost, although any of them may crash and return at any moment with whatever it had on disk, and although none of them can tell a crashed member from one that is merely slow or cut off. The agreement must be such that 2 members are never leaders at once, since each would then accept writes of which the other knew nothing. And whoever is chosen must already hold every write that has ever been acknowledged, since a leader that lacked one would proceed to overwrite it. A group of machines reaching a decision under these conditions is said to reach consensus.

The algorithm used in this chapter is Raft, published by Diego Ongaro and John Ousterhout in 2014 with the stated aim of being understandable, which it is to a degree that its predecessors were not. Raft combines the replication of chapter 3 with a procedure for voting. We first describe the roles of a member and the rule by which it votes, and show why that rule guarantees that every new leader holds every acknowledged record. We then deal with a case that chapter 3 excluded, that of a member whose log differs from its leader’s. Raft is implemented as a library without any network code, tested by a simulation that crashes members and loses messages at random, and then run over gRPC as kvraft. Finally we measure for how long a shard of 3 nodes is unavailable when its leader is killed.

Overview

Every member of a shard runs the same program and is started in the same way, and none is configured as the leader. At any moment each member is in 1 of 3 roles.

Time is divided into terms, numbered from 1. Every term begins with an election and has at most 1 leader. When no candidate obtains a majority, the term has no leader, and another election follows in the next. The implementation keeps the invariants of chapter 3 and adds 3 more.

5. At most 1 member leads any given term.
6. An elected leader holds every record that was acknowledged in an earlier term.
7. A member removes a record from its log only if the record was not committed.

Invariant 5 holds because every member votes at most once in a term and a candidate needs the votes of a majority. Any 2 majorities have a member in common, and for 2 candidates to win the same term that member would have had to vote twice. Invariants 6 and 7 are established in the sections on voting and on the repair of logs.

The messages

Elections add a second call to the protocol of chapter 1, and 2 fields to the existing Append call.

Listing 4.1 The election service in the schema. proto/store.proto

// Sent by a candidate to every other member of the shard.
service Election {
  rpc RequestVote(VoteRequest) returns (VoteResponse);
}

message VoteRequest {
  uint64 term = 1;
  uint32 candidate = 2;
  // The end of the candidate's log, and the term of its last record.
  uint64 last_sequence = 3;
  uint64 last_term = 4;
}

message VoteResponse {
  uint64 term = 1;
  bool granted = 2;
}

Listing 4.2 The replication service, with fields 6 and 7 added. proto/store.proto

// Sent by the leader of a shard to each follower.
service Replication {
  // Appends records to the follower's log. A call with no records is a
  // heartbeat: it tells the follower that the leader is alive.
  rpc Append(AppendRequest) returns (AppendResponse);
}

message AppendRequest {
  // The shard, named by the last position it owns.
  fixed64 shard = 1;
  // The term of the sender. A follower rejects a leader of an older term.
  uint64 term = 2;
  // The sequence of the record that precedes `records` in the leader's log.
  uint64 previous_sequence = 3;
  repeated Record records = 4;
  // Every record up to this sequence is on a majority of the replicas.
  uint64 committed_sequence = 5;
  // Added for leader election. The term of the record at previous_sequence,
  // and the id of the sender.
  uint64 previous_term = 6;
  uint32 leader = 7;
}

message AppendResponse {
  uint64 term = 1;
  // False when the follower's log does not end at `previous_sequence`.
  bool appended = 2;
  // The sequence of the last record in the follower's log.
  uint64 last_sequence = 3;
}

AppendRequest has gained the fields previous_term and leader, under the unused numbers 6 and 7. This is the evolution of a schema as chapter 1 described it. The existing fields have kept their numbers, kvnode, the program of chapter 3, makes no use of the new ones, and a node built from the earlier schema passes over them when it decodes a message.

As in chapter 3, the library has message types of its own.

Listing 4.3 The imports of raft/src/lib.rs. raft/src/lib.rs

use std::collections::BTreeMap;
use std::fs::{self, File};
use std::io::{self, Write};
use std::path::{Path, PathBuf};

use kv::{Change, Condition, Entry, Log, Record, StoreError, apply, check};

Listing 4.4 The messages of the library. raft/src/lib.rs

pub type NodeId = u32;

/// Sent by a leader to a follower.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AppendRequest {
    pub term: u64,
    pub leader: NodeId,
    /// The sequence of the record that precedes `records`, and its term.
    pub previous_sequence: u64,
    pub previous_term: u64,
    pub records: Vec<Record>,
    pub committed_sequence: u64,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AppendAnswer {
    pub term: u64,
    pub from: NodeId,
    pub appended: bool,
    /// When appended: the last sequence that is known to match the leader's
    /// log. When refused: the sequence the leader should continue from.
    pub last_sequence: u64,
}

/// Sent by a candidate to every other member.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct VoteRequest {
    pub term: u64,
    pub candidate: NodeId,
    /// The end of the candidate's log, and the term of its last record.
    pub last_sequence: u64,
    pub last_term: u64,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct VoteAnswer {
    pub term: u64,
    pub from: NodeId,
    pub granted: bool,
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Message {
    Append(AppendRequest),
    Appended(AppendAnswer),
    Vote(VoteRequest),
    Voted(VoteAnswer),
}

/// A message and the member it is addressed to.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Outgoing {
    pub to: NodeId,
    pub message: Message,
}

The state of a member

Listing 4.5 The roles, the election timeout, and the state of a member. raft/src/lib.rs

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Role {
    Follower,
    Candidate,
    Leader,
}

/// The ticks a follower waits for a leader before it stands for election:
/// 10 to 19, different for each member and each term, so that 2 members
/// seldom stand at the same moment. A tick is 100 ms in the node.
fn election_timeout(id: NodeId, term: u64) -> u64 {
    10 + (u64::from(id) * 7 + term * 3) % 10
}

/// What the leader knows about one follower.
#[derive(Debug, Clone, Copy, Default)]
struct Progress {
    /// The sequence after which the next message to the follower begins.
    next: u64,
    /// The last sequence the follower has confirmed to match.
    matched: u64,
    /// The tick at which the follower last answered.
    heard: u64,
}

pub struct Member {
    id: NodeId,
    peers: Vec<NodeId>,
    directory: PathBuf,
    sync: bool,

    log: Log,
    /// Every record of the log. Record `n` is at index `n - 1`.
    records: Vec<Record>,
    /// The term in which each record was written.
    terms: Vec<u64>,
    entries: BTreeMap<String, Entry>,
    committed: u64,

    /// The current term, and the member this one voted for in it. Both are
    /// stored on disk before any answer that depends on them is sent.
    term: u64,
    voted_for: Option<NodeId>,

    role: Role,
    /// The leader of the current term, when this member knows it.
    leader: Option<NodeId>,
    votes: usize,
    progress: BTreeMap<NodeId, Progress>,
    /// Ticks since the member started, and the tick of the last message
    /// from a leader or the last vote granted.
    now: u64,
    quiet_since: u64,
    /// The records this member has removed from its log because they
    /// differed from a leader's.
    discarded: u64,
}

A member holds 3 kinds of state, which differ in whether they have to survive a restart, and the distinction is among the most important things to understand about the algorithm.

The second row is easily overlooked. It is natural to think of a vote as a passing event, of no interest once the election is over. But the whole guarantee of invariant 5 rests on each member voting once, and a member that crashed after voting and returned with no memory of having done so could give a second vote in the same term to a different candidate. A vote is therefore a promise, and it is written to disk before it is sent.

Member::open reads both files. From the log it obtains the records and, by means of the term markers described below, the term of each. From the state file it obtains the term and the vote. Everything else begins from fixed values: a member that has been opened is a follower, knows of no leader, and has applied nothing.

Listing 4.6 Opening a member. raft/src/lib.rs

impl Member {
    /// Opens the member in `directory`. It starts as a follower, with
    /// nothing applied, in the term it had reached before it stopped.
    pub fn open(
        directory: &Path,
        id: NodeId,
        peers: Vec<NodeId>,
        sync: bool,
    ) -> io::Result<Member> {
        let (log, recovery) = Log::open(directory, sync)?;
        let (term, voted_for) = read_state(directory)?;
        let terms = terms_of(&recovery.records, 0);

        Ok(Member {
            id,
            peers,
            directory: directory.to_owned(),
            sync,
            log,
            records: recovery.records,
            terms,
            entries: BTreeMap::new(),
            committed: 0,
            term,
            voted_for,
            role: Role::Follower,
            leader: None,
            votes: 0,
            progress: BTreeMap::new(),
            now: 0,
            quiet_since: 0,
            discarded: 0,
        })
    }

    pub fn id(&self) -> NodeId {
        self.id
    }

    pub fn role(&self) -> Role {
        self.role
    }

    pub fn term(&self) -> u64 {
        self.term
    }

    pub fn leader(&self) -> Option<NodeId> {
        self.leader
    }

    pub fn last_sequence(&self) -> u64 {
        self.records.len() as u64
    }

    pub fn committed(&self) -> u64 {
        self.committed
    }

    pub fn records(&self) -> &[Record] {
        &self.records
    }

    pub fn discarded(&self) -> u64 {
        self.discarded
    }

    fn last_term(&self) -> u64 {
        self.terms
            .last()
            .copied()
            .unwrap_or(0)
    }

    /// The term of record `sequence`, or 0 for the position before the log.
    fn term_at(&self, sequence: u64) -> u64 {
        if sequence == 0 {
            0
        } else {
            self.terms[sequence as usize - 1]
        }
    }

    fn majority(&self) -> usize {
        let members = self.peers.len() + 1;

        members / 2 + 1
    }
}

The state file is replaced by writing a new file and renaming it over the old one. On POSIX file systems, a rename within a directory is atomic, so that after a crash the file holds either the old state or the new and never a mixture. A member writes this file before it sends any message that depends on a new term or a new vote.

Listing 4.7 Reading and writing the term and the vote. raft/src/lib.rs

const STATE_FILE: &str = "state";

/// Reads the term and the vote. A member that has no state file has not
/// voted and is in term 0.
fn read_state(directory: &Path) -> io::Result<(u64, Option<NodeId>)> {
    let text = match fs::read_to_string(directory.join(STATE_FILE)) {
        Ok(text) => text,
        Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok((0, None)),
        Err(error) => return Err(error),
    };

    let mut fields = text
        .split_whitespace()
        .map(str::parse::<u64>);
    let invalid = || io::Error::new(io::ErrorKind::InvalidData, "the state file is malformed");
    let term = fields
        .next()
        .and_then(Result::ok)
        .ok_or_else(invalid)?;
    let voted_for = fields
        .next()
        .and_then(Result::ok)
        .map(|id| id as NodeId);

    Ok((term, voted_for))
}

impl Member {
    /// Stores the term and the vote. The new file replaces the old one in a
    /// single step, so a crash leaves one of the 2 and not a mixture.
    fn save_state(&self) -> io::Result<()> {
        let temporary = self.directory.join("state.new");
        let vote = self
            .voted_for
            .map_or(String::new(), |id| id.to_string());
        let mut file = File::create(&temporary)?;

        file.write_all(format!("{} {vote}\n", self.term).as_bytes())?;

        if self.sync {
            file.sync_all()?;
        }

        fs::rename(&temporary, self.directory.join(STATE_FILE))?;

        if self.sync {
            File::open(&self.directory)?.sync_all()?;
        }

        Ok(())
    }
}

The term of a record

Raft needs to know in which term each record was written. The record format of chapter 2 has no field for it, and the implementation records terms without altering that format, by means of a term marker: the first record that a new leader writes is a Put with an empty key and the leader’s term as its value. Clients cannot write an empty key, since the API requires at least 1 character, so that a marker cannot collide with any client’s data. The term of any record is then the term of the most recent marker at or before it.

Listing 4.8 Term markers, and the term of each record of a log. raft/src/lib.rs

/// The key of a term marker. A client cannot write the empty key, so the
/// log can use it for a record of its own: the first record a new leader
/// appends, whose value is the leader's term.
const MARKER: &str = "";

fn marker(term: u64) -> Change {
    Change::Put {
        key: MARKER.to_owned(),
        value: term.to_le_bytes().to_vec(),
    }
}

/// The term a record starts, if it is a term marker.
fn marked_term(record: &Record) -> Option<u64> {
    let Change::Put { key, value } = &record.change else {
        return None;
    };

    if key != MARKER {
        return None;
    }

    let term: [u8; 8] = value.as_slice().try_into().ok()?;

    Some(u64::from_le_bytes(term))
}

/// The term of each record: the term of the last marker at or before it.
fn terms_of(records: &[Record], mut term: u64) -> Vec<u64> {
    let mut terms = Vec::with_capacity(records.len());

    for record in records {
        term = marked_term(record).unwrap_or(term);
        terms.push(term);
    }

    terms
}

The markers have a second function, which is described under the commit rule.

Elections

The election timeout

A follower expects a message from its leader at least once in every tick of 100 ms. Having received none for the whole of its election timeout, it concludes that the leader has failed and stands for election.

If all followers used the same timeout, they would all stand at the same moment, each would vote for itself, and no candidate would obtain a majority. The timeouts are therefore made to differ. election_timeout, in the listing above, computes a value between 10 and 19 ticks from the id of the member and its term. Within a term, the members of a shard obtain different values, and each member obtains a different value in each term, so that 2 members which happen to coincide once do not coincide again. Raft as published uses random timeouts for this purpose. Timeouts that are computed make the simulation later in the chapter repeatable.

On reaching its timeout, a member calls stand. It increases its term, records a vote for itself, and stores both on disk before any request leaves. Every VoteRequest carries the end of the candidate’s log, which the voting rule below compares with the log of each voter.

Listing 4.9 Standing for election. raft/src/lib.rs

impl Member {
    /// Stands for election in the next term.
    fn stand(&mut self) -> io::Result<Vec<Outgoing>> {
        self.term += 1;
        self.voted_for = Some(self.id);
        self.save_state()?;

        self.role = Role::Candidate;
        self.leader = None;
        self.votes = 1;
        self.quiet_since = self.now;

        let request = VoteRequest {
            term: self.term,
            candidate: self.id,
            last_sequence: self.last_sequence(),
            last_term: self.last_term(),
        };

        Ok(self
            .peers
            .iter()
            .map(|&to| Outgoing {
                to,
                message: Message::Vote(request),
            })
            .collect())
    }
}

The voting rule

A member grants its vote to a candidate when 3 conditions are satisfied.

Listing 4.10 Granting a vote, and counting the votes received. raft/src/lib.rs

impl Member {
    /// A member grants its vote once in a term, and only to a candidate
    /// whose log is at least as complete as its own.
    fn on_vote(&mut self, request: VoteRequest) -> io::Result<VoteAnswer> {
        if request.term > self.term {
            self.step_down(request.term)?;
        }

        let complete =
            (request.last_term, request.last_sequence) >= (self.last_term(), self.last_sequence());
        let free = self
            .voted_for
            .is_none_or(|id| id == request.candidate);
        let granted = request.term == self.term && complete && free;

        if granted {
            self.voted_for = Some(request.candidate);
            self.save_state()?;
            self.quiet_since = self.now;
        }

        Ok(VoteAnswer {
            term: self.term,
            from: self.id,
            granted,
        })
    }

    fn on_voted(&mut self, answer: VoteAnswer) -> io::Result<Vec<Outgoing>> {
        if answer.term > self.term {
            self.step_down(answer.term)?;

            return Ok(Vec::new());
        }

        if self.role != Role::Candidate || answer.term != self.term || !answer.granted {
            return Ok(Vec::new());
        }

        self.votes += 1;

        if self.votes >= self.majority() {
            self.lead()
        } else {
            Ok(Vec::new())
        }
    }
}

The third condition establishes invariant 6, by the argument that chapter 1 gave in outline. An acknowledged record is stored on a majority of the members, and a candidate needs the votes of a majority. Since 2 majorities share at least 1 member, some voter holds the record. That voter grants its vote only to a candidate whose log ends at least as far as its own, and such a candidate holds the record as well. A candidate that lacks an acknowledged record therefore cannot collect a majority. The choice that an operator made in chapter 3, by reading 2 numbers and promoting the follower with the longer log, is here made by the members themselves, each refusing its vote to anyone who knows less than it does.

In on_vote, the 3 conditions are request.term == self.term, free and complete. Tuples in Rust are compared by their first elements, with the second consulted only in a tie, so that (last_term, last_sequence) >= (...) is precisely the comparison that the rule requires. A vote that is granted is written to disk before any answer is returned. on_voted counts the votes for a candidate, and the vote that completes a majority triggers lead.

Terms are compared before sequences because a longer log is not necessarily a more complete one. In figure 4.4, a member that had been isolated might have accumulated several uncommitted records of term 1, and its log would then be longer than that of member 2. A record of term 2 at the end of member 2’s log shows that member 2 has received records from a later leader, and its log takes precedence.

Becoming the leader

lead does 3 things. For every follower it records where to begin sending, namely at the end of its own log, as a first guess that a refusal will correct. It then appends the marker of its term, and finally sends its first messages.

Listing 4.11 Becoming the leader. raft/src/lib.rs

impl Member {
    /// Becomes the leader of the current term. The first record of the term
    /// is its marker. When a majority has stored the marker, every earlier
    /// record is committed with it.
    fn lead(&mut self) -> io::Result<Vec<Outgoing>> {
        self.role = Role::Leader;
        self.leader = Some(self.id);

        let next = self.last_sequence();
        self.progress = self
            .peers
            .iter()
            .map(|&peer| {
                (
                    peer,
                    Progress {
                        next,
                        matched: 0,
                        heard: self.now,
                    },
                )
            })
            .collect();

        let record = self.log.append(marker(self.term))?;
        self.records.push(record);
        self.terms.push(self.term);

        Ok(self.replicate())
    }
}

Returning to follower

Every message carries the term of its sender. On seeing, in any message, a term higher than its own, a member calls step_down: it adopts that term, clears its vote, since the vote belonged to an earlier term, and continues as a follower. For a leader, this is how the arrival of a successor makes itself known.

Listing 4.12 Adopting a higher term. raft/src/lib.rs

impl Member {
    /// Moves to a newer term as a follower. The vote of the old term does
    /// not carry over.
    fn step_down(&mut self, term: u64) -> io::Result<()> {
        if term > self.term {
            self.term = term;
            self.voted_for = None;
            self.save_state()?;
        }

        self.role = Role::Follower;
        self.leader = None;
        self.progress.clear();

        Ok(())
    }
}

Repairing a log that differs

In chapter 3 the log of a follower was always a prefix of its leader’s, and a former leader had to rejoin with an empty log. With elections, a member can hold records that its present leader lacks. This comes about when a leader appends a record and fails before sending it. The record exists on 1 member only and was never acknowledged, and the new leader writes a different record at the same sequence.

Append detects such a difference through the field previous_term. A follower appends records only if its own record at previous_sequence has the term previous_term. The check is sufficient on account of a single property: if 2 logs contain a record with the same sequence and the same term, the 2 logs are identical up to that record. The property holds because each term has 1 leader, a leader writes each sequence once, and the check is applied on every append, so that agreement at 1 position has always been built upon agreement at the position before.

On the follower’s side, the handling of Append consists of 3 checks followed by the repair itself.

The third check carries a hint for the leader. When its record at previous_sequence has a different term, the member answers with previous_sequence - 1, and the leader begins its next message 1 record earlier. By repeating this, the leader moves backwards through its log until it arrives at a record on which the 2 logs agree.

Listing 4.13 The checks of Append. raft/src/lib.rs

impl Member {
    fn on_append(&mut self, request: AppendRequest) -> io::Result<AppendAnswer> {
        let refuse = |member: &Member, last_sequence| AppendAnswer {
            term: member.term,
            from: member.id,
            appended: false,
            last_sequence,
        };

        if request.term < self.term {
            return Ok(refuse(self, self.last_sequence()));
        }

        // A message of the current term or a newer one comes from the leader.
        self.step_down(request.term)?;
        self.leader = Some(request.leader);
        self.quiet_since = self.now;

        if request.previous_sequence > self.last_sequence() {
            return Ok(refuse(self, self.last_sequence()));
        }

        if self.term_at(request.previous_sequence) != request.previous_term {
            return Ok(refuse(self, request.previous_sequence - 1));
        }

        let matched = request.previous_sequence + request.records.len() as u64;
        self.store(
            request.previous_sequence,
            request.previous_term,
            request.records,
        )?;
        self.commit(
            request
                .committed_sequence
                .min(matched),
        );

        Ok(AppendAnswer {
            term: self.term,
            from: self.id,
            appended: true,
            last_sequence: matched,
        })
    }
}

Once all the checks have passed, the 2 logs are known to be equal up to previous_sequence, and store makes the member’s log equal to the leader’s for every record that follows.

Listing 4.14 Skipping equal records, removing differing ones, and appending the rest. raft/src/lib.rs

impl Member {
    /// Makes the log equal to the leader's from `previous` onward: records
    /// the member already has are skipped, a record that differs is removed
    /// together with everything after it, and the rest is appended.
    fn store(&mut self, previous: u64, previous_term: u64, records: Vec<Record>) -> io::Result<()> {
        let terms = terms_of(&records, previous_term);
        let mut fresh = 0;

        for (index, record) in records.iter().enumerate() {
            let sequence = previous + index as u64 + 1;

            if sequence > self.last_sequence() {
                break;
            }

            if self.term_at(sequence) != terms[index]
                || &self.records[sequence as usize - 1] != record
            {
                self.truncate(sequence - 1)?;

                break;
            }

            fresh = index + 1;
        }

        if fresh < records.len() {
            self.log
                .append_records(&records[fresh..])?;
            self.records
                .extend_from_slice(&records[fresh..]);
            self.terms
                .extend_from_slice(&terms[fresh..]);
        }

        Ok(())
    }
}

For each incoming record, store computes a term from previous_term and from the markers among the incoming records, exactly as terms_of does for a whole log. 2 records count as equal only if both their terms and their contents agree.

Listing 4.15 Removing uncommitted records. raft/src/lib.rs

impl Member {
    /// Removes the records after `keep`. Only records that are not
    /// committed can differ from the leader's, so none of them is applied.
    fn truncate(&mut self, keep: u64) -> io::Result<()> {
        if keep < self.committed {
            return Err(io::Error::other(format!(
                "record {} is committed and differs from the leader's",
                keep + 1
            )));
        }

        let length: usize = self.records[..keep as usize]
            .iter()
            .map(|record| record.encode().len())
            .sum();

        self.discarded += self.last_sequence() - keep;
        self.log
            .truncate(keep, length as u64)?;
        self.records.truncate(keep as usize);
        self.terms.truncate(keep as usize);

        Ok(())
    }
}

truncate refuses to remove a committed record and returns an error. By invariant 6 the situation cannot arise, since a leader holds every committed record and its log cannot differ from a follower’s at a committed position. The check is there nonetheless, and for a good reason. Should the invariant ever be violated, through a defect in this code or a fault in a disk, the result without the check would be acknowledged data silently destroyed, and with it the member stops. A program that verifies what it believes to be impossible converts its own undiscovered defects from corruption into failures that somebody will notice.

In order to truncate its file, a member needs the byte offset at which the last record to be kept ends. It computes that offset by encoding the records that are kept and summing their lengths, which avoids storing an offset for every record.

The log of chapter 2 gains a method for removing records from its end.

Listing 4.16 The imports of kv/src/log.rs. kv/src/log.rs

use std::fs::{self, File, OpenOptions};
use std::io::{self, Read, Write};
use std::path::Path;

use crate::record::{Change, Decoded, Record};

Listing 4.17 Removing records from the end of the log. kv/src/log.rs

impl Log {
    /// Removes the records after record `keep`, which ends at byte `length`
    /// of the file. A follower uses this to discard records that were never
    /// committed and that differ from the leader's.
    pub fn truncate(&mut self, keep: u64, length: u64) -> io::Result<()> {
        self.file.set_len(length)?;

        if self.sync {
            self.file.sync_all()?;
        }

        self.next = keep + 1;

        Ok(())
    }
}

This method qualifies invariant 4 of chapter 2, which stated that the log is modified only by appending. Stated exactly, no committed record is ever modified. Records beyond the committed sequence may be removed, and invariant 7 of this chapter ensures that only such records are.

The leader’s view of a follower

For every follower, the leader keeps 2 sequences. next is where its next message will begin, and it moves backwards on a refusal. matched is the highest sequence that the follower has confirmed, and it only ever increases. The commit rule uses matched, since next is a guess. on_appended updates both from an answer, applies the commit rule, and returns a further message when the follower is still behind.

Listing 4.18 Processing the answer of a follower. raft/src/lib.rs

impl Member {
    fn on_appended(&mut self, answer: AppendAnswer) -> io::Result<Vec<Outgoing>> {
        if answer.term > self.term {
            self.step_down(answer.term)?;

            return Ok(Vec::new());
        }

        if self.role != Role::Leader || answer.term != self.term {
            return Ok(Vec::new());
        }

        let now = self.now;
        let Some(progress) = self.progress.get_mut(&answer.from) else {
            return Ok(Vec::new());
        };

        progress.heard = now;
        progress.next = answer.last_sequence;

        if answer.appended {
            progress.matched = progress
                .matched
                .max(answer.last_sequence);
        }

        let behind = !answer.appended || answer.last_sequence < self.last_sequence();
        self.advance();

        Ok(if behind {
            self.append_for(answer.from)
                .into_iter()
                .collect()
        } else {
            Vec::new()
        })
    }
}

The commit rule

Listing 4.19 Applying committed records, and the leader's commit rule. raft/src/lib.rs

impl Member {
    /// Applies the records up to `sequence` that are not applied yet. Term
    /// markers are records of the log and are not entries.
    fn commit(&mut self, sequence: u64) {
        let until = sequence.min(self.last_sequence());

        // A record that marks the start of a term changes no key.
        let changes: Vec<Record> = (self.committed..until)
            .map(|index| self.records[index as usize].clone())
            .filter(|record| marked_term(record).is_none())
            .collect();

        for record in changes {
            apply(&mut self.entries, record);
        }

        self.committed = self.committed.max(until);
    }

    /// The leader's commit rule. A record of an earlier term is not counted
    /// on its own: it becomes committed together with the first record of
    /// the leader's own term that a majority has stored.
    fn advance(&mut self) {
        let mut stored: Vec<u64> = self
            .progress
            .values()
            .map(|progress| progress.matched)
            .collect();
        stored.push(self.last_sequence());
        stored.sort_unstable_by(|a, b| b.cmp(a));

        let on_majority = stored[self.majority() - 1];

        if on_majority > self.committed && self.term_at(on_majority) == self.term {
            self.commit(on_majority);
        }
    }
}

The commit rule of chapter 3 committed a record as soon as a majority had stored it. advance adds a condition, that the record at the majority position must belong to the leader’s own term, and the reason for it is the subtlest point in Raft. A record of an earlier term can still be replaced even when a majority holds it, if a member that lacks it comes to be elected by members that lack it too, which remains possible for as long as no record of the current term is stored on a majority. Counting replicas is therefore safe only for records of the current term. Records of earlier terms are committed indirectly, at the moment when a record of the current term at a later sequence is committed.

It is here that the markers perform their second function. A new leader appends a marker as the first record of its term, and as soon as a majority has stored it, the marker and every record before it are committed together. Without markers, the records inherited from earlier terms would remain uncommitted until the first write of a client in the new term happened to arrive.

The clock and the messages

The library reads no clock and opens no connection. Its caller invokes tick at a fixed interval and handle for every incoming message, and sends whatever messages either method returns. As in chapter 3, this separation is what makes the algorithm testable, since a test can then supply the clock and the network itself.

Listing 4.20 The clock: heartbeats for a leader, the election timeout for the others. raft/src/lib.rs

impl Member {
    /// Advances the member's clock by one tick. A leader sends every
    /// follower a message. A follower or a candidate that has waited for
    /// its election timeout stands for election.
    pub fn tick(&mut self) -> io::Result<Vec<Outgoing>> {
        self.now += 1;

        if self.role == Role::Leader {
            return Ok(self.replicate());
        }

        if self.now - self.quiet_since >= election_timeout(self.id, self.term) {
            return self.stand();
        }

        Ok(Vec::new())
    }

    /// A message for every follower: the records it lacks, or a heartbeat.
    pub fn replicate(&self) -> Vec<Outgoing> {
        self.peers
            .iter()
            .filter_map(|&peer| self.append_for(peer))
            .collect()
    }

    fn append_for(&self, peer: NodeId) -> Option<Outgoing> {
        let from = self
            .progress
            .get(&peer)?
            .next
            .min(self.last_sequence());

        let request = AppendRequest {
            term: self.term,
            leader: self.id,
            previous_sequence: from,
            previous_term: self.term_at(from),
            records: self.records[from as usize..].to_vec(),
            committed_sequence: self.committed,
        };

        Some(Outgoing {
            to: peer,
            message: Message::Append(request),
        })
    }
}

Listing 4.21 Dispatching a message. raft/src/lib.rs

impl Member {
    /// Processes a message and returns the messages to send in response.
    pub fn handle(&mut self, message: Message) -> io::Result<Vec<Outgoing>> {
        match message {
            Message::Append(request) => {
                let to = request.leader;
                let answer = self.on_append(request)?;

                Ok(vec![Outgoing {
                    to,
                    message: Message::Appended(answer),
                }])
            }
            Message::Vote(request) => {
                let to = request.candidate;
                let answer = self.on_vote(request)?;

                Ok(vec![Outgoing {
                    to,
                    message: Message::Voted(answer),
                }])
            }
            Message::Appended(answer) => self.on_appended(answer),
            Message::Voted(answer) => self.on_voted(answer),
        }
    }
}

Listing 4.22 The operations for client requests. raft/src/lib.rs

impl Member {
    /// Whether this member is the leader and every record of its log is
    /// committed, so that it can accept a write.
    pub fn is_settled(&self) -> bool {
        self.role == Role::Leader && self.committed == self.last_sequence()
    }

    /// Checks the condition of a write and appends its record to the log.
    /// The caller sends the messages of `replicate` afterwards.
    pub fn propose(&mut self, change: Change, condition: Condition) -> Result<u64, StoreError> {
        let key = match &change {
            Change::Put { key, .. } | Change::Delete { key } => key,
        };
        let current = self
            .entries
            .get(key)
            .map(|entry| entry.version);

        if matches!(change, Change::Delete { .. }) && current.is_none() {
            return Err(StoreError::NotFound);
        }

        check(condition, current)?;

        let record = self.log.append(change)?;
        let sequence = record.sequence;
        self.records.push(record);
        self.terms.push(self.term);

        Ok(sequence)
    }

    /// Whether this member may answer a read: it is the leader, the marker
    /// of its term is committed, and a majority of the members, itself
    /// included, has answered within the last 10 ticks.
    pub fn can_read(&self) -> bool {
        let recent = self
            .progress
            .values()
            .filter(|progress| self.now - progress.heard < 10)
            .count();

        self.role == Role::Leader
            && self.term_at(self.committed) == self.term
            && recent + 1 >= self.majority()
    }

    pub fn get(&self, key: &str) -> Option<&Entry> {
        self.entries.get(key)
    }
}

can_read permits a read when the member is the leader, when the marker of its term is committed, and when a majority of the members has answered within the last 10 ticks. The second of these conditions ensures that the entries reflect every record that was committed in earlier terms.

Simulating crashes and lost messages

Because the library is free of clocks and connections, a test has control of all of them. raftsim runs 3 members in a single process. It decides when the clock of each member ticks, which message is delivered next, which messages are lost, and when a member crashes and restarts.

Listing 4.23 The imports of raftsim/src/main.rs. raftsim/src/main.rs

use std::collections::BTreeMap;
use std::env;
use std::error::Error;
use std::fs;
use std::path::{Path, PathBuf};
use std::process;

use kv::{Change, Condition};
use raft::{Member, NodeId, Outgoing, Role};

Listing 4.24 A repeatable source of pseudo-random numbers. raftsim/src/main.rs

/// A generator of pseudo-random numbers. The same seed gives the same
/// numbers, so a run that fails can be repeated exactly.
struct Random(u64);

impl Random {
    fn next(&mut self) -> u64 {
        self.0 ^= self.0 << 13;
        self.0 ^= self.0 >> 7;
        self.0 ^= self.0 << 17;

        self.0
    }

    /// A number below `bound`.
    fn below(&mut self, bound: usize) -> usize {
        (self.next() % bound as u64) as usize
    }
}

Listing 4.25 The cluster: members, messages in transit, and writes. raftsim/src/main.rs

const MEMBERS: [NodeId; 3] = [1, 2, 3];

/// A write that a leader has accepted and not yet committed.
struct Pending {
    leader: NodeId,
    term: u64,
    sequence: u64,
    key: String,
}

struct Cluster {
    root: PathBuf,
    /// The members that are running. A crashed member is absent.
    up: BTreeMap<NodeId, Member>,
    /// The messages that are on their way.
    network: Vec<Outgoing>,
    pending: Vec<Pending>,
    /// The sequence and the key of every acknowledged write.
    acknowledged: Vec<(u64, String)>,
    /// The leader that was seen in each term.
    leaders: BTreeMap<u64, NodeId>,
    discarded: u64,
    writes: u64,
}

impl Cluster {
    fn open(root: &Path) -> Result<Cluster, Box<dyn Error>> {
        let mut cluster = Cluster {
            root: root.to_owned(),
            up: BTreeMap::new(),
            network: Vec::new(),
            pending: Vec::new(),
            acknowledged: Vec::new(),
            leaders: BTreeMap::new(),
            discarded: 0,
            writes: 0,
        };

        for id in MEMBERS {
            cluster.start(id)?;
        }

        Ok(cluster)
    }

    fn start(&mut self, id: NodeId) -> Result<(), Box<dyn Error>> {
        let peers = MEMBERS
            .iter()
            .copied()
            .filter(|&peer| peer != id)
            .collect();
        let member = Member::open(
            &self
                .root
                .join(format!("member-{id}")),
            id,
            peers,
            false,
        )?;
        self.up.insert(id, member);

        Ok(())
    }

    /// Stops a member. Its files remain, and its memory is lost.
    fn crash(&mut self, id: NodeId) {
        if let Some(member) = self.up.remove(&id) {
            self.discarded += member.discarded();
        }
    }

    fn tick(&mut self, id: NodeId) -> Result<(), Box<dyn Error>> {
        if let Some(member) = self.up.get_mut(&id) {
            self.network.extend(member.tick()?);
        }

        self.check()
    }

    /// Delivers the message at `index`. A message for a member that is
    /// down is lost.
    fn deliver(&mut self, index: usize) -> Result<(), Box<dyn Error>> {
        let Outgoing { to, message } = self.network.remove(index);

        if let Some(member) = self.up.get_mut(&to) {
            self.network
                .extend(member.handle(message)?);
        }

        self.check()
    }

    /// Delivers one of the messages that are on their way, chosen at random.
    fn deliver_any(&mut self, random: &mut Random) -> Result<(), Box<dyn Error>> {
        if self.network.is_empty() {
            return Ok(());
        }

        self.deliver(random.below(self.network.len()))
    }

    /// Loses one of the messages that are on their way.
    fn lose_any(&mut self, random: &mut Random) {
        if !self.network.is_empty() {
            self.network
                .remove(random.below(self.network.len()));
        }
    }

    /// Starts a member that is down.
    fn restart(&mut self, id: NodeId) -> Result<(), Box<dyn Error>> {
        if self.up.contains_key(&id) {
            return Ok(());
        }

        self.start(id)
    }

    /// Delivers messages until none is on its way.
    fn deliver_all(&mut self) -> Result<(), Box<dyn Error>> {
        while !self.network.is_empty() {
            self.deliver(0)?;
        }

        Ok(())
    }

    /// Sends a write to a leader that can accept one, if there is one.
    fn write(&mut self) -> Result<bool, Box<dyn Error>> {
        let Some(leader) = self
            .up
            .values_mut()
            .find(|member| member.is_settled())
        else {
            return Ok(false);
        };

        self.writes += 1;
        let key = format!("key-{}", self.writes);
        let change = Change::Put {
            key: key.clone(),
            value: b"1".to_vec(),
        };
        let sequence = leader.propose(change, Condition::Always)?;

        self.pending.push(Pending {
            leader: leader.id(),
            term: leader.term(),
            sequence,
            key,
        });
        self.network
            .extend(leader.replicate());

        Ok(true)
    }

    fn leader(&self) -> Option<&Member> {
        self.up
            .values()
            .filter(|member| member.role() == Role::Leader)
            .max_by_key(|member| member.term())
    }
}

After every step, raftsim checks 3 properties.

Listing 4.26 The properties checked after every step. raftsim/src/main.rs

impl Cluster {
    /// The properties that must hold after every step.
    fn check(&mut self) -> Result<(), Box<dyn Error>> {
        // No 2 members lead the same term.
        for member in self
            .up
            .values()
            .filter(|member| member.role() == Role::Leader)
        {
            let first = *self
                .leaders
                .entry(member.term())
                .or_insert(member.id());

            if first != member.id() {
                return Err(format!(
                    "members {first} and {} both lead term {}",
                    member.id(),
                    member.term()
                )
                .into());
            }
        }

        // A write is acknowledged when the leader that accepted it, still
        // leading the same term, has committed its record.
        let up = &self.up;
        let committed = |write: &Pending| {
            up.get(&write.leader)
                .is_some_and(|leader| {
                    leader.role() == Role::Leader
                        && leader.term() == write.term
                        && leader.committed() >= write.sequence
                })
        };
        let (done, waiting): (Vec<Pending>, Vec<Pending>) = self
            .pending
            .drain(..)
            .partition(committed);

        self.acknowledged.extend(
            done.into_iter()
                .map(|write| (write.sequence, write.key)),
        );
        self.pending = waiting;

        // Every member that has committed the sequence of an acknowledged
        // write holds that write at that sequence.
        for (sequence, key) in &self.acknowledged {
            for member in self
                .up
                .values()
                .filter(|member| member.committed() >= *sequence)
            {
                let holds = matches!(&member.records()[*sequence as usize - 1].change, Change::Put { key: held, .. } if held == key);

                if !holds {
                    return Err(format!(
                        "member {} lost the acknowledged write of {key}",
                        member.id()
                    )
                    .into());
                }
            }
        }

        Ok(())
    }
}

A write counts as acknowledged under exactly the condition that the node itself applies: the member that accepted it has committed its record while still leading the same term.

Listing 4.27 A scripted election, a crashed leader, and the repair of its log. raftsim/src/main.rs

fn show(cluster: &Cluster) -> String {
    let describe = |id: &NodeId| match cluster.up.get(id) {
        Some(member) => format!(
            "{id}: {:?} term {} log {}",
            member.role(),
            member.term(),
            member.last_sequence()
        ),
        None => format!("{id}: down"),
    };

    MEMBERS
        .iter()
        .map(describe)
        .collect::<Vec<_>>()
        .join(" | ")
}

/// Ticks every running member, delivering all messages after each tick,
/// until a member leads. It returns the number of ticks.
fn elect(cluster: &mut Cluster) -> Result<u64, Box<dyn Error>> {
    let mut ticks = 0;

    while cluster
        .up
        .values()
        .all(|member| !member.is_settled())
    {
        ticks += 1;

        for id in MEMBERS {
            cluster.tick(id)?;
        }

        cluster.deliver_all()?;
    }

    Ok(ticks)
}

fn cases(root: &Path) -> Result<(), Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;

    println!("start:            {}", show(&cluster));

    let ticks = elect(&mut cluster)?;
    println!("after {ticks:2} ticks:   {}", show(&cluster));

    for _ in 0..3 {
        cluster.write()?;
        cluster.deliver_all()?;
    }

    println!(
        "after 3 writes:   {}, {} acknowledged",
        show(&cluster),
        cluster.acknowledged.len()
    );

    // The leader accepts a write and crashes before it sends the record.
    let old = cluster
        .leader()
        .map(Member::id)
        .ok_or("no leader")?;
    cluster.write()?;
    cluster.network.clear();
    cluster.crash(old);
    println!(
        "leader {old} appends record 5 and crashes: {}",
        show(&cluster)
    );

    let ticks = elect(&mut cluster)?;
    println!("after {ticks:2} ticks:   {}", show(&cluster));

    cluster.write()?;
    cluster.deliver_all()?;
    println!(
        "after 1 write:    {}, {} acknowledged",
        show(&cluster),
        cluster.acknowledged.len()
    );

    cluster.start(old)?;
    println!("member {old} returns: {}", show(&cluster));

    for _ in 0..2 {
        for id in MEMBERS {
            cluster.tick(id)?;
        }

        cluster.deliver_all()?;
    }

    let discarded = cluster
        .up
        .get(&old)
        .map_or(0, Member::discarded);
    println!("after 2 ticks:    {}", show(&cluster));
    println!(
        "member {old} discarded {discarded} record that was never committed, and holds the leader's records in its place"
    );

    Ok(())
}

Listing 4.28 200 random runs, and the recovery at the end of each. raftsim/src/main.rs

const RUNS: u64 = 200;
const STEPS: usize = 600;

/// One run: random ticks, deliveries, losses, crashes, restarts and writes.
fn run(root: &Path, seed: u64) -> Result<Cluster, Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;
    let mut random = Random(seed);

    for _ in 0..STEPS {
        let id = MEMBERS[random.below(MEMBERS.len())];

        match random.below(100) {
            0..40 => cluster.tick(id)?,
            40..82 => cluster.deliver_any(&mut random)?,
            82..87 => cluster.lose_any(&mut random),
            87..90 => cluster.crash(id),
            90..94 => cluster.restart(id)?,
            94.. => cluster.write().map(drop)?,
        }
    }

    recover(&mut cluster)?;

    Ok(cluster)
}

/// Ends the faults: every member runs and every message arrives. The
/// members must then agree on a leader and on a log that holds every
/// acknowledged write.
fn recover(cluster: &mut Cluster) -> Result<(), Box<dyn Error>> {
    for id in MEMBERS {
        if !cluster.up.contains_key(&id) {
            cluster.start(id)?;
        }
    }

    let agreed = |cluster: &Cluster| {
        cluster
            .leader()
            .is_some_and(|leader| {
                leader.is_settled()
                    && cluster
                        .up
                        .values()
                        .all(|member| member.records() == leader.records())
            })
    };

    for _ in 0..200 {
        if agreed(cluster) {
            break;
        }

        for id in MEMBERS {
            cluster.tick(id)?;
        }

        cluster.deliver_all()?;
    }

    if !agreed(cluster) {
        return Err("the members did not agree after the faults ended".into());
    }

    let leader = cluster
        .leader()
        .ok_or("the members have no leader after the faults ended")?;

    for (sequence, key) in &cluster.acknowledged {
        if leader
            .get(key)
            .is_none_or(|entry| entry.version != *sequence)
        {
            return Err(
                format!("the acknowledged write of {key} is missing after recovery").into(),
            );
        }
    }

    Ok(())
}

fn runs(root: &Path) -> Result<(), Box<dyn Error>> {
    let (mut leaders, mut acknowledged, mut discarded, mut most_terms) = (0, 0, 0, 0);

    for seed in 1..=RUNS {
        let directory = root.join(format!("run-{seed}"));
        let mut cluster = run(&directory, seed)?;

        for id in MEMBERS {
            cluster.crash(id);
        }

        leaders += cluster.leaders.len();
        acknowledged += cluster.acknowledged.len();
        discarded += cluster.discarded;
        most_terms = most_terms.max(
            cluster
                .leaders
                .keys()
                .next_back()
                .copied()
                .unwrap_or(0),
        );

        fs::remove_dir_all(&directory)?;
    }

    println!("{RUNS} runs of {STEPS} random steps:");
    println!("  {leaders} leaders were elected, and no term had 2 leaders");
    println!("  {acknowledged} writes were acknowledged, and every one was present after recovery");
    println!(
        "  {discarded} uncommitted records were discarded by members whose log differed from a leader's"
    );
    println!("  the highest term reached in a run was {most_terms}");

    Ok(())
}

Each run takes 600 random steps. A step ticks the clock of a member (40%), delivers a message (42%), loses a message (5%), crashes a member (3%), restarts a member (4%) or sends a write (6%). At the end of a run, all members are started, all messages are delivered, and raftsim verifies that the members agree on a leader and on a log, and that the leader returns every acknowledged write.

Listing 4.29 Running the simulation. raftsim/src/main.rs

fn main() -> Result<(), Box<dyn Error>> {
    let root = env::temp_dir().join(format!("raftsim-{}", process::id()));
    let result = cases(&root.join("cases")).and_then(|()| runs(&root));

    fs::remove_dir_all(&root)?;

    result
}

$ cargo run --release -p raftsim
start:            1: Follower term 0 log 0 | 2: Follower term 0 log 0 | 3: Follower term 0 log 0
after 11 ticks:   1: Follower term 1 log 1 | 2: Follower term 1 log 1 | 3: Leader term 1 log 1
after 3 writes:   1: Follower term 1 log 4 | 2: Follower term 1 log 4 | 3: Leader term 1 log 4, 3 acknowledged
leader 3 appends record 5 and crashes: 1: Follower term 1 log 4 | 2: Follower term 1 log 4 | 3: down
after 10 ticks:   1: Leader term 2 log 5 | 2: Follower term 2 log 5 | 3: down
after 1 write:    1: Leader term 2 log 6 | 2: Follower term 2 log 6 | 3: down, 4 acknowledged
member 3 returns: 1: Leader term 2 log 6 | 2: Follower term 2 log 6 | 3: Follower term 1 log 5
after 2 ticks:    1: Leader term 2 log 6 | 2: Follower term 2 log 6 | 3: Follower term 2 log 6
member 3 discarded 1 record that was never committed, and holds the leader's records in its place
200 runs of 600 random steps:
  480 leaders were elected, and no term had 2 leaders
  504 writes were acknowledged, and every one was present after recovery
  18 uncommitted records were discarded by members whose log differed from a leader's
  the highest term reached in a run was 8

Checking the simulation

A simulation that reports no failure is informative only if it would have reported one for an incorrect implementation, and this is a question that can be put to the test. We disabled 3 rules of Raft in turn and ran the simulation against each modified library.

The third result has to be stated with care. The correctness of Raft requires this condition, and the implementation contains it. With 3 members and 200 runs, however, the simulation did not produce a sequence of events in which the absence of the condition leads to an error. In the paper on Raft, such a sequence involves 5 members and several successive failures of leaders at particular moments. The simulation therefore provides evidence for the voting rules and for the repair of logs, and provides none for this condition. It would have been easy to report “200 runs passed” and to leave the reader with the impression that everything had been verified, and the difference between what a test exercises and what it merely fails to contradict is one that this book tries to keep in view.

Design overview

The node

kvraft connects the library to a clock, to a network and to an HTTP API. All members are started with the same options, apart from their ids and addresses.

Listing 4.30 The imports of kvraft/src/main.rs. kvraft/src/main.rs

use std::collections::BTreeMap;
use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;

use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_NONE_MATCH};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use kv::{Change, Condition};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, version_header, write_condition};
use raft::{Member, NodeId, Role};
use serde::Serialize;
use tokio::net::TcpListener;
use tonic::transport::Server;

use crate::node::Node;
use crate::rpc::{ElectionServer, ReplicationServer, Service};

Listing 4.31 The options of the command line. kvraft/src/main.rs

/// Reads `2=http://127.0.0.1:7102` as member 2 at that address.
fn parse_peer(text: &str) -> Result<(NodeId, String), String> {
    let (id, address) = text
        .split_once('=')
        .ok_or("a peer is written as id=address")?;
    let Ok(id) = id.parse() else {
        return Err("the id of a peer is a number".to_owned());
    };

    Ok((id, address.to_owned()))
}

#[derive(Parser)]
struct Options {
    /// The id of this member, a number that no other member has.
    #[arg(long)]
    id: NodeId,

    /// The address of the HTTP API.
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    /// The address on which this member accepts messages from the others.
    #[arg(long, default_value = "127.0.0.1:7101")]
    replication: SocketAddr,

    /// Another member and its replication address, as id=address.
    #[arg(long = "peer", value_parser = parse_peer)]
    peers: Vec<(NodeId, String)>,

    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// Respond without waiting for the disk.
    #[arg(long)]
    no_sync: bool,
}

Listing 4.32 The imports of kvraft/src/rpc.rs. kvraft/src/rpc.rs

use std::sync::Arc;

use kv::{Change, Record};
use raft::{AppendAnswer, AppendRequest, Message, NodeId, VoteAnswer, VoteRequest};
use tonic::{Request, Response, Status};

use crate::node::Node;

Listing 4.33 Conversion between the generated types and the library's messages. kvraft/src/rpc.rs

fn record_to_proto(record: &Record) -> proto::Record {
    let change = match record.change.clone() {
        Change::Put { key, value } => proto::record::Change::Put(proto::Put { key, value }),
        Change::Delete { key } => proto::record::Change::Delete(proto::Delete { key }),
    };

    proto::Record {
        sequence: record.sequence,
        change: Some(change),
    }
}

fn record_from_proto(record: proto::Record) -> Result<Record, Status> {
    let change = match record.change {
        Some(proto::record::Change::Put(put)) => Change::Put {
            key: put.key,
            value: put.value,
        },
        Some(proto::record::Change::Delete(delete)) => Change::Delete { key: delete.key },
        None => return Err(Status::invalid_argument("a record without a change")),
    };

    Ok(Record {
        sequence: record.sequence,
        change,
    })
}

impl From<AppendRequest> for proto::AppendRequest {
    fn from(request: AppendRequest) -> Self {
        proto::AppendRequest {
            shard: 0,
            term: request.term,
            previous_sequence: request.previous_sequence,
            records: request
                .records
                .iter()
                .map(record_to_proto)
                .collect(),
            committed_sequence: request.committed_sequence,
            previous_term: request.previous_term,
            leader: request.leader,
        }
    }
}

impl TryFrom<proto::AppendRequest> for AppendRequest {
    type Error = Status;

    fn try_from(request: proto::AppendRequest) -> Result<Self, Status> {
        Ok(AppendRequest {
            term: request.term,
            leader: request.leader,
            previous_sequence: request.previous_sequence,
            previous_term: request.previous_term,
            records: request
                .records
                .into_iter()
                .map(record_from_proto)
                .collect::<Result<_, _>>()?,
            committed_sequence: request.committed_sequence,
        })
    }
}

impl From<VoteRequest> for proto::VoteRequest {
    fn from(request: VoteRequest) -> Self {
        proto::VoteRequest {
            term: request.term,
            candidate: request.candidate,
            last_sequence: request.last_sequence,
            last_term: request.last_term,
        }
    }
}

/// The answer of member `from` to an `Append` call, as a message.
pub fn appended(from: NodeId, answer: proto::AppendResponse) -> Message {
    Message::Appended(AppendAnswer {
        term: answer.term,
        from,
        appended: answer.appended,
        last_sequence: answer.last_sequence,
    })
}

/// The answer of member `from` to a `RequestVote` call, as a message.
pub fn voted(from: NodeId, answer: proto::VoteResponse) -> Message {
    Message::Voted(VoteAnswer {
        term: answer.term,
        from,
        granted: answer.granted,
    })
}

Each gRPC call delivers 1 message to a member, and the member’s answer is returned as the response to the call.

Listing 4.34 The server side of Append and RequestVote. kvraft/src/rpc.rs

/// The server side of both services. Each call is one message for the
/// member, and the member's answer is the response of the call.
pub struct Service {
    pub node: Arc<Node>,
}

#[tonic::async_trait]
impl Replication for Service {
    async fn append(
        &self,
        request: Request<proto::AppendRequest>,
    ) -> Result<Response<proto::AppendResponse>, Status> {
        let request = AppendRequest::try_from(request.into_inner())?;

        let answer = self
            .node
            .answer(Message::Append(request))
            .map_err(|error| Status::unavailable(error.to_string()))?;

        let Message::Appended(answer) = answer else {
            return Err(Status::internal(
                "the member answered with another kind of message",
            ));
        };

        Ok(Response::new(proto::AppendResponse {
            term: answer.term,
            appended: answer.appended,
            last_sequence: answer.last_sequence,
        }))
    }
}

#[tonic::async_trait]
impl Election for Service {
    async fn request_vote(
        &self,
        request: Request<proto::VoteRequest>,
    ) -> Result<Response<proto::VoteResponse>, Status> {
        let request = request.into_inner();
        let request = VoteRequest {
            term: request.term,
            candidate: request.candidate,
            last_sequence: request.last_sequence,
            last_term: request.last_term,
        };

        let answer = self
            .node
            .answer(Message::Vote(request))
            .map_err(|error| Status::unavailable(error.to_string()))?;

        let Message::Voted(answer) = answer else {
            return Err(Status::internal(
                "the member answered with another kind of message",
            ));
        };

        Ok(Response::new(proto::VoteResponse {
            term: answer.term,
            granted: answer.granted,
        }))
    }
}

Around the member, a node consists of a small loop, which is shown in figure 4.elect-node-loop.

Listing 4.35 The imports of kvraft/src/node.rs. kvraft/src/node.rs

use std::collections::BTreeMap;
use std::io;
use std::sync::Arc;
use std::time::Duration;

use kv::{Change, Condition, Entry, Written};
use kvd::api::ApiError;
use parking_lot::Mutex;
use raft::{Member, Message, NodeId, Outgoing, Role};
use tokio::sync::watch;
use tokio::task;
use tokio::time::{interval, timeout};
use tonic::transport::{Channel, Endpoint};

use crate::rpc::{ElectionClient, ReplicationClient, appended, proto, voted};

Listing 4.36 The state of a node. kvraft/src/node.rs

/// The length of a tick of the member's clock.
pub const TICK: Duration = Duration::from_millis(100);

/// How long a member waits for the answer to one message.
const ANSWER: Duration = Duration::from_millis(500);

/// How long a write waits for its record to be committed.
const COMMIT: Duration = Duration::from_secs(1);

pub struct Node {
    pub member: Mutex<Member>,
    /// A connection to each of the other members.
    peers: BTreeMap<NodeId, Channel>,
    /// Held by the write that is in progress.
    turn: tokio::sync::Mutex<()>,
    /// The committed sequence, for writes that wait for it to advance.
    committed: watch::Sender<u64>,
}

impl Node {
    pub fn new(
        member: Member,
        addresses: BTreeMap<NodeId, String>,
    ) -> Result<Arc<Node>, tonic::transport::Error> {
        let mut peers = BTreeMap::new();

        for (id, address) in addresses {
            peers.insert(
                id,
                Endpoint::from_shared(address)?
                    .connect_timeout(ANSWER)
                    .connect_lazy(),
            );
        }

        Ok(Arc::new(Node {
            member: Mutex::new(member),
            peers,
            turn: tokio::sync::Mutex::new(()),
            committed: watch::Sender::new(0),
        }))
    }
}

with_member is the single point of entry to the member. It takes the lock, runs the operation it is given, and publishes the committed sequence afterwards, so that no operation can change that sequence without the waiting writes being informed.

Listing 4.37 Running an operation on the member. kvraft/src/node.rs

impl Node {
    /// Runs one operation on the member and publishes the committed
    /// sequence. The member waits for the disk, so the operation runs where
    /// blocking is allowed.
    fn with_member<T>(&self, operation: impl FnOnce(&mut Member) -> T) -> T {
        task::block_in_place(|| {
            let mut member = self.member.lock();
            let result = operation(&mut member);
            self.committed
                .send_replace(member.committed());

            result
        })
    }
}

dispatch starts 1 task for each message, and send performs a gRPC call with a time limit. Each response is converted into the corresponding answer message and handed to deliver, which passes it to the member and dispatches whatever the member returns. A call that fails or times out produces no message at all, which Raft tolerates without any special provision, since the next tick sends a new message to the same member.

Listing 4.38 Sending a message and delivering the answer. kvraft/src/node.rs

impl Node {
    /// Sends each message in a task of its own, so that a member that does
    /// not answer delays nothing else.
    fn dispatch(self: &Arc<Node>, messages: Vec<Outgoing>) {
        for message in messages {
            tokio::spawn(Arc::clone(self).send(message));
        }
    }

    /// Carries one request to a member and gives the answer to this member.
    async fn send(self: Arc<Node>, outgoing: Outgoing) {
        let Some(channel) = self
            .peers
            .get(&outgoing.to)
            .cloned()
        else {
            return;
        };

        let answer = match outgoing.message {
            Message::Append(request) => {
                let call = async move {
                    ReplicationClient::new(channel)
                        .append(proto::AppendRequest::from(request))
                        .await
                };

                timeout(ANSWER, call)
                    .await
                    .ok()
                    .and_then(Result::ok)
                    .map(|answer| appended(outgoing.to, answer.into_inner()))
            }
            Message::Vote(request) => {
                let call = async move {
                    ElectionClient::new(channel)
                        .request_vote(proto::VoteRequest::from(request))
                        .await
                };

                timeout(ANSWER, call)
                    .await
                    .ok()
                    .and_then(Result::ok)
                    .map(|answer| voted(outgoing.to, answer.into_inner()))
            }
            // An answer travels as the response of the call that asked.
            Message::Appended(..) | Message::Voted(..) => None,
        };

        if let Some(answer) = answer {
            self.deliver(answer);
        }
    }

    /// Gives the member a message and sends what the member returns.
    fn deliver(self: &Arc<Node>, message: Message) {
        match self.with_member(|member| member.handle(message)) {
            Ok(messages) => self.dispatch(messages),
            Err(error) => eprintln!("the member could not process a message: {error}"),
        }
    }
}

run_clock calls tick every 100 ms. answer serves the gRPC handlers: an incoming request is 1 message for the member, which returns exactly 1 answer to it.

Listing 4.39 The clock, and answering a request from another member. kvraft/src/node.rs

impl Node {
    /// The member's answer to a request that arrived over gRPC.
    pub fn answer(&self, request: Message) -> io::Result<Message> {
        let mut messages = self.with_member(|member| member.handle(request))?;

        messages
            .pop()
            .map(|outgoing| outgoing.message)
            .ok_or_else(|| io::Error::other("the member gave no answer"))
    }

    /// Advances the member's clock, for as long as the node runs.
    pub async fn run_clock(self: Arc<Node>) {
        let mut ticks = interval(TICK);

        loop {
            ticks.tick().await;

            match self.with_member(Member::tick) {
                Ok(messages) => self.dispatch(messages),
                Err(error) => eprintln!("the member could not advance its clock: {error}"),
            }
        }
    }
}

Listing 4.40 Writes and reads. kvraft/src/node.rs

impl Node {
    fn not_leader(member: &Member) -> ApiError {
        match member.leader() {
            Some(leader) => ApiError::Misdirected(format!("the leader is member {leader}")),
            None => ApiError::NoLeader,
        }
    }

    /// A write on the leader. It is acknowledged when its record is
    /// committed and this member has led the same term throughout.
    pub async fn write(
        self: &Arc<Node>,
        change: Change,
        condition: Condition,
    ) -> Result<Written, ApiError> {
        let turn = self.turn.lock().await;

        let last = {
            let member = self.member.lock();

            if member.role() != Role::Leader {
                return Err(Node::not_leader(&member));
            }

            member.last_sequence()
        };

        self.wait_for(last).await?;

        let (written, term, messages) = self.with_member(|member| {
            if !member.is_settled() {
                return Err(Node::not_leader(member));
            }

            let created = match &change {
                Change::Put { key, .. } | Change::Delete { key } => member.get(key).is_none(),
            };
            let version = member.propose(change, condition)?;

            Ok((
                Written { version, created },
                member.term(),
                member.replicate(),
            ))
        })?;

        self.dispatch(messages);
        self.wait_for(written.version)
            .await?;
        drop(turn);

        // A member that lost the leadership while it waited may hold another
        // leader's record at this sequence.
        let member = self.member.lock();

        if member.role() == Role::Leader && member.term() == term {
            Ok(written)
        } else {
            Err(ApiError::Unavailable(
                "the leadership changed before the record was committed".into(),
            ))
        }
    }

    async fn wait_for(&self, sequence: u64) -> Result<(), ApiError> {
        let mut committed = self.committed.subscribe();

        let reached = committed.wait_for(|&committed| committed >= sequence);
        let in_time = timeout(COMMIT, reached)
            .await
            .is_ok_and(|reached| reached.is_ok());

        if in_time {
            Ok(())
        } else {
            Err(ApiError::Unavailable(format!(
                "record {sequence} is not on a majority of the members"
            )))
        }
    }

    pub fn get(&self, key: &str) -> Result<Entry, ApiError> {
        let member = self.member.lock();

        if member.role() != Role::Leader {
            return Err(Node::not_leader(&member));
        }

        if !member.can_read() {
            return Err(ApiError::Unavailable(
                "this leader cannot confirm that it still leads".into(),
            ));
        }

        member
            .get(key)
            .cloned()
            .ok_or(ApiError::NotFound)
    }
}

The write path has the structure of chapter 3, with 1 addition after the wait. If a member loses its leadership while a write is waiting, its uncommitted record may be replaced by a record of the new leader at the same sequence, and the committed sequence may then pass that position without the write having been stored at all. A handler therefore acknowledges a write only if its member still leads the term in which the record was appended. Otherwise it answers 503, and the outcome of the write is unknown to its client, to be resolved as chapter 3 described.

Any member that is not the leader answers 421 Misdirected Request and names the leader it knows of in the detail field.

Listing 4.41 The HTTP handlers and the health of a node. kvraft/src/main.rs

async fn put_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    let written = node
        .write(
            Change::Put {
                key,
                value: body.to_vec(),
            },
            condition,
        )
        .await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    node.write(Change::Delete { key }, condition)
        .await?;

    Ok(StatusCode::NO_CONTENT)
}

async fn get_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let cached = version_header(&headers, IF_NONE_MATCH)?;
    let entry = node.get(&key)?;
    let tag = [(ETAG, etag(entry.version))];

    if cached == Some(Condition::Version(entry.version)) {
        return Ok((StatusCode::NOT_MODIFIED, tag).into_response());
    }

    Ok((
        tag,
        [(CONTENT_TYPE, "application/octet-stream")],
        entry.value,
    )
        .into_response())
}

#[derive(Serialize)]
struct Health {
    id: NodeId,
    role: &'static str,
    term: u64,
    /// The leader of the term, when this member knows it.
    leader: Option<NodeId>,
    last_sequence: u64,
    committed: u64,
}

async fn health(State(node): State<Arc<Node>>) -> Json<Health> {
    let member = node.member.lock();

    let role = match member.role() {
        Role::Leader => "leader",
        Role::Candidate => "candidate",
        Role::Follower => "follower",
    };

    Json(Health {
        id: member.id(),
        role,
        term: member.term(),
        leader: member.leader(),
        last_sequence: member.last_sequence(),
        committed: member.committed(),
    })
}

Listing 4.42 Starting a node. kvraft/src/main.rs

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let peers: BTreeMap<NodeId, String> = options.peers.into_iter().collect();
    let member = Member::open(
        &options.data,
        options.id,
        peers.keys().copied().collect(),
        !options.no_sync,
    )?;

    println!(
        "member {}, term {}, log ends at {}",
        member.id(),
        member.term(),
        member.last_sequence()
    );

    let node = Node::new(member, peers)?;

    let replication = ReplicationServer::new(Service {
        node: Arc::clone(&node),
    });
    let election = ElectionServer::new(Service {
        node: Arc::clone(&node),
    });

    tokio::spawn(
        Server::builder()
            .add_service(replication)
            .add_service(election)
            .serve(options.replication),
    );
    tokio::spawn(Arc::clone(&node).run_clock());

    let routes = Router::new()
        .route(
            "/v1/keys/{key}",
            get(get_key)
                .put(put_key)
                .delete(delete_key),
        )
        .route("/healthz", get(health))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(node);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Running 3 nodes

$ ./target/release/kvraft --id 1 --listen 127.0.0.1:7001 --replication 127.0.0.1:7101 --data /var/tmp/m1 \
      --peer 2=http://127.0.0.1:7102 --peer 3=http://127.0.0.1:7103 &
member 1, term 0, log ends at 0
listening on 127.0.0.1:7001
$ ./target/release/kvraft --id 2 --listen 127.0.0.1:7002 --replication 127.0.0.1:7102 --data /var/tmp/m2 \
      --peer 1=http://127.0.0.1:7101 --peer 3=http://127.0.0.1:7103 &
$ ./target/release/kvraft --id 3 --listen 127.0.0.1:7003 --replication 127.0.0.1:7103 --data /var/tmp/m3 \
      --peer 1=http://127.0.0.1:7101 --peer 2=http://127.0.0.1:7102 &
$ for port in 7001 7002 7003; do curl http://127.0.0.1:$port/healthz; echo; done
{"id":1,"role":"follower","term":1,"leader":3,"last_sequence":1,"committed":1}
{"id":2,"role":"follower","term":1,"leader":3,"last_sequence":1,"committed":1}
{"id":3,"role":"leader","term":1,"leader":3,"last_sequence":1,"committed":1}

All 3 members were started without any indication of who should lead. Within 2 seconds, member 3 had been elected in term 1. Every log contains 1 record, the marker of term 1.

$ curl -i -X PUT http://127.0.0.1:7003/v1/keys/user-1 --data-binary 'ada'
HTTP/1.1 201 Created
etag: "2"
content-length: 0

$ curl -i -X PUT http://127.0.0.1:7001/v1/keys/user-1 --data-binary 'x'
HTTP/1.1 421 Misdirected Request
content-type: application/problem+json
content-length: 116

{"type":"/problems/not-leader","title":"This node is not the leader","status":421,"detail":"the leader is member 3"}

The first write of a client receives version 2, since record 1 is the term marker. Versions remain strictly increasing, which is all that a client is entitled to rely upon.

Experiments

As in chapter 3, the 3 nodes run on 1 machine and share its disk.

The leader is lost

probe writes keys continuously for a fixed time. When a node refuses a write or fails to respond, probe sends the write to the next node. It records every acknowledged key, and the longest interval between 2 consecutive acknowledged writes.

Listing 4.43 The imports of probe/src/main.rs. probe/src/main.rs

use std::error::Error;
use std::fs::File;
use std::io::{BufWriter, Write};
use std::path::PathBuf;
use std::thread;
use std::time::{Duration, Instant};

use clap::Parser;
use ureq::Agent;

Listing 4.44 Measuring the longest pause between acknowledged writes. probe/src/main.rs

fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let agent: Agent = Agent::config_builder()
        .timeout_global(Some(Duration::from_secs(2)))
        .build()
        .into();
    let mut file = BufWriter::new(File::create(&options.acked)?);

    let started = Instant::now();
    let (mut server, mut number, mut refused) = (0, 1, 0);
    let (mut last_success, mut longest, mut longest_at) = (started, Duration::ZERO, Duration::ZERO);

    while started.elapsed() < Duration::from_secs(options.seconds) {
        let key = format!("key-{number:06}");
        let address = format!("{}/v1/keys/{key}", options.servers[server]);

        if agent
            .put(&address)
            .send(format!("value of {key}"))
            .is_err()
        {
            // This node did not accept the write. Try the next one.
            refused += 1;
            server = (server + 1) % options.servers.len();
            thread::sleep(Duration::from_millis(20));

            continue;
        }

        let pause = last_success.elapsed();

        if pause > longest {
            (longest, longest_at) = (pause, started.elapsed());
        }

        last_success = Instant::now();
        writeln!(file, "{key}")?;
        number += 1;
    }

    file.flush()?;

    println!(
        "{} writes acknowledged in {} seconds, {refused} attempts refused",
        number - 1,
        options.seconds
    );
    println!(
        "the longest pause between 2 acknowledged writes was {:.2} s, ending {:.1} s after the start",
        longest.as_secs_f64(),
        longest_at.as_secs_f64()
    );

    Ok(())
}

The leader is killed 6 seconds into a run of 20 seconds.

$ ./target/release/probe --seconds 20 --acked acked.txt \
      --server http://127.0.0.1:7001 --server http://127.0.0.1:7002 --server http://127.0.0.1:7003 &
$ sleep 6; kill -9 <pid of member 3, the leader>
$ sleep 3; for port in 7001 7002 7003; do curl http://127.0.0.1:$port/healthz; echo; done
{"id":1,"role":"leader","term":2,"leader":1,"last_sequence":610,"committed":609}
{"id":2,"role":"follower","term":2,"leader":1,"last_sequence":611,"committed":610}
curl: (7) Failed to connect to 127.0.0.1 port 7003
1443 writes acknowledged in 20 seconds, 51 attempts refused
the longest pause between 2 acknowledged writes was 1.05 s, ending 7.1 s after the start
$ ./target/release/kvctl verify --server http://127.0.0.1:7001 --acked acked.txt
1443 acknowledged, 1443 found, 0 missing

The measured pause can be set against the availability allowance of chapter 1. For the shards led by 1 node of 24, a pause of 1 second consumes about 0.04 seconds of a monthly allowance of 1296. The promotion by hand of chapter 3 would have consumed minutes.

The former leader returns

The member that was killed is started again with its data directory intact, which chapter 3 did not permit.

$ ./target/release/kvraft --id 3 --listen 127.0.0.1:7003 --replication 127.0.0.1:7103 --data /var/tmp/m3 \
      --peer 1=http://127.0.0.1:7101 --peer 2=http://127.0.0.1:7102 &
$ for port in 7001 7002 7003; do curl http://127.0.0.1:$port/healthz; echo; done
{"id":1,"role":"leader","term":2,"leader":1,"last_sequence":1447,"committed":1447}
{"id":2,"role":"follower","term":2,"leader":1,"last_sequence":1447,"committed":1447}
{"id":3,"role":"follower","term":2,"leader":1,"last_sequence":1447,"committed":1447}
$ cmp /var/tmp/m1/kv.log /var/tmp/m2/kv.log && cmp /var/tmp/m1/kv.log /var/tmp/m3/kv.log && echo identical
identical
$ cat /var/tmp/m1/state
2 1

It rejoined as a follower of term 2 and received the records it lacked. The 3 logs are identical, and the state file of member 1 contains 2 1: term 2, and a vote for member 1, which it gave to itself when it stood for election.

A member is left alone

The leader and 1 follower are killed, so that a single member remains.

$ kill -9 <pid of member 2, the leader> <pid of member 3>
$ sleep 5; curl http://127.0.0.1:7001/healthz
{"id":1,"role":"candidate","term":6,"leader":null,"last_sequence":1449,"committed":1449}
$ curl -X PUT http://127.0.0.1:7001/v1/keys/alone --data-binary 'x'
{"type":"/problems/store-unavailable","title":"The node cannot serve the request","status":503,"detail":"the shard has no leader at the moment"}
$ sleep 3; curl http://127.0.0.1:7001/healthz
{"id":1,"role":"candidate","term":8,"leader":null,"last_sequence":1449,"committed":1449}

$ (members 2 and 3 are started again)
$ sleep 4; for port in 7001 7002 7003; do curl http://127.0.0.1:$port/healthz; echo; done
{"id":1,"role":"leader","term":9,"leader":1,"last_sequence":1450,"committed":1450}
{"id":2,"role":"follower","term":9,"leader":1,"last_sequence":1450,"committed":1450}
{"id":3,"role":"follower","term":9,"leader":1,"last_sequence":1450,"committed":1450}

This is the minority side of a partition once more, seen from the inside. The lone member behaves exactly as it would if the other 2 were running and unreachable, as from its own position it must, since it has no means of telling the 2 situations apart.

Limitations

The complete programs

Listing 4.45 The complete raft/src/lib.rs. raft/src/lib.rs

//! A member of a shard that elects its own leader.
//!
//! A `Member` is a replica with a role. It follows the Raft algorithm: a
//! follower that hears nothing from a leader stands for election, a
//! candidate that receives the votes of a majority becomes the leader, and
//! a leader replicates its log. The type does no I/O on the network and
//! reads no clock. The caller delivers messages to `handle`, calls `tick`
//! at a fixed interval, and sends the messages that both return.

use std::collections::BTreeMap;
use std::fs::{self, File};
use std::io::{self, Write};
use std::path::{Path, PathBuf};

use kv::{Change, Condition, Entry, Log, Record, StoreError, apply, check};

pub type NodeId = u32;

/// Sent by a leader to a follower.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AppendRequest {
    pub term: u64,
    pub leader: NodeId,
    /// The sequence of the record that precedes `records`, and its term.
    pub previous_sequence: u64,
    pub previous_term: u64,
    pub records: Vec<Record>,
    pub committed_sequence: u64,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AppendAnswer {
    pub term: u64,
    pub from: NodeId,
    pub appended: bool,
    /// When appended: the last sequence that is known to match the leader's
    /// log. When refused: the sequence the leader should continue from.
    pub last_sequence: u64,
}

/// Sent by a candidate to every other member.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct VoteRequest {
    pub term: u64,
    pub candidate: NodeId,
    /// The end of the candidate's log, and the term of its last record.
    pub last_sequence: u64,
    pub last_term: u64,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct VoteAnswer {
    pub term: u64,
    pub from: NodeId,
    pub granted: bool,
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Message {
    Append(AppendRequest),
    Appended(AppendAnswer),
    Vote(VoteRequest),
    Voted(VoteAnswer),
}

/// A message and the member it is addressed to.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Outgoing {
    pub to: NodeId,
    pub message: Message,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Role {
    Follower,
    Candidate,
    Leader,
}

/// The ticks a follower waits for a leader before it stands for election:
/// 10 to 19, different for each member and each term, so that 2 members
/// seldom stand at the same moment. A tick is 100 ms in the node.
fn election_timeout(id: NodeId, term: u64) -> u64 {
    10 + (u64::from(id) * 7 + term * 3) % 10
}

/// What the leader knows about one follower.
#[derive(Debug, Clone, Copy, Default)]
struct Progress {
    /// The sequence after which the next message to the follower begins.
    next: u64,
    /// The last sequence the follower has confirmed to match.
    matched: u64,
    /// The tick at which the follower last answered.
    heard: u64,
}

pub struct Member {
    id: NodeId,
    peers: Vec<NodeId>,
    directory: PathBuf,
    sync: bool,

    log: Log,
    /// Every record of the log. Record `n` is at index `n - 1`.
    records: Vec<Record>,
    /// The term in which each record was written.
    terms: Vec<u64>,
    entries: BTreeMap<String, Entry>,
    committed: u64,

    /// The current term, and the member this one voted for in it. Both are
    /// stored on disk before any answer that depends on them is sent.
    term: u64,
    voted_for: Option<NodeId>,

    role: Role,
    /// The leader of the current term, when this member knows it.
    leader: Option<NodeId>,
    votes: usize,
    progress: BTreeMap<NodeId, Progress>,
    /// Ticks since the member started, and the tick of the last message
    /// from a leader or the last vote granted.
    now: u64,
    quiet_since: u64,
    /// The records this member has removed from its log because they
    /// differed from a leader's.
    discarded: u64,
}

/// The key of a term marker. A client cannot write the empty key, so the
/// log can use it for a record of its own: the first record a new leader
/// appends, whose value is the leader's term.
const MARKER: &str = "";

fn marker(term: u64) -> Change {
    Change::Put {
        key: MARKER.to_owned(),
        value: term.to_le_bytes().to_vec(),
    }
}

/// The term a record starts, if it is a term marker.
fn marked_term(record: &Record) -> Option<u64> {
    let Change::Put { key, value } = &record.change else {
        return None;
    };

    if key != MARKER {
        return None;
    }

    let term: [u8; 8] = value.as_slice().try_into().ok()?;

    Some(u64::from_le_bytes(term))
}

/// The term of each record: the term of the last marker at or before it.
fn terms_of(records: &[Record], mut term: u64) -> Vec<u64> {
    let mut terms = Vec::with_capacity(records.len());

    for record in records {
        term = marked_term(record).unwrap_or(term);
        terms.push(term);
    }

    terms
}

impl Member {
    /// Opens the member in `directory`. It starts as a follower, with
    /// nothing applied, in the term it had reached before it stopped.
    pub fn open(
        directory: &Path,
        id: NodeId,
        peers: Vec<NodeId>,
        sync: bool,
    ) -> io::Result<Member> {
        let (log, recovery) = Log::open(directory, sync)?;
        let (term, voted_for) = read_state(directory)?;
        let terms = terms_of(&recovery.records, 0);

        Ok(Member {
            id,
            peers,
            directory: directory.to_owned(),
            sync,
            log,
            records: recovery.records,
            terms,
            entries: BTreeMap::new(),
            committed: 0,
            term,
            voted_for,
            role: Role::Follower,
            leader: None,
            votes: 0,
            progress: BTreeMap::new(),
            now: 0,
            quiet_since: 0,
            discarded: 0,
        })
    }

    pub fn id(&self) -> NodeId {
        self.id
    }

    pub fn role(&self) -> Role {
        self.role
    }

    pub fn term(&self) -> u64 {
        self.term
    }

    pub fn leader(&self) -> Option<NodeId> {
        self.leader
    }

    pub fn last_sequence(&self) -> u64 {
        self.records.len() as u64
    }

    pub fn committed(&self) -> u64 {
        self.committed
    }

    pub fn records(&self) -> &[Record] {
        &self.records
    }

    pub fn discarded(&self) -> u64 {
        self.discarded
    }

    fn last_term(&self) -> u64 {
        self.terms
            .last()
            .copied()
            .unwrap_or(0)
    }

    /// The term of record `sequence`, or 0 for the position before the log.
    fn term_at(&self, sequence: u64) -> u64 {
        if sequence == 0 {
            0
        } else {
            self.terms[sequence as usize - 1]
        }
    }

    fn majority(&self) -> usize {
        let members = self.peers.len() + 1;

        members / 2 + 1
    }
}


const STATE_FILE: &str = "state";

/// Reads the term and the vote. A member that has no state file has not
/// voted and is in term 0.
fn read_state(directory: &Path) -> io::Result<(u64, Option<NodeId>)> {
    let text = match fs::read_to_string(directory.join(STATE_FILE)) {
        Ok(text) => text,
        Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok((0, None)),
        Err(error) => return Err(error),
    };

    let mut fields = text
        .split_whitespace()
        .map(str::parse::<u64>);
    let invalid = || io::Error::new(io::ErrorKind::InvalidData, "the state file is malformed");
    let term = fields
        .next()
        .and_then(Result::ok)
        .ok_or_else(invalid)?;
    let voted_for = fields
        .next()
        .and_then(Result::ok)
        .map(|id| id as NodeId);

    Ok((term, voted_for))
}

impl Member {
    /// Stores the term and the vote. The new file replaces the old one in a
    /// single step, so a crash leaves one of the 2 and not a mixture.
    fn save_state(&self) -> io::Result<()> {
        let temporary = self.directory.join("state.new");
        let vote = self
            .voted_for
            .map_or(String::new(), |id| id.to_string());
        let mut file = File::create(&temporary)?;

        file.write_all(format!("{} {vote}\n", self.term).as_bytes())?;

        if self.sync {
            file.sync_all()?;
        }

        fs::rename(&temporary, self.directory.join(STATE_FILE))?;

        if self.sync {
            File::open(&self.directory)?.sync_all()?;
        }

        Ok(())
    }
}

impl Member {
    /// Applies the records up to `sequence` that are not applied yet. Term
    /// markers are records of the log and are not entries.
    fn commit(&mut self, sequence: u64) {
        let until = sequence.min(self.last_sequence());

        // A record that marks the start of a term changes no key.
        let changes: Vec<Record> = (self.committed..until)
            .map(|index| self.records[index as usize].clone())
            .filter(|record| marked_term(record).is_none())
            .collect();

        for record in changes {
            apply(&mut self.entries, record);
        }

        self.committed = self.committed.max(until);
    }

    /// The leader's commit rule. A record of an earlier term is not counted
    /// on its own: it becomes committed together with the first record of
    /// the leader's own term that a majority has stored.
    fn advance(&mut self) {
        let mut stored: Vec<u64> = self
            .progress
            .values()
            .map(|progress| progress.matched)
            .collect();
        stored.push(self.last_sequence());
        stored.sort_unstable_by(|a, b| b.cmp(a));

        let on_majority = stored[self.majority() - 1];

        if on_majority > self.committed && self.term_at(on_majority) == self.term {
            self.commit(on_majority);
        }
    }
}

impl Member {
    /// Moves to a newer term as a follower. The vote of the old term does
    /// not carry over.
    fn step_down(&mut self, term: u64) -> io::Result<()> {
        if term > self.term {
            self.term = term;
            self.voted_for = None;
            self.save_state()?;
        }

        self.role = Role::Follower;
        self.leader = None;
        self.progress.clear();

        Ok(())
    }
}

impl Member {
    /// Stands for election in the next term.
    fn stand(&mut self) -> io::Result<Vec<Outgoing>> {
        self.term += 1;
        self.voted_for = Some(self.id);
        self.save_state()?;

        self.role = Role::Candidate;
        self.leader = None;
        self.votes = 1;
        self.quiet_since = self.now;

        let request = VoteRequest {
            term: self.term,
            candidate: self.id,
            last_sequence: self.last_sequence(),
            last_term: self.last_term(),
        };

        Ok(self
            .peers
            .iter()
            .map(|&to| Outgoing {
                to,
                message: Message::Vote(request),
            })
            .collect())
    }
}

impl Member {
    /// Becomes the leader of the current term. The first record of the term
    /// is its marker. When a majority has stored the marker, every earlier
    /// record is committed with it.
    fn lead(&mut self) -> io::Result<Vec<Outgoing>> {
        self.role = Role::Leader;
        self.leader = Some(self.id);

        let next = self.last_sequence();
        self.progress = self
            .peers
            .iter()
            .map(|&peer| {
                (
                    peer,
                    Progress {
                        next,
                        matched: 0,
                        heard: self.now,
                    },
                )
            })
            .collect();

        let record = self.log.append(marker(self.term))?;
        self.records.push(record);
        self.terms.push(self.term);

        Ok(self.replicate())
    }
}

impl Member {
    /// Advances the member's clock by one tick. A leader sends every
    /// follower a message. A follower or a candidate that has waited for
    /// its election timeout stands for election.
    pub fn tick(&mut self) -> io::Result<Vec<Outgoing>> {
        self.now += 1;

        if self.role == Role::Leader {
            return Ok(self.replicate());
        }

        if self.now - self.quiet_since >= election_timeout(self.id, self.term) {
            return self.stand();
        }

        Ok(Vec::new())
    }

    /// A message for every follower: the records it lacks, or a heartbeat.
    pub fn replicate(&self) -> Vec<Outgoing> {
        self.peers
            .iter()
            .filter_map(|&peer| self.append_for(peer))
            .collect()
    }

    fn append_for(&self, peer: NodeId) -> Option<Outgoing> {
        let from = self
            .progress
            .get(&peer)?
            .next
            .min(self.last_sequence());

        let request = AppendRequest {
            term: self.term,
            leader: self.id,
            previous_sequence: from,
            previous_term: self.term_at(from),
            records: self.records[from as usize..].to_vec(),
            committed_sequence: self.committed,
        };

        Some(Outgoing {
            to: peer,
            message: Message::Append(request),
        })
    }
}

impl Member {
    /// Processes a message and returns the messages to send in response.
    pub fn handle(&mut self, message: Message) -> io::Result<Vec<Outgoing>> {
        match message {
            Message::Append(request) => {
                let to = request.leader;
                let answer = self.on_append(request)?;

                Ok(vec![Outgoing {
                    to,
                    message: Message::Appended(answer),
                }])
            }
            Message::Vote(request) => {
                let to = request.candidate;
                let answer = self.on_vote(request)?;

                Ok(vec![Outgoing {
                    to,
                    message: Message::Voted(answer),
                }])
            }
            Message::Appended(answer) => self.on_appended(answer),
            Message::Voted(answer) => self.on_voted(answer),
        }
    }
}

impl Member {
    /// A member grants its vote once in a term, and only to a candidate
    /// whose log is at least as complete as its own.
    fn on_vote(&mut self, request: VoteRequest) -> io::Result<VoteAnswer> {
        if request.term > self.term {
            self.step_down(request.term)?;
        }

        let complete =
            (request.last_term, request.last_sequence) >= (self.last_term(), self.last_sequence());
        let free = self
            .voted_for
            .is_none_or(|id| id == request.candidate);
        let granted = request.term == self.term && complete && free;

        if granted {
            self.voted_for = Some(request.candidate);
            self.save_state()?;
            self.quiet_since = self.now;
        }

        Ok(VoteAnswer {
            term: self.term,
            from: self.id,
            granted,
        })
    }

    fn on_voted(&mut self, answer: VoteAnswer) -> io::Result<Vec<Outgoing>> {
        if answer.term > self.term {
            self.step_down(answer.term)?;

            return Ok(Vec::new());
        }

        if self.role != Role::Candidate || answer.term != self.term || !answer.granted {
            return Ok(Vec::new());
        }

        self.votes += 1;

        if self.votes >= self.majority() {
            self.lead()
        } else {
            Ok(Vec::new())
        }
    }
}

impl Member {
    fn on_append(&mut self, request: AppendRequest) -> io::Result<AppendAnswer> {
        let refuse = |member: &Member, last_sequence| AppendAnswer {
            term: member.term,
            from: member.id,
            appended: false,
            last_sequence,
        };

        if request.term < self.term {
            return Ok(refuse(self, self.last_sequence()));
        }

        // A message of the current term or a newer one comes from the leader.
        self.step_down(request.term)?;
        self.leader = Some(request.leader);
        self.quiet_since = self.now;

        if request.previous_sequence > self.last_sequence() {
            return Ok(refuse(self, self.last_sequence()));
        }

        if self.term_at(request.previous_sequence) != request.previous_term {
            return Ok(refuse(self, request.previous_sequence - 1));
        }

        let matched = request.previous_sequence + request.records.len() as u64;
        self.store(
            request.previous_sequence,
            request.previous_term,
            request.records,
        )?;
        self.commit(
            request
                .committed_sequence
                .min(matched),
        );

        Ok(AppendAnswer {
            term: self.term,
            from: self.id,
            appended: true,
            last_sequence: matched,
        })
    }
}

impl Member {
    /// Makes the log equal to the leader's from `previous` onward: records
    /// the member already has are skipped, a record that differs is removed
    /// together with everything after it, and the rest is appended.
    fn store(&mut self, previous: u64, previous_term: u64, records: Vec<Record>) -> io::Result<()> {
        let terms = terms_of(&records, previous_term);
        let mut fresh = 0;

        for (index, record) in records.iter().enumerate() {
            let sequence = previous + index as u64 + 1;

            if sequence > self.last_sequence() {
                break;
            }

            if self.term_at(sequence) != terms[index]
                || &self.records[sequence as usize - 1] != record
            {
                self.truncate(sequence - 1)?;

                break;
            }

            fresh = index + 1;
        }

        if fresh < records.len() {
            self.log
                .append_records(&records[fresh..])?;
            self.records
                .extend_from_slice(&records[fresh..]);
            self.terms
                .extend_from_slice(&terms[fresh..]);
        }

        Ok(())
    }
}

impl Member {
    /// Removes the records after `keep`. Only records that are not
    /// committed can differ from the leader's, so none of them is applied.
    fn truncate(&mut self, keep: u64) -> io::Result<()> {
        if keep < self.committed {
            return Err(io::Error::other(format!(
                "record {} is committed and differs from the leader's",
                keep + 1
            )));
        }

        let length: usize = self.records[..keep as usize]
            .iter()
            .map(|record| record.encode().len())
            .sum();

        self.discarded += self.last_sequence() - keep;
        self.log
            .truncate(keep, length as u64)?;
        self.records.truncate(keep as usize);
        self.terms.truncate(keep as usize);

        Ok(())
    }
}

impl Member {
    fn on_appended(&mut self, answer: AppendAnswer) -> io::Result<Vec<Outgoing>> {
        if answer.term > self.term {
            self.step_down(answer.term)?;

            return Ok(Vec::new());
        }

        if self.role != Role::Leader || answer.term != self.term {
            return Ok(Vec::new());
        }

        let now = self.now;
        let Some(progress) = self.progress.get_mut(&answer.from) else {
            return Ok(Vec::new());
        };

        progress.heard = now;
        progress.next = answer.last_sequence;

        if answer.appended {
            progress.matched = progress
                .matched
                .max(answer.last_sequence);
        }

        let behind = !answer.appended || answer.last_sequence < self.last_sequence();
        self.advance();

        Ok(if behind {
            self.append_for(answer.from)
                .into_iter()
                .collect()
        } else {
            Vec::new()
        })
    }
}

impl Member {
    /// Whether this member is the leader and every record of its log is
    /// committed, so that it can accept a write.
    pub fn is_settled(&self) -> bool {
        self.role == Role::Leader && self.committed == self.last_sequence()
    }

    /// Checks the condition of a write and appends its record to the log.
    /// The caller sends the messages of `replicate` afterwards.
    pub fn propose(&mut self, change: Change, condition: Condition) -> Result<u64, StoreError> {
        let key = match &change {
            Change::Put { key, .. } | Change::Delete { key } => key,
        };
        let current = self
            .entries
            .get(key)
            .map(|entry| entry.version);

        if matches!(change, Change::Delete { .. }) && current.is_none() {
            return Err(StoreError::NotFound);
        }

        check(condition, current)?;

        let record = self.log.append(change)?;
        let sequence = record.sequence;
        self.records.push(record);
        self.terms.push(self.term);

        Ok(sequence)
    }

    /// Whether this member may answer a read: it is the leader, the marker
    /// of its term is committed, and a majority of the members, itself
    /// included, has answered within the last 10 ticks.
    pub fn can_read(&self) -> bool {
        let recent = self
            .progress
            .values()
            .filter(|progress| self.now - progress.heard < 10)
            .count();

        self.role == Role::Leader
            && self.term_at(self.committed) == self.term
            && recent + 1 >= self.majority()
    }

    pub fn get(&self, key: &str) -> Option<&Entry> {
        self.entries.get(key)
    }
}

Listing 4.46 The complete raftsim/src/main.rs. raftsim/src/main.rs

//! Command raftsim runs 3 members in one process and decides itself when
//! each clock ticks, which message arrives, which is lost, and which member
//! crashes. It first walks through an election and the repair of a log,
//! and then makes 200 runs of 600 random steps, checking after every step
//! that no 2 members lead the same term and that no acknowledged write is
//! missing.
//!
//! Run it from code/ with:
//!
//!     cargo run --release -p raftsim

use std::collections::BTreeMap;
use std::env;
use std::error::Error;
use std::fs;
use std::path::{Path, PathBuf};
use std::process;

use kv::{Change, Condition};
use raft::{Member, NodeId, Outgoing, Role};

/// A generator of pseudo-random numbers. The same seed gives the same
/// numbers, so a run that fails can be repeated exactly.
struct Random(u64);

impl Random {
    fn next(&mut self) -> u64 {
        self.0 ^= self.0 << 13;
        self.0 ^= self.0 >> 7;
        self.0 ^= self.0 << 17;

        self.0
    }

    /// A number below `bound`.
    fn below(&mut self, bound: usize) -> usize {
        (self.next() % bound as u64) as usize
    }
}

const MEMBERS: [NodeId; 3] = [1, 2, 3];

/// A write that a leader has accepted and not yet committed.
struct Pending {
    leader: NodeId,
    term: u64,
    sequence: u64,
    key: String,
}

struct Cluster {
    root: PathBuf,
    /// The members that are running. A crashed member is absent.
    up: BTreeMap<NodeId, Member>,
    /// The messages that are on their way.
    network: Vec<Outgoing>,
    pending: Vec<Pending>,
    /// The sequence and the key of every acknowledged write.
    acknowledged: Vec<(u64, String)>,
    /// The leader that was seen in each term.
    leaders: BTreeMap<u64, NodeId>,
    discarded: u64,
    writes: u64,
}

impl Cluster {
    fn open(root: &Path) -> Result<Cluster, Box<dyn Error>> {
        let mut cluster = Cluster {
            root: root.to_owned(),
            up: BTreeMap::new(),
            network: Vec::new(),
            pending: Vec::new(),
            acknowledged: Vec::new(),
            leaders: BTreeMap::new(),
            discarded: 0,
            writes: 0,
        };

        for id in MEMBERS {
            cluster.start(id)?;
        }

        Ok(cluster)
    }

    fn start(&mut self, id: NodeId) -> Result<(), Box<dyn Error>> {
        let peers = MEMBERS
            .iter()
            .copied()
            .filter(|&peer| peer != id)
            .collect();
        let member = Member::open(
            &self
                .root
                .join(format!("member-{id}")),
            id,
            peers,
            false,
        )?;
        self.up.insert(id, member);

        Ok(())
    }

    /// Stops a member. Its files remain, and its memory is lost.
    fn crash(&mut self, id: NodeId) {
        if let Some(member) = self.up.remove(&id) {
            self.discarded += member.discarded();
        }
    }

    fn tick(&mut self, id: NodeId) -> Result<(), Box<dyn Error>> {
        if let Some(member) = self.up.get_mut(&id) {
            self.network.extend(member.tick()?);
        }

        self.check()
    }

    /// Delivers the message at `index`. A message for a member that is
    /// down is lost.
    fn deliver(&mut self, index: usize) -> Result<(), Box<dyn Error>> {
        let Outgoing { to, message } = self.network.remove(index);

        if let Some(member) = self.up.get_mut(&to) {
            self.network
                .extend(member.handle(message)?);
        }

        self.check()
    }

    /// Delivers one of the messages that are on their way, chosen at random.
    fn deliver_any(&mut self, random: &mut Random) -> Result<(), Box<dyn Error>> {
        if self.network.is_empty() {
            return Ok(());
        }

        self.deliver(random.below(self.network.len()))
    }

    /// Loses one of the messages that are on their way.
    fn lose_any(&mut self, random: &mut Random) {
        if !self.network.is_empty() {
            self.network
                .remove(random.below(self.network.len()));
        }
    }

    /// Starts a member that is down.
    fn restart(&mut self, id: NodeId) -> Result<(), Box<dyn Error>> {
        if self.up.contains_key(&id) {
            return Ok(());
        }

        self.start(id)
    }

    /// Delivers messages until none is on its way.
    fn deliver_all(&mut self) -> Result<(), Box<dyn Error>> {
        while !self.network.is_empty() {
            self.deliver(0)?;
        }

        Ok(())
    }

    /// Sends a write to a leader that can accept one, if there is one.
    fn write(&mut self) -> Result<bool, Box<dyn Error>> {
        let Some(leader) = self
            .up
            .values_mut()
            .find(|member| member.is_settled())
        else {
            return Ok(false);
        };

        self.writes += 1;
        let key = format!("key-{}", self.writes);
        let change = Change::Put {
            key: key.clone(),
            value: b"1".to_vec(),
        };
        let sequence = leader.propose(change, Condition::Always)?;

        self.pending.push(Pending {
            leader: leader.id(),
            term: leader.term(),
            sequence,
            key,
        });
        self.network
            .extend(leader.replicate());

        Ok(true)
    }

    fn leader(&self) -> Option<&Member> {
        self.up
            .values()
            .filter(|member| member.role() == Role::Leader)
            .max_by_key(|member| member.term())
    }
}

impl Cluster {
    /// The properties that must hold after every step.
    fn check(&mut self) -> Result<(), Box<dyn Error>> {
        // No 2 members lead the same term.
        for member in self
            .up
            .values()
            .filter(|member| member.role() == Role::Leader)
        {
            let first = *self
                .leaders
                .entry(member.term())
                .or_insert(member.id());

            if first != member.id() {
                return Err(format!(
                    "members {first} and {} both lead term {}",
                    member.id(),
                    member.term()
                )
                .into());
            }
        }

        // A write is acknowledged when the leader that accepted it, still
        // leading the same term, has committed its record.
        let up = &self.up;
        let committed = |write: &Pending| {
            up.get(&write.leader)
                .is_some_and(|leader| {
                    leader.role() == Role::Leader
                        && leader.term() == write.term
                        && leader.committed() >= write.sequence
                })
        };
        let (done, waiting): (Vec<Pending>, Vec<Pending>) = self
            .pending
            .drain(..)
            .partition(committed);

        self.acknowledged.extend(
            done.into_iter()
                .map(|write| (write.sequence, write.key)),
        );
        self.pending = waiting;

        // Every member that has committed the sequence of an acknowledged
        // write holds that write at that sequence.
        for (sequence, key) in &self.acknowledged {
            for member in self
                .up
                .values()
                .filter(|member| member.committed() >= *sequence)
            {
                let holds = matches!(&member.records()[*sequence as usize - 1].change, Change::Put { key: held, .. } if held == key);

                if !holds {
                    return Err(format!(
                        "member {} lost the acknowledged write of {key}",
                        member.id()
                    )
                    .into());
                }
            }
        }

        Ok(())
    }
}

fn show(cluster: &Cluster) -> String {
    let describe = |id: &NodeId| match cluster.up.get(id) {
        Some(member) => format!(
            "{id}: {:?} term {} log {}",
            member.role(),
            member.term(),
            member.last_sequence()
        ),
        None => format!("{id}: down"),
    };

    MEMBERS
        .iter()
        .map(describe)
        .collect::<Vec<_>>()
        .join(" | ")
}

/// Ticks every running member, delivering all messages after each tick,
/// until a member leads. It returns the number of ticks.
fn elect(cluster: &mut Cluster) -> Result<u64, Box<dyn Error>> {
    let mut ticks = 0;

    while cluster
        .up
        .values()
        .all(|member| !member.is_settled())
    {
        ticks += 1;

        for id in MEMBERS {
            cluster.tick(id)?;
        }

        cluster.deliver_all()?;
    }

    Ok(ticks)
}

fn cases(root: &Path) -> Result<(), Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;

    println!("start:            {}", show(&cluster));

    let ticks = elect(&mut cluster)?;
    println!("after {ticks:2} ticks:   {}", show(&cluster));

    for _ in 0..3 {
        cluster.write()?;
        cluster.deliver_all()?;
    }

    println!(
        "after 3 writes:   {}, {} acknowledged",
        show(&cluster),
        cluster.acknowledged.len()
    );

    // The leader accepts a write and crashes before it sends the record.
    let old = cluster
        .leader()
        .map(Member::id)
        .ok_or("no leader")?;
    cluster.write()?;
    cluster.network.clear();
    cluster.crash(old);
    println!(
        "leader {old} appends record 5 and crashes: {}",
        show(&cluster)
    );

    let ticks = elect(&mut cluster)?;
    println!("after {ticks:2} ticks:   {}", show(&cluster));

    cluster.write()?;
    cluster.deliver_all()?;
    println!(
        "after 1 write:    {}, {} acknowledged",
        show(&cluster),
        cluster.acknowledged.len()
    );

    cluster.start(old)?;
    println!("member {old} returns: {}", show(&cluster));

    for _ in 0..2 {
        for id in MEMBERS {
            cluster.tick(id)?;
        }

        cluster.deliver_all()?;
    }

    let discarded = cluster
        .up
        .get(&old)
        .map_or(0, Member::discarded);
    println!("after 2 ticks:    {}", show(&cluster));
    println!(
        "member {old} discarded {discarded} record that was never committed, and holds the leader's records in its place"
    );

    Ok(())
}

const RUNS: u64 = 200;
const STEPS: usize = 600;

/// One run: random ticks, deliveries, losses, crashes, restarts and writes.
fn run(root: &Path, seed: u64) -> Result<Cluster, Box<dyn Error>> {
    let mut cluster = Cluster::open(root)?;
    let mut random = Random(seed);

    for _ in 0..STEPS {
        let id = MEMBERS[random.below(MEMBERS.len())];

        match random.below(100) {
            0..40 => cluster.tick(id)?,
            40..82 => cluster.deliver_any(&mut random)?,
            82..87 => cluster.lose_any(&mut random),
            87..90 => cluster.crash(id),
            90..94 => cluster.restart(id)?,
            94.. => cluster.write().map(drop)?,
        }
    }

    recover(&mut cluster)?;

    Ok(cluster)
}

/// Ends the faults: every member runs and every message arrives. The
/// members must then agree on a leader and on a log that holds every
/// acknowledged write.
fn recover(cluster: &mut Cluster) -> Result<(), Box<dyn Error>> {
    for id in MEMBERS {
        if !cluster.up.contains_key(&id) {
            cluster.start(id)?;
        }
    }

    let agreed = |cluster: &Cluster| {
        cluster
            .leader()
            .is_some_and(|leader| {
                leader.is_settled()
                    && cluster
                        .up
                        .values()
                        .all(|member| member.records() == leader.records())
            })
    };

    for _ in 0..200 {
        if agreed(cluster) {
            break;
        }

        for id in MEMBERS {
            cluster.tick(id)?;
        }

        cluster.deliver_all()?;
    }

    if !agreed(cluster) {
        return Err("the members did not agree after the faults ended".into());
    }

    let leader = cluster
        .leader()
        .ok_or("the members have no leader after the faults ended")?;

    for (sequence, key) in &cluster.acknowledged {
        if leader
            .get(key)
            .is_none_or(|entry| entry.version != *sequence)
        {
            return Err(
                format!("the acknowledged write of {key} is missing after recovery").into(),
            );
        }
    }

    Ok(())
}

fn runs(root: &Path) -> Result<(), Box<dyn Error>> {
    let (mut leaders, mut acknowledged, mut discarded, mut most_terms) = (0, 0, 0, 0);

    for seed in 1..=RUNS {
        let directory = root.join(format!("run-{seed}"));
        let mut cluster = run(&directory, seed)?;

        for id in MEMBERS {
            cluster.crash(id);
        }

        leaders += cluster.leaders.len();
        acknowledged += cluster.acknowledged.len();
        discarded += cluster.discarded;
        most_terms = most_terms.max(
            cluster
                .leaders
                .keys()
                .next_back()
                .copied()
                .unwrap_or(0),
        );

        fs::remove_dir_all(&directory)?;
    }

    println!("{RUNS} runs of {STEPS} random steps:");
    println!("  {leaders} leaders were elected, and no term had 2 leaders");
    println!("  {acknowledged} writes were acknowledged, and every one was present after recovery");
    println!(
        "  {discarded} uncommitted records were discarded by members whose log differed from a leader's"
    );
    println!("  the highest term reached in a run was {most_terms}");

    Ok(())
}

fn main() -> Result<(), Box<dyn Error>> {
    let root = env::temp_dir().join(format!("raftsim-{}", process::id()));
    let result = cases(&root.join("cases")).and_then(|()| runs(&root));

    fs::remove_dir_all(&root)?;

    result
}

Listing 4.47 The complete kvraft/Cargo.toml. kvraft/Cargo.toml

[package]
name = "kvraft"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true
license.workspace = true
publish.workspace = true

[dependencies]
kv.workspace = true
kvd.workspace = true
raft.workspace = true
axum.workspace = true
clap.workspace = true
parking_lot.workspace = true
prost.workspace = true
serde.workspace = true
tokio.workspace = true
tonic.workspace = true
tonic-prost.workspace = true

[build-dependencies]
tonic-prost-build.workspace = true

Listing 4.48 The complete kvraft/src/rpc.rs. kvraft/src/rpc.rs

//! The gRPC services of a member: the generated types, their conversion to
//! the messages of the `raft` crate, and the server side of both calls.

use std::sync::Arc;

use kv::{Change, Record};
use raft::{AppendAnswer, AppendRequest, Message, NodeId, VoteAnswer, VoteRequest};
use tonic::{Request, Response, Status};

use crate::node::Node;

/// The types that `build.rs` generates from `proto/store.proto`. The schema
/// also defines the ring, which this program does not use.
#[allow(dead_code)]
pub mod proto {
    tonic::include_proto!("store.v1");
}

pub use proto::election_client::ElectionClient;
pub use proto::election_server::{Election, ElectionServer};
pub use proto::replication_client::ReplicationClient;
pub use proto::replication_server::{Replication, ReplicationServer};

fn record_to_proto(record: &Record) -> proto::Record {
    let change = match record.change.clone() {
        Change::Put { key, value } => proto::record::Change::Put(proto::Put { key, value }),
        Change::Delete { key } => proto::record::Change::Delete(proto::Delete { key }),
    };

    proto::Record {
        sequence: record.sequence,
        change: Some(change),
    }
}

fn record_from_proto(record: proto::Record) -> Result<Record, Status> {
    let change = match record.change {
        Some(proto::record::Change::Put(put)) => Change::Put {
            key: put.key,
            value: put.value,
        },
        Some(proto::record::Change::Delete(delete)) => Change::Delete { key: delete.key },
        None => return Err(Status::invalid_argument("a record without a change")),
    };

    Ok(Record {
        sequence: record.sequence,
        change,
    })
}

impl From<AppendRequest> for proto::AppendRequest {
    fn from(request: AppendRequest) -> Self {
        proto::AppendRequest {
            shard: 0,
            term: request.term,
            previous_sequence: request.previous_sequence,
            records: request
                .records
                .iter()
                .map(record_to_proto)
                .collect(),
            committed_sequence: request.committed_sequence,
            previous_term: request.previous_term,
            leader: request.leader,
        }
    }
}

impl TryFrom<proto::AppendRequest> for AppendRequest {
    type Error = Status;

    fn try_from(request: proto::AppendRequest) -> Result<Self, Status> {
        Ok(AppendRequest {
            term: request.term,
            leader: request.leader,
            previous_sequence: request.previous_sequence,
            previous_term: request.previous_term,
            records: request
                .records
                .into_iter()
                .map(record_from_proto)
                .collect::<Result<_, _>>()?,
            committed_sequence: request.committed_sequence,
        })
    }
}

impl From<VoteRequest> for proto::VoteRequest {
    fn from(request: VoteRequest) -> Self {
        proto::VoteRequest {
            term: request.term,
            candidate: request.candidate,
            last_sequence: request.last_sequence,
            last_term: request.last_term,
        }
    }
}

/// The answer of member `from` to an `Append` call, as a message.
pub fn appended(from: NodeId, answer: proto::AppendResponse) -> Message {
    Message::Appended(AppendAnswer {
        term: answer.term,
        from,
        appended: answer.appended,
        last_sequence: answer.last_sequence,
    })
}

/// The answer of member `from` to a `RequestVote` call, as a message.
pub fn voted(from: NodeId, answer: proto::VoteResponse) -> Message {
    Message::Voted(VoteAnswer {
        term: answer.term,
        from,
        granted: answer.granted,
    })
}

/// The server side of both services. Each call is one message for the
/// member, and the member's answer is the response of the call.
pub struct Service {
    pub node: Arc<Node>,
}

#[tonic::async_trait]
impl Replication for Service {
    async fn append(
        &self,
        request: Request<proto::AppendRequest>,
    ) -> Result<Response<proto::AppendResponse>, Status> {
        let request = AppendRequest::try_from(request.into_inner())?;

        let answer = self
            .node
            .answer(Message::Append(request))
            .map_err(|error| Status::unavailable(error.to_string()))?;

        let Message::Appended(answer) = answer else {
            return Err(Status::internal(
                "the member answered with another kind of message",
            ));
        };

        Ok(Response::new(proto::AppendResponse {
            term: answer.term,
            appended: answer.appended,
            last_sequence: answer.last_sequence,
        }))
    }
}

#[tonic::async_trait]
impl Election for Service {
    async fn request_vote(
        &self,
        request: Request<proto::VoteRequest>,
    ) -> Result<Response<proto::VoteResponse>, Status> {
        let request = request.into_inner();
        let request = VoteRequest {
            term: request.term,
            candidate: request.candidate,
            last_sequence: request.last_sequence,
            last_term: request.last_term,
        };

        let answer = self
            .node
            .answer(Message::Vote(request))
            .map_err(|error| Status::unavailable(error.to_string()))?;

        let Message::Voted(answer) = answer else {
            return Err(Status::internal(
                "the member answered with another kind of message",
            ));
        };

        Ok(Response::new(proto::VoteResponse {
            term: answer.term,
            granted: answer.granted,
        }))
    }
}

Listing 4.49 The complete kvraft/src/node.rs. kvraft/src/node.rs

//! A node: a member, its clock, and the connections that carry its
//! messages to the other members.

use std::collections::BTreeMap;
use std::io;
use std::sync::Arc;
use std::time::Duration;

use kv::{Change, Condition, Entry, Written};
use kvd::api::ApiError;
use parking_lot::Mutex;
use raft::{Member, Message, NodeId, Outgoing, Role};
use tokio::sync::watch;
use tokio::task;
use tokio::time::{interval, timeout};
use tonic::transport::{Channel, Endpoint};

use crate::rpc::{ElectionClient, ReplicationClient, appended, proto, voted};

/// The length of a tick of the member's clock.
pub const TICK: Duration = Duration::from_millis(100);

/// How long a member waits for the answer to one message.
const ANSWER: Duration = Duration::from_millis(500);

/// How long a write waits for its record to be committed.
const COMMIT: Duration = Duration::from_secs(1);

pub struct Node {
    pub member: Mutex<Member>,
    /// A connection to each of the other members.
    peers: BTreeMap<NodeId, Channel>,
    /// Held by the write that is in progress.
    turn: tokio::sync::Mutex<()>,
    /// The committed sequence, for writes that wait for it to advance.
    committed: watch::Sender<u64>,
}

impl Node {
    pub fn new(
        member: Member,
        addresses: BTreeMap<NodeId, String>,
    ) -> Result<Arc<Node>, tonic::transport::Error> {
        let mut peers = BTreeMap::new();

        for (id, address) in addresses {
            peers.insert(
                id,
                Endpoint::from_shared(address)?
                    .connect_timeout(ANSWER)
                    .connect_lazy(),
            );
        }

        Ok(Arc::new(Node {
            member: Mutex::new(member),
            peers,
            turn: tokio::sync::Mutex::new(()),
            committed: watch::Sender::new(0),
        }))
    }
}

impl Node {
    /// Runs one operation on the member and publishes the committed
    /// sequence. The member waits for the disk, so the operation runs where
    /// blocking is allowed.
    fn with_member<T>(&self, operation: impl FnOnce(&mut Member) -> T) -> T {
        task::block_in_place(|| {
            let mut member = self.member.lock();
            let result = operation(&mut member);
            self.committed
                .send_replace(member.committed());

            result
        })
    }
}

impl Node {
    /// Sends each message in a task of its own, so that a member that does
    /// not answer delays nothing else.
    fn dispatch(self: &Arc<Node>, messages: Vec<Outgoing>) {
        for message in messages {
            tokio::spawn(Arc::clone(self).send(message));
        }
    }

    /// Carries one request to a member and gives the answer to this member.
    async fn send(self: Arc<Node>, outgoing: Outgoing) {
        let Some(channel) = self
            .peers
            .get(&outgoing.to)
            .cloned()
        else {
            return;
        };

        let answer = match outgoing.message {
            Message::Append(request) => {
                let call = async move {
                    ReplicationClient::new(channel)
                        .append(proto::AppendRequest::from(request))
                        .await
                };

                timeout(ANSWER, call)
                    .await
                    .ok()
                    .and_then(Result::ok)
                    .map(|answer| appended(outgoing.to, answer.into_inner()))
            }
            Message::Vote(request) => {
                let call = async move {
                    ElectionClient::new(channel)
                        .request_vote(proto::VoteRequest::from(request))
                        .await
                };

                timeout(ANSWER, call)
                    .await
                    .ok()
                    .and_then(Result::ok)
                    .map(|answer| voted(outgoing.to, answer.into_inner()))
            }
            // An answer travels as the response of the call that asked.
            Message::Appended(..) | Message::Voted(..) => None,
        };

        if let Some(answer) = answer {
            self.deliver(answer);
        }
    }

    /// Gives the member a message and sends what the member returns.
    fn deliver(self: &Arc<Node>, message: Message) {
        match self.with_member(|member| member.handle(message)) {
            Ok(messages) => self.dispatch(messages),
            Err(error) => eprintln!("the member could not process a message: {error}"),
        }
    }
}

impl Node {
    /// The member's answer to a request that arrived over gRPC.
    pub fn answer(&self, request: Message) -> io::Result<Message> {
        let mut messages = self.with_member(|member| member.handle(request))?;

        messages
            .pop()
            .map(|outgoing| outgoing.message)
            .ok_or_else(|| io::Error::other("the member gave no answer"))
    }

    /// Advances the member's clock, for as long as the node runs.
    pub async fn run_clock(self: Arc<Node>) {
        let mut ticks = interval(TICK);

        loop {
            ticks.tick().await;

            match self.with_member(Member::tick) {
                Ok(messages) => self.dispatch(messages),
                Err(error) => eprintln!("the member could not advance its clock: {error}"),
            }
        }
    }
}

impl Node {
    fn not_leader(member: &Member) -> ApiError {
        match member.leader() {
            Some(leader) => ApiError::Misdirected(format!("the leader is member {leader}")),
            None => ApiError::NoLeader,
        }
    }

    /// A write on the leader. It is acknowledged when its record is
    /// committed and this member has led the same term throughout.
    pub async fn write(
        self: &Arc<Node>,
        change: Change,
        condition: Condition,
    ) -> Result<Written, ApiError> {
        let turn = self.turn.lock().await;

        let last = {
            let member = self.member.lock();

            if member.role() != Role::Leader {
                return Err(Node::not_leader(&member));
            }

            member.last_sequence()
        };

        self.wait_for(last).await?;

        let (written, term, messages) = self.with_member(|member| {
            if !member.is_settled() {
                return Err(Node::not_leader(member));
            }

            let created = match &change {
                Change::Put { key, .. } | Change::Delete { key } => member.get(key).is_none(),
            };
            let version = member.propose(change, condition)?;

            Ok((
                Written { version, created },
                member.term(),
                member.replicate(),
            ))
        })?;

        self.dispatch(messages);
        self.wait_for(written.version)
            .await?;
        drop(turn);

        // A member that lost the leadership while it waited may hold another
        // leader's record at this sequence.
        let member = self.member.lock();

        if member.role() == Role::Leader && member.term() == term {
            Ok(written)
        } else {
            Err(ApiError::Unavailable(
                "the leadership changed before the record was committed".into(),
            ))
        }
    }

    async fn wait_for(&self, sequence: u64) -> Result<(), ApiError> {
        let mut committed = self.committed.subscribe();

        let reached = committed.wait_for(|&committed| committed >= sequence);
        let in_time = timeout(COMMIT, reached)
            .await
            .is_ok_and(|reached| reached.is_ok());

        if in_time {
            Ok(())
        } else {
            Err(ApiError::Unavailable(format!(
                "record {sequence} is not on a majority of the members"
            )))
        }
    }

    pub fn get(&self, key: &str) -> Result<Entry, ApiError> {
        let member = self.member.lock();

        if member.role() != Role::Leader {
            return Err(Node::not_leader(&member));
        }

        if !member.can_read() {
            return Err(ApiError::Unavailable(
                "this leader cannot confirm that it still leads".into(),
            ));
        }

        member
            .get(key)
            .cloned()
            .ok_or(ApiError::NotFound)
    }
}

Listing 4.50 The complete kvraft/src/main.rs. kvraft/src/main.rs

//! Command kvraft is a storage node whose shard elects its own leader.
//! Every node is started the same way, with its id and the replication
//! addresses of the other members.
//!
//! Run 3 nodes from code/ with:
//!
//!     cargo run -p kvraft -- --id 1 --listen 127.0.0.1:7001 --replication 127.0.0.1:7101 --data /var/tmp/m1 \
//!         --peer 2=http://127.0.0.1:7102 --peer 3=http://127.0.0.1:7103
//!
//! and the same for ids 2 and 3, each with its own ports and the other 2 as peers.

mod node;
mod rpc;

use std::collections::BTreeMap;
use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;

use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_NONE_MATCH};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use kv::{Change, Condition};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, version_header, write_condition};
use raft::{Member, NodeId, Role};
use serde::Serialize;
use tokio::net::TcpListener;
use tonic::transport::Server;

use crate::node::Node;
use crate::rpc::{ElectionServer, ReplicationServer, Service};

/// Reads `2=http://127.0.0.1:7102` as member 2 at that address.
fn parse_peer(text: &str) -> Result<(NodeId, String), String> {
    let (id, address) = text
        .split_once('=')
        .ok_or("a peer is written as id=address")?;
    let Ok(id) = id.parse() else {
        return Err("the id of a peer is a number".to_owned());
    };

    Ok((id, address.to_owned()))
}

#[derive(Parser)]
struct Options {
    /// The id of this member, a number that no other member has.
    #[arg(long)]
    id: NodeId,

    /// The address of the HTTP API.
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    /// The address on which this member accepts messages from the others.
    #[arg(long, default_value = "127.0.0.1:7101")]
    replication: SocketAddr,

    /// Another member and its replication address, as id=address.
    #[arg(long = "peer", value_parser = parse_peer)]
    peers: Vec<(NodeId, String)>,

    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// Respond without waiting for the disk.
    #[arg(long)]
    no_sync: bool,
}

async fn put_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    let written = node
        .write(
            Change::Put {
                key,
                value: body.to_vec(),
            },
            condition,
        )
        .await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;
    let condition = write_condition(&headers)?;

    node.write(Change::Delete { key }, condition)
        .await?;

    Ok(StatusCode::NO_CONTENT)
}

async fn get_key(
    State(node): State<Arc<Node>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<Response, ApiError> {
    valid_key(&key)?;
    let cached = version_header(&headers, IF_NONE_MATCH)?;
    let entry = node.get(&key)?;
    let tag = [(ETAG, etag(entry.version))];

    if cached == Some(Condition::Version(entry.version)) {
        return Ok((StatusCode::NOT_MODIFIED, tag).into_response());
    }

    Ok((
        tag,
        [(CONTENT_TYPE, "application/octet-stream")],
        entry.value,
    )
        .into_response())
}

#[derive(Serialize)]
struct Health {
    id: NodeId,
    role: &'static str,
    term: u64,
    /// The leader of the term, when this member knows it.
    leader: Option<NodeId>,
    last_sequence: u64,
    committed: u64,
}

async fn health(State(node): State<Arc<Node>>) -> Json<Health> {
    let member = node.member.lock();

    let role = match member.role() {
        Role::Leader => "leader",
        Role::Candidate => "candidate",
        Role::Follower => "follower",
    };

    Json(Health {
        id: member.id(),
        role,
        term: member.term(),
        leader: member.leader(),
        last_sequence: member.last_sequence(),
        committed: member.committed(),
    })
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let peers: BTreeMap<NodeId, String> = options.peers.into_iter().collect();
    let member = Member::open(
        &options.data,
        options.id,
        peers.keys().copied().collect(),
        !options.no_sync,
    )?;

    println!(
        "member {}, term {}, log ends at {}",
        member.id(),
        member.term(),
        member.last_sequence()
    );

    let node = Node::new(member, peers)?;

    let replication = ReplicationServer::new(Service {
        node: Arc::clone(&node),
    });
    let election = ElectionServer::new(Service {
        node: Arc::clone(&node),
    });

    tokio::spawn(
        Server::builder()
            .add_service(replication)
            .add_service(election)
            .serve(options.replication),
    );
    tokio::spawn(Arc::clone(&node).run_clock());

    let routes = Router::new()
        .route(
            "/v1/keys/{key}",
            get(get_key)
                .put(put_key)
                .delete(delete_key),
        )
        .route("/healthz", get(health))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(node);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Listing 4.51 The complete probe/src/main.rs. probe/src/main.rs

//! Command probe measures how long a shard is unavailable when its leader
//! is lost. It writes keys for a number of seconds, trying each node in turn
//! until one accepts the write, and reports the longest pause between 2
//! acknowledged writes.
//!
//! Run it from code/ with:
//!
//!     cargo run --release -p probe -- --seconds 20 --acked acked.txt \
//!         --server http://127.0.0.1:7001 --server http://127.0.0.1:7002 --server http://127.0.0.1:7003

use std::error::Error;
use std::fs::File;
use std::io::{BufWriter, Write};
use std::path::PathBuf;
use std::thread;
use std::time::{Duration, Instant};

use clap::Parser;
use ureq::Agent;

#[derive(Parser)]
struct Options {
    /// The HTTP address of a node. Given once for each node of the shard.
    #[arg(long = "server", required = true)]
    servers: Vec<String>,

    #[arg(long, default_value_t = 20)]
    seconds: u64,

    /// The file that receives the key of every acknowledged write.
    #[arg(long)]
    acked: PathBuf,
}

fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let agent: Agent = Agent::config_builder()
        .timeout_global(Some(Duration::from_secs(2)))
        .build()
        .into();
    let mut file = BufWriter::new(File::create(&options.acked)?);

    let started = Instant::now();
    let (mut server, mut number, mut refused) = (0, 1, 0);
    let (mut last_success, mut longest, mut longest_at) = (started, Duration::ZERO, Duration::ZERO);

    while started.elapsed() < Duration::from_secs(options.seconds) {
        let key = format!("key-{number:06}");
        let address = format!("{}/v1/keys/{key}", options.servers[server]);

        if agent
            .put(&address)
            .send(format!("value of {key}"))
            .is_err()
        {
            // This node did not accept the write. Try the next one.
            refused += 1;
            server = (server + 1) % options.servers.len();
            thread::sleep(Duration::from_millis(20));

            continue;
        }

        let pause = last_success.elapsed();

        if pause > longest {
            (longest, longest_at) = (pause, started.elapsed());
        }

        last_success = Instant::now();
        writeln!(file, "{key}")?;
        number += 1;
    }

    file.flush()?;

    println!(
        "{} writes acknowledged in {} seconds, {refused} attempts refused",
        number - 1,
        options.seconds
    );
    println!(
        "the longest pause between 2 acknowledged writes was {:.2} s, ending {:.1} s after the start",
        longest.as_secs_f64(),
        longest_at.as_secs_f64()
    );

    Ok(())
}

Further reading

• D. Ongaro and J. Ousterhout, “In Search of an Understandable Consensus Algorithm”, USENIX ATC 2014.
• D. Ongaro, Consensus: Bridging Theory and Practice, doctoral dissertation, Stanford University, 2014.
• L. Lamport, “Paxos Made Simple”, ACM SIGACT News, 2001.
• T. Chandra, R. Griesemer and J. Redstone, “Paxos Made Live: An Engineering Perspective”, PODC 2007.
• M. Fischer, N. Lynch and M. Paterson, “Impossibility of Distributed Consensus with One Faulty Process”, Journal of the ACM, 1985.
• K. Kingsbury, the Jepsen analyses, jepsen.io.

Key-Value Store: Router (Low-Level Design)

With chapter 4 complete, we have a shard that survives the loss of its leader without anybody’s help. Measured against the design of chapter 1, 2 things are still wanting. A client has to discover for itself which member of the shard is the leader, as the program probe did by trying each node in turn and collecting 51 refusals in the process. And all the data is in a single shard, whereas the design calls for the keys to be divided among many.

A single small component supplies both. The router stands between the clients and the shards. Clients send every request to it, and for each request it works out which shard owns the key and which member of that shard is at present the leader, and passes the request on. The clients are thereby relieved of all knowledge of the cluster. They know 1 address, and neither the number of shards nor a change of leader is visible to them.

The router is the least intricate program in this part of the book, and it contains 1 decision that is easy to get wrong and expensive when it is: what to do when the member it has asked does not give a clear answer. A router that retries too readily will perform a write twice, and a router that retries too reluctantly will report failures that it could have concealed. The chapter builds the router, states the rule that settles this question, and then repeats the experiment of chapter 4, killing a leader under load, with the router in front.

What a router does

A router keeps no data. Everything it knows comes from a ring file, read when it starts, and from what the members tell it in the course of its work. This is the statelessness that chapter 1 required of routers, and its practical value is considerable: any number of routers may be run side by side, and any of them may be stopped or restarted at any time, since a router that has lost its memory rebuilds it from the first few answers it receives.

The ring file

$ cat ring.json
{
  "virtual_nodes": 256,
  "shards": [
    { "name": "shard-a", "members": ["http://127.0.0.1:7011", "http://127.0.0.1:7012", "http://127.0.0.1:7013"] },
    { "name": "shard-b", "members": ["http://127.0.0.1:7021", "http://127.0.0.1:7022", "http://127.0.0.1:7023"] }
  ]
}

Each shard is a group of 3 kvraft members, listed in the order of their ids. In the design of chapter 1, the ring is held by a coordinator and changes while the system runs. Here it is a file that the router reads once at startup. That is sufficient for routing requests, and it leaves changes to the ring, which are a substantial subject in themselves, outside this chapter.

Listing 5.1 The imports of kvrouter/src/main.rs. kvrouter/src/main.rs

use std::collections::HashMap;
use std::error::Error;
use std::fs;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;

use axum::body::Bytes;
use axum::extract::{Path, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_MATCH, IF_NONE_MATCH};
use axum::http::{HeaderMap, Method, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use kvd::api::{ApiError, valid_key};
use ring::Ring;
use serde::{Deserialize, Serialize};
use tokio::net::TcpListener;
use tokio::time::sleep;

Listing 5.2 The ring file, and what the router keeps for each shard. kvrouter/src/main.rs

#[derive(Parser)]
struct Options {
    #[arg(long, default_value = "127.0.0.1:7000")]
    listen: SocketAddr,

    /// A JSON file that lists the shards and the addresses of their members.
    #[arg(long)]
    ring: PathBuf,
}

/// The contents of the ring file.
#[derive(Deserialize)]
struct Config {
    virtual_nodes: u32,
    shards: Vec<ShardConfig>,
}

#[derive(Deserialize)]
struct ShardConfig {
    name: String,
    /// The HTTP address of each member, in the order of their ids: member 1 first.
    members: Vec<String>,
}

/// A shard, as the router knows it.
struct Shard {
    members: Vec<String>,
    /// The position in `members` of the member that led when the router last asked.
    leader: AtomicUsize,
}

struct Routes {
    ring: Ring,
    shards: HashMap<String, Shard>,
    client: reqwest::Client,
}

For every shard the router remembers a single number, the position in the list of the member that was the leader when it last asked. This is a guess and nothing more. The sections that follow show what happens when the guess is wrong, and the important property of the design is that a wrong guess costs an extra request and never a wrong result.

Finding the shard

Each shard is placed on the ring of chapter 1 with 256 tokens. To look up a key then takes 2 steps: the ring is asked for the name of the owner, and the shard of that name is fetched.

Listing 5.3 Building the ring, and finding the shard of a key. kvrouter/src/main.rs

impl Routes {
    /// Builds the ring from the file: every shard receives its tokens.
    fn load(config: Config) -> Routes {
        let mut ring = Ring::new(config.virtual_nodes);
        let mut shards = HashMap::new();

        for shard in config.shards {
            ring.add(&shard.name);
            shards.insert(
                shard.name,
                Shard {
                    members: shard.members,
                    leader: AtomicUsize::new(0),
                },
            );
        }

        Routes {
            ring,
            shards,
            client: reqwest::Client::new(),
        }
    }

    /// The shard that owns a key.
    fn shard_of(&self, key: &str) -> Option<&Shard> {
        self.shards
            .get(self.ring.owner(key)?)
    }
}

Finding the leader

The router sends the request to the member that it believes to be the leader. Usually the belief is correct. When it is not, the answer of the member tells the router what to do next.

All the cases are decided by a single rule: the router asks another member only if the first attempt has certainly changed nothing. A follower that answers 421 did not touch its log. Nor did a member that reports that the shard is electing a leader, nor one whose process is not running and whose connection was refused. In all 3 cases, to ask another member is safe, since whatever that member does will be the only effect of the request.

A timeout is a different matter altogether. The leader may have appended the record, and even committed it, before the connection failed, and the router cannot know. If it were to send the request to another member, a write might be performed twice, and a conditional write that had succeeded might be reported as failed. The router therefore answers 503 and leaves the decision to the client, which is the only party that knows whether its request is safe to repeat. This is the lost response of chapter 1 once again, and the router’s rule is a refusal to guess about it on somebody else’s behalf.

In the code, the possible outcomes of asking a member are the variants of an enum.

Listing 5.4 The 3 outcomes of asking a member. kvrouter/src/main.rs

/// What a member said about a request it did not serve.
#[derive(Deserialize)]
struct Problem {
    #[serde(rename = "type")]
    kind: String,
    #[serde(default)]
    detail: String,
}

impl Problem {
    /// The position of the leader in the shard's list of members, when the
    /// member named it, as in "the leader is member 3".
    fn named_leader(&self) -> Option<usize> {
        let id: usize = self
            .detail
            .rsplit(' ')
            .next()?
            .parse()
            .ok()?;

        id.checked_sub(1)
    }
}

/// The result of asking one member.
enum Attempt {
    /// The member served the request. This is its response.
    Served(Response),
    /// The member changed nothing. It may have named the leader.
    Refused(Option<usize>),
    /// The request may or may not have taken effect.
    Unknown(String),
}

ask sends the request to 1 member and classifies what comes back as 1 of the outcomes.

Listing 5.5 Asking 1 member. kvrouter/src/main.rs

const NO_LEADER: &str = "/problems/no-leader";

impl Routes {
    /// Sends the request to one member and classifies what comes back.
    async fn ask(
        &self,
        member: &str,
        method: Method,
        key: &str,
        headers: &HeaderMap,
        body: Bytes,
    ) -> Attempt {
        let conditions = [IF_MATCH, IF_NONE_MATCH]
            .into_iter()
            .filter_map(|name| {
                headers
                    .get(&name)
                    .map(|value| (name, value))
            });
        let request = self
            .client
            .request(method, format!("{member}/v1/keys/{key}"))
            .body(body);
        let request = conditions.fold(request, |request, (name, value)| {
            request.header(name, value)
        });

        let response = match request.send().await {
            Ok(response) => response,
            Err(error) if error.is_connect() => return Attempt::Refused(None),
            Err(error) => return Attempt::Unknown(error.to_string()),
        };

        let status = response.status();
        let redirected = status == StatusCode::MISDIRECTED_REQUEST;

        if !redirected && status != StatusCode::SERVICE_UNAVAILABLE {
            return Attempt::Served(relay(response).await);
        }

        match response.json::<Problem>().await {
            Ok(problem) if redirected || problem.kind == NO_LEADER => {
                Attempt::Refused(problem.named_leader())
            }
            Ok(problem) => Attempt::Unknown(problem.detail),
            Err(error) => Attempt::Unknown(error.to_string()),
        }
    }
}

/// Passes a member's response on to the client: status, version and body.
async fn relay(response: reqwest::Response) -> Response {
    let status = response.status();
    let headers: HeaderMap = [ETAG, CONTENT_TYPE]
        .into_iter()
        .filter_map(|name| {
            response
                .headers()
                .get(&name)
                .map(|value| (name, value.clone()))
        })
        .collect();

    match response.bytes().await {
        Ok(body) => (status, headers, body).into_response(),
        Err(error) => ApiError::Unavailable(error.to_string()).into_response(),
    }
}

forward is the loop around it. A refusal moves the guess to the member that the refusal named, or failing that to the next in the list. When every member has refused once, an election must be in progress, and the router then waits 150 ms between further attempts, so that it does not flood a shard which is busy choosing a leader.

Listing 5.6 Trying members until one serves the request. kvrouter/src/main.rs

/// How many times the router asks a member before it gives up.
const ATTEMPTS: usize = 12;

/// How long the router waits before it asks again, once every member of a
/// shard has refused.
const PAUSE: Duration = Duration::from_millis(150);

impl Routes {
    /// Sends a request to the leader of the key's shard. A member that
    /// refuses has changed nothing, so the router asks another.
    async fn forward(
        &self,
        method: Method,
        key: &str,
        headers: &HeaderMap,
        body: Bytes,
    ) -> Result<Response, ApiError> {
        valid_key(key)?;

        let shard = self
            .shard_of(key)
            .ok_or(ApiError::NoLeader)?;

        for attempt in 1..=ATTEMPTS {
            let tried = shard.leader.load(Ordering::Relaxed);
            let member = &shard.members[tried];

            let named = match self
                .ask(member, method.clone(), key, headers, body.clone())
                .await
            {
                Attempt::Served(response) => return Ok(response),
                Attempt::Unknown(detail) => return Err(ApiError::Unavailable(detail)),
                Attempt::Refused(named) => named,
            };

            let next = named
                .filter(|&position| position < shard.members.len())
                .unwrap_or((tried + 1) % shard.members.len());

            shard
                .leader
                .store(next, Ordering::Relaxed);

            // Every member has refused once: the shard is electing a leader.
            if attempt >= shard.members.len() {
                sleep(PAUSE).await;
            }
        }

        Err(ApiError::NoLeader)
    }
}

Listing 5.7 The HTTP handlers, and the start of the program. kvrouter/src/main.rs

async fn key(
    State(routes): State<Arc<Routes>>,
    method: Method,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    routes
        .forward(method, &key, &headers, body)
        .await
}

#[derive(Serialize)]
struct ShardHealth<'a> {
    name: &'a str,
    /// The address the router currently sends this shard's requests to.
    leader: &'a str,
}

async fn health(State(routes): State<Arc<Routes>>) -> Json<serde_json::Value> {
    let mut shards: Vec<ShardHealth> = routes
        .shards
        .iter()
        .map(|(name, shard)| ShardHealth {
            name,
            leader: &shard.members[shard.leader.load(Ordering::Relaxed)],
        })
        .collect();
    shards.sort_by_key(|shard| shard.name);

    Json(serde_json::json!({ "status": "ok", "shards": shards }))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let config: Config = serde_json::from_str(&fs::read_to_string(&options.ring)?)?;
    let routes = Arc::new(Routes::load(config));

    println!("routing to {} shards", routes.shards.len());

    let app = Router::new()
        .route("/v1/keys/{key}", get(key).put(key).delete(key))
        .route("/healthz", get(health))
        .with_state(routes);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, app).await?;

    Ok(())
}

2 shards behind 1 router

We start 6 kvraft members in 2 groups of 3, then the router, and write 400 keys through it.

$ ./target/release/kvrouter --listen 127.0.0.1:7000 --ring ring.json &
routing to 2 shards
listening on 127.0.0.1:7000
$ ./target/release/kvctl load --server http://127.0.0.1:7000 --count 400 --acked acked.txt
400 writes acknowledged
$ for port in 7013 7023; do curl http://127.0.0.1:$port/healthz; echo; done
{"id":3,"role":"leader","term":1,"leader":3,"last_sequence":206,"committed":206}
{"id":3,"role":"leader","term":1,"leader":3,"last_sequence":196,"committed":196}
$ ./target/release/kvctl verify --server http://127.0.0.1:7000 --acked acked.txt
400 acknowledged, 400 found, 0 missing

The client used a single address throughout. It never learned how many shards exist, or which members lead them.

A leader is lost behind the router

probe, from chapter 4, is now given the router’s address and no other. While it is writing, the leader of shard-a is killed.

$ ./target/release/probe --seconds 12 --acked acked.txt --server http://127.0.0.1:7000 &
$ sleep 4; kill -9 <pid of member 3 of shard-a, its leader>
780 writes acknowledged in 12 seconds, 0 attempts refused
the longest pause between 2 acknowledged writes was 1.08 s, ending 5.2 s after the start
$ curl http://127.0.0.1:7000/healthz
{"shards":[{"leader":"http://127.0.0.1:7011","name":"shard-a"},{"leader":"http://127.0.0.1:7023","name":"shard-b"}],"status":"ok"}
$ ./target/release/kvctl verify --server http://127.0.0.1:7000 --acked acked.txt
780 acknowledged, 780 found, 0 missing

The comparison with chapter 4 shows what the router contributes. The election took as long as it did before, about a second, since the router does nothing to hasten it. What has changed is what the client experiences. A second of refusals, which every client would have had to be written to handle, has become a second of waiting, which requires nothing of the client at all.

Writes to shard-b were at no time affected. probe sends its keys in turn, and so it waited behind the 1 request for shard-a that was being held. Independent clients writing to shard-b would have noticed nothing.

What is still missing

The complete program

Listing 5.8 The complete kvrouter/Cargo.toml. kvrouter/Cargo.toml

[package]
name = "kvrouter"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true
license.workspace = true
publish.workspace = true

[dependencies]
kvd.workspace = true
ring.workspace = true
axum.workspace = true
clap.workspace = true
reqwest.workspace = true
serde.workspace = true
serde_json.workspace = true
tokio.workspace = true

Listing 5.9 The complete kvrouter/src/main.rs. kvrouter/src/main.rs

//! Command kvrouter sends each request to the shard that owns its key, and
//! within that shard to the member that leads. Clients talk to the router
//! and need to know nothing about shards or leaders.
//!
//! Run it from code/ with:
//!
//!     cargo run -p kvrouter -- --listen 127.0.0.1:7000 --ring ring.json

use std::collections::HashMap;
use std::error::Error;
use std::fs;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;

use axum::body::Bytes;
use axum::extract::{Path, State};
use axum::http::header::{CONTENT_TYPE, ETAG, IF_MATCH, IF_NONE_MATCH};
use axum::http::{HeaderMap, Method, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use kvd::api::{ApiError, valid_key};
use ring::Ring;
use serde::{Deserialize, Serialize};
use tokio::net::TcpListener;
use tokio::time::sleep;

#[derive(Parser)]
struct Options {
    #[arg(long, default_value = "127.0.0.1:7000")]
    listen: SocketAddr,

    /// A JSON file that lists the shards and the addresses of their members.
    #[arg(long)]
    ring: PathBuf,
}

/// The contents of the ring file.
#[derive(Deserialize)]
struct Config {
    virtual_nodes: u32,
    shards: Vec<ShardConfig>,
}

#[derive(Deserialize)]
struct ShardConfig {
    name: String,
    /// The HTTP address of each member, in the order of their ids: member 1 first.
    members: Vec<String>,
}

/// A shard, as the router knows it.
struct Shard {
    members: Vec<String>,
    /// The position in `members` of the member that led when the router last asked.
    leader: AtomicUsize,
}

struct Routes {
    ring: Ring,
    shards: HashMap<String, Shard>,
    client: reqwest::Client,
}

impl Routes {
    /// Builds the ring from the file: every shard receives its tokens.
    fn load(config: Config) -> Routes {
        let mut ring = Ring::new(config.virtual_nodes);
        let mut shards = HashMap::new();

        for shard in config.shards {
            ring.add(&shard.name);
            shards.insert(
                shard.name,
                Shard {
                    members: shard.members,
                    leader: AtomicUsize::new(0),
                },
            );
        }

        Routes {
            ring,
            shards,
            client: reqwest::Client::new(),
        }
    }

    /// The shard that owns a key.
    fn shard_of(&self, key: &str) -> Option<&Shard> {
        self.shards
            .get(self.ring.owner(key)?)
    }
}

/// What a member said about a request it did not serve.
#[derive(Deserialize)]
struct Problem {
    #[serde(rename = "type")]
    kind: String,
    #[serde(default)]
    detail: String,
}

impl Problem {
    /// The position of the leader in the shard's list of members, when the
    /// member named it, as in "the leader is member 3".
    fn named_leader(&self) -> Option<usize> {
        let id: usize = self
            .detail
            .rsplit(' ')
            .next()?
            .parse()
            .ok()?;

        id.checked_sub(1)
    }
}

/// The result of asking one member.
enum Attempt {
    /// The member served the request. This is its response.
    Served(Response),
    /// The member changed nothing. It may have named the leader.
    Refused(Option<usize>),
    /// The request may or may not have taken effect.
    Unknown(String),
}

const NO_LEADER: &str = "/problems/no-leader";

impl Routes {
    /// Sends the request to one member and classifies what comes back.
    async fn ask(
        &self,
        member: &str,
        method: Method,
        key: &str,
        headers: &HeaderMap,
        body: Bytes,
    ) -> Attempt {
        let conditions = [IF_MATCH, IF_NONE_MATCH]
            .into_iter()
            .filter_map(|name| {
                headers
                    .get(&name)
                    .map(|value| (name, value))
            });
        let request = self
            .client
            .request(method, format!("{member}/v1/keys/{key}"))
            .body(body);
        let request = conditions.fold(request, |request, (name, value)| {
            request.header(name, value)
        });

        let response = match request.send().await {
            Ok(response) => response,
            Err(error) if error.is_connect() => return Attempt::Refused(None),
            Err(error) => return Attempt::Unknown(error.to_string()),
        };

        let status = response.status();
        let redirected = status == StatusCode::MISDIRECTED_REQUEST;

        if !redirected && status != StatusCode::SERVICE_UNAVAILABLE {
            return Attempt::Served(relay(response).await);
        }

        match response.json::<Problem>().await {
            Ok(problem) if redirected || problem.kind == NO_LEADER => {
                Attempt::Refused(problem.named_leader())
            }
            Ok(problem) => Attempt::Unknown(problem.detail),
            Err(error) => Attempt::Unknown(error.to_string()),
        }
    }
}

/// Passes a member's response on to the client: status, version and body.
async fn relay(response: reqwest::Response) -> Response {
    let status = response.status();
    let headers: HeaderMap = [ETAG, CONTENT_TYPE]
        .into_iter()
        .filter_map(|name| {
            response
                .headers()
                .get(&name)
                .map(|value| (name, value.clone()))
        })
        .collect();

    match response.bytes().await {
        Ok(body) => (status, headers, body).into_response(),
        Err(error) => ApiError::Unavailable(error.to_string()).into_response(),
    }
}

/// How many times the router asks a member before it gives up.
const ATTEMPTS: usize = 12;

/// How long the router waits before it asks again, once every member of a
/// shard has refused.
const PAUSE: Duration = Duration::from_millis(150);

impl Routes {
    /// Sends a request to the leader of the key's shard. A member that
    /// refuses has changed nothing, so the router asks another.
    async fn forward(
        &self,
        method: Method,
        key: &str,
        headers: &HeaderMap,
        body: Bytes,
    ) -> Result<Response, ApiError> {
        valid_key(key)?;

        let shard = self
            .shard_of(key)
            .ok_or(ApiError::NoLeader)?;

        for attempt in 1..=ATTEMPTS {
            let tried = shard.leader.load(Ordering::Relaxed);
            let member = &shard.members[tried];

            let named = match self
                .ask(member, method.clone(), key, headers, body.clone())
                .await
            {
                Attempt::Served(response) => return Ok(response),
                Attempt::Unknown(detail) => return Err(ApiError::Unavailable(detail)),
                Attempt::Refused(named) => named,
            };

            let next = named
                .filter(|&position| position < shard.members.len())
                .unwrap_or((tried + 1) % shard.members.len());

            shard
                .leader
                .store(next, Ordering::Relaxed);

            // Every member has refused once: the shard is electing a leader.
            if attempt >= shard.members.len() {
                sleep(PAUSE).await;
            }
        }

        Err(ApiError::NoLeader)
    }
}

async fn key(
    State(routes): State<Arc<Routes>>,
    method: Method,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    routes
        .forward(method, &key, &headers, body)
        .await
}

#[derive(Serialize)]
struct ShardHealth<'a> {
    name: &'a str,
    /// The address the router currently sends this shard's requests to.
    leader: &'a str,
}

async fn health(State(routes): State<Arc<Routes>>) -> Json<serde_json::Value> {
    let mut shards: Vec<ShardHealth> = routes
        .shards
        .iter()
        .map(|(name, shard)| ShardHealth {
            name,
            leader: &shard.members[shard.leader.load(Ordering::Relaxed)],
        })
        .collect();
    shards.sort_by_key(|shard| shard.name);

    Json(serde_json::json!({ "status": "ok", "shards": shards }))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let config: Config = serde_json::from_str(&fs::read_to_string(&options.ring)?)?;
    let routes = Arc::new(Routes::load(config));

    println!("routing to {} shards", routes.shards.len());

    let app = Router::new()
        .route("/v1/keys/{key}", get(key).put(key).delete(key))
        .route("/healthz", get(health))
        .with_state(routes);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, app).await?;

    Ok(())
}

Key-Value Store: Group Commit (Low-Level Design)

The capacity estimate of chapter 1 arrived at 24 nodes by assuming that a node stores 20 writes with every wait for its disk. Without that assumption the same calculation gave 325 nodes. Nothing built so far does anything of the kind. The storage node of chapter 2 waits once for every write, and reaches its limit at about 180 writes per second however many clients are sending to it, since each must wait its turn for a disk that confirms 185 times a second.

This chapter adds the missing technique. Under group commit, a node stores all the writes that are waiting at a given moment with a single wait for the disk. The change is small, some 60 lines in the store and 1 thread in the server, and its effect is out of all proportion to its size: the measured rate rises from 180 writes per second to almost 5000 with 64 clients, and beyond 17000 with 256. It is among the best examples in systems programming of a large gain obtained by rearranging work, with no part of the work made faster.

The idea

A wait for the disk takes about 5.4 ms whether it stores 1 record or 100. Nearly all of that time is spent by the disk in confirming that the data is safe, and very little in transferring the data, so that the cost is close to a fixed charge for each wait and almost independent of how much is written. Writes that arrive while 1 wait is in progress can therefore share the next.

The design uses no timer, and the point deserves emphasis because the alternative is common. Some systems hold a write back for a millisecond or so in the hope that others will arrive to share its wait. This improves throughput under heavy load and penalises every write under light load, when no others are coming. Here a batch is formed only from writes that are already waiting. A write that arrives alone is stored at once, with no delay, and the batches grow of their own accord as the load grows, since the busier the node is, the more writes accumulate during each wait. The mechanism adjusts itself to the load without any parameter that an operator has to tune.

A batch in the store

Store::commit takes a list of writes and returns a result for each of them.

Listing 6.1 The imports of kv/src/store.rs. kv/src/store.rs

use std::collections::BTreeMap;
use std::io;
use std::ops::Bound;
use std::path::Path;

use thiserror::Error;

use crate::log::Log;
use crate::record::{Change, Record};

Listing 6.2 Storing a batch of writes with a single wait for the disk. kv/src/store.rs

/// One write of a batch: the change, and the condition it carries.
pub struct Write {
    pub change: Change,
    pub condition: Condition,
}

impl Store {
    /// Performs several writes with a single wait for the disk. Each write
    /// is checked as if the writes before it in the batch had already
    /// happened. The result of each is returned in the same order.
    pub fn commit(&mut self, writes: Vec<Write>) -> Vec<Result<Written, StoreError>> {
        // The versions that keys will have once the accepted writes of this
        // batch are applied. `None` is a key that a write of the batch deletes.
        let mut pending: BTreeMap<String, Option<u64>> = BTreeMap::new();
        let mut records = Vec::new();
        let mut results = Vec::new();
        let first = self.log.last_sequence() + 1;

        for write in writes {
            let key = match &write.change {
                Change::Put { key, .. } | Change::Delete { key } => key.clone(),
            };
            let current = match pending.get(&key) {
                Some(version) => *version,
                None => self.version(&key),
            };
            let deletes = matches!(write.change, Change::Delete { .. });

            let checked = if deletes && current.is_none() {
                Err(StoreError::NotFound)
            } else {
                check(write.condition, current)
            };

            if let Err(error) = checked {
                results.push(Err(error));

                continue;
            }

            let sequence = first + records.len() as u64;
            let version = if deletes { None } else { Some(sequence) };

            pending.insert(key, version);
            results.push(Ok(Written {
                version: sequence,
                created: current.is_none(),
            }));
            records.push(Record {
                sequence,
                change: write.change,
            });
        }

        // One write and one wait for the disk, for the whole batch.
        if let Err(error) = self.log.append_records(&records) {
            let failure = error.to_string();

            return results
                .into_iter()
                .map(|result| result.and(Err(StoreError::Log(io::Error::other(failure.clone())))))
                .collect();
        }

        for record in records {
            apply(&mut self.entries, record);
        }

        results
    }
}

The method works in 3 passes.

Conditional writes need care in the first pass. Suppose that a batch contains 2 writes to user-1, each carrying If-None-Match: *, which means “create only”. The first ought to succeed and the second to fail, although neither is yet in the map, and a check against the map alone would accept both. The pending map resolves the difficulty. It records the version that each key will have once the accepted writes of the batch have been applied, and a later write in the same batch is checked against it. The batch thus behaves, as far as any client can tell, exactly as if its writes had been performed in sequence, which is the property that makes batching invisible.

The order of the passes preserves invariant 1 of chapter 2. No result leaves commit before the wait for the disk has ended, and so no client is answered early.

The committer

A single thread, the committer, owns the store. The handlers of requests do not touch the store at all. Each hands its write to the committer through a channel and waits for the result.

Listing 6.3 The imports of kvgroup/src/main.rs. kvgroup/src/main.rs

use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::mpsc::{self, Receiver, Sender};
use std::thread;

use axum::Router;
use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, State};
use axum::http::header::ETAG;
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::put;
use clap::Parser;
use kv::{Change, Store, StoreError, Write, Written};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, write_condition};
use tokio::net::TcpListener;
use tokio::sync::oneshot;

Listing 6.4 The options, and a write on its way to the committer. kvgroup/src/main.rs

#[derive(Parser)]
struct Options {
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// The largest number of writes stored with one wait for the disk.
    #[arg(long, default_value_t = 256)]
    batch: usize,
}

/// A write on its way to the committer, with the channel for its result.
struct Request {
    write: Write,
    reply: oneshot::Sender<Result<Written, StoreError>>,
}

Listing 6.5 The committer thread. kvgroup/src/main.rs

/// The committer: the one thread that owns the store. It takes the write
/// that has waited longest, adds every write that arrived meanwhile, and
/// stores them together.
fn run_committer(mut store: Store, requests: Receiver<Request>, limit: usize) {
    let (mut batches, mut writes) = (0u64, 0u64);

    while let Ok(first) = requests.recv() {
        // The first request, and whatever else is already waiting.
        let mut batch = vec![first];

        batch.extend(requests.try_iter().take(limit - 1));

        let (changes, replies): (Vec<Write>, Vec<_>) = batch
            .into_iter()
            .map(|request| (request.write, request.reply))
            .unzip();

        batches += 1;
        writes += changes.len() as u64;

        // A handler that has gone away no longer wants its result.
        store
            .commit(changes)
            .into_iter()
            .zip(replies)
            .for_each(|(result, reply)| drop(reply.send(result)));

        if batches.is_multiple_of(200) {
            let average = writes as f64 / batches as f64;
            println!("{writes} writes in {batches} batches, {average:.1} writes per batch");
        }
    }
}

This loop is the whole of group commit. recv blocks until a write arrives. try_iter then yields every write that is already in the channel, without waiting for more, up to a limit of 256 for the batch. While commit is waiting for the disk, new writes accumulate in the channel, and the next turn of the loop takes all of them.

Giving the store to 1 thread has a further merit. Since no other thread touches it, the store needs no lock, and the question of what 2 concurrent writers might do to the map does not arise. Concurrency has been confined to the channel, where it is handled by a well-tested library, and the logic of the store remains sequential.

Listing 6.6 Handing a write to the committer, and the HTTP handlers. kvgroup/src/main.rs

/// Hands a write to the committer and waits for its result.
async fn submit(committer: &Sender<Request>, write: Write) -> Result<Written, ApiError> {
    let (reply, result) = oneshot::channel();
    let stopped = || ApiError::Unavailable("the committer has stopped".into());

    if committer
        .send(Request { write, reply })
        .is_err()
    {
        return Err(stopped());
    }

    let Ok(outcome) = result.await else {
        return Err(stopped());
    };

    Ok(outcome?)
}

async fn put_key(
    State(committer): State<Sender<Request>>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;

    let write = Write {
        change: Change::Put {
            key,
            value: body.to_vec(),
        },
        condition: write_condition(&headers)?,
    };

    let written = submit(&committer, write).await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(committer): State<Sender<Request>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;

    let write = Write {
        change: Change::Delete { key },
        condition: write_condition(&headers)?,
    };

    submit(&committer, write).await?;

    Ok(StatusCode::NO_CONTENT)
}

Listing 6.7 Starting the committer and the server. kvgroup/src/main.rs

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let (store, applied, discarded) = Store::open(&options.data, true)?;

    println!("recovered {applied} records, cut {discarded} bytes from the end of the log");

    let (committer, requests) = mpsc::channel();
    thread::spawn(move || run_committer(store, requests, options.batch));

    let routes = Router::new()
        .route("/v1/keys/{key}", put(put_key).delete(delete_key))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(committer);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Measurements

kvbench runs a number of clients simultaneously. Each client sends its writes in succession and waits for every answer before sending the next.

Listing 6.8 The imports of kvbench/src/main.rs. kvbench/src/main.rs

use std::error::Error;
use std::fs;
use std::path::PathBuf;
use std::thread;
use std::time::Instant;

use clap::Parser;
use ureq::Agent;

Listing 6.9 Writing from several clients at once. kvbench/src/main.rs

#[derive(Parser)]
struct Options {
    #[arg(long)]
    server: String,

    /// The number of clients that write at the same time.
    #[arg(long, default_value_t = 8)]
    clients: u32,

    /// The number of writes each client makes.
    #[arg(long, default_value_t = 200)]
    writes: u32,

    #[arg(long)]
    acked: PathBuf,
}

/// The writes of one client. It returns the keys that were acknowledged.
fn client(server: &str, number: u32, writes: u32) -> Vec<String> {
    let agent = Agent::new_with_defaults();
    let mut acknowledged = Vec::new();

    for write in 1..=writes {
        let key = format!("key-{number:03}-{write:06}");
        let address = format!("{server}/v1/keys/{key}");

        if agent
            .put(&address)
            .send(format!("value of {key}"))
            .is_ok()
        {
            acknowledged.push(key);
        }
    }

    acknowledged
}

fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let (server, writes) = (options.server.as_str(), options.writes);
    let started = Instant::now();

    let acknowledged: Vec<String> = thread::scope(|scope| {
        let clients: Vec<_> = (1..=options.clients)
            .map(|number| scope.spawn(move || client(server, number, writes)))
            .collect();

        clients
            .into_iter()
            .flat_map(|client| client.join().unwrap_or_default())
            .collect()
    });

    let seconds = started.elapsed().as_secs_f64();
    let rate = acknowledged.len() as f64 / seconds;

    fs::write(&options.acked, acknowledged.join("\n") + "\n")?;

    println!(
        "{} clients: {} writes acknowledged in {seconds:.2} s, {rate:.0} writes per second",
        options.clients,
        acknowledged.len()
    );

    Ok(())
}

$ ./target/release/kvd --data /var/tmp/kvd &
$ ./target/release/kvbench --server http://127.0.0.1:7001 --clients 64 --writes 12 --acked acked.txt
64 clients: 768 writes acknowledged in 4.27 s, 180 writes per second

$ ./target/release/kvgroup --data /var/tmp/kvgroup &
$ ./target/release/kvbench --server http://127.0.0.1:7001 --clients 64 --writes 400 --acked acked.txt
64 clients: 25600 writes acknowledged in 5.13 s, 4993 writes per second
25535 writes in 800 batches, 31.9 writes per batch

Several things can be read from the table. A single client gains nothing, since it sends its next write only when the previous one has been answered, and there is consequently never a second write with which a wait could be shared. Group commit improves throughput under concurrency and does nothing for the latency of an individual write, which remains 1 wait for the disk.

The size of a batch is about half the number of clients: 4 of 8, 32 of 64, 120 of 256. The explanation is that the clients fall into 2 groups that alternate. While 1 half is waiting in the batch that is being stored, the other half has just been answered and is sending its next writes, which form the following batch.

And the assumption of chapter 1 is confirmed with something to spare. The estimate supposed 20 writes for each wait and 3700 writes per second. With 64 clients the node achieved 31.9 writes for each wait and 4993 writes per second.

A crash in the middle of a batch

The server is killed while 64 clients are writing. The program kvd of chapter 2 then opens the same data directory, which incidentally demonstrates that the 2 programs use the same format for the log.

$ ./target/release/kvgroup --data /var/tmp/kvgroup &
$ ./target/release/kvbench --server http://127.0.0.1:7001 --clients 64 --writes 2000 --acked acked.txt &
$ sleep 2; kill -9 <pid of kvgroup>
64 clients: 9341 writes acknowledged in 2.74 s, 3412 writes per second
$ ./target/release/kvd --data /var/tmp/kvgroup &
recovered 9366 keys from 9366 records, cut 0 bytes from the end of the log
$ ./target/release/kvctl verify --server http://127.0.0.1:7001 --acked acked.txt
9341 acknowledged, 9341 found, 0 missing

Every one of the 9341 acknowledged writes is present. The log contains 25 more, which are the batch that had reached the disk when the process was killed and whose answers had not yet been sent. These are the durable and unacknowledged writes of chapter 2, the third case of its figure of crash points, now occurring 25 together. Group commit has not altered the guarantee in any respect. It has enlarged the number of writes that may be in that uncertain state at the moment of a crash from 1 to the size of a batch, and each of the 25 clients concerned resolves the uncertainty in the usual way, by repeating its request.

What is still missing

The complete programs

Store::commit is part of kv/src/store.rs, which is listed in full in chapter 2.

Listing 6.10 The complete kvgroup/src/main.rs. kvgroup/src/main.rs

//! Command kvgroup is the storage node of chapter 2 with group commit: one
//! thread writes the log, and it stores every write that is waiting with a
//! single wait for the disk.
//!
//! Run it from code/ with:
//!
//!     cargo run --release -p kvgroup -- --listen 127.0.0.1:7001 --data /var/tmp/kvgroup

use std::error::Error;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::mpsc::{self, Receiver, Sender};
use std::thread;

use axum::Router;
use axum::body::Bytes;
use axum::extract::{DefaultBodyLimit, Path, State};
use axum::http::header::ETAG;
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::put;
use clap::Parser;
use kv::{Change, Store, StoreError, Write, Written};
use kvd::api::{ApiError, MAX_VALUE, etag, valid_key, write_condition};
use tokio::net::TcpListener;
use tokio::sync::oneshot;

#[derive(Parser)]
struct Options {
    #[arg(long, default_value = "127.0.0.1:7001")]
    listen: SocketAddr,

    #[arg(long, default_value = "data")]
    data: PathBuf,

    /// The largest number of writes stored with one wait for the disk.
    #[arg(long, default_value_t = 256)]
    batch: usize,
}

/// A write on its way to the committer, with the channel for its result.
struct Request {
    write: Write,
    reply: oneshot::Sender<Result<Written, StoreError>>,
}

/// The committer: the one thread that owns the store. It takes the write
/// that has waited longest, adds every write that arrived meanwhile, and
/// stores them together.
fn run_committer(mut store: Store, requests: Receiver<Request>, limit: usize) {
    let (mut batches, mut writes) = (0u64, 0u64);

    while let Ok(first) = requests.recv() {
        // The first request, and whatever else is already waiting.
        let mut batch = vec![first];

        batch.extend(requests.try_iter().take(limit - 1));

        let (changes, replies): (Vec<Write>, Vec<_>) = batch
            .into_iter()
            .map(|request| (request.write, request.reply))
            .unzip();

        batches += 1;
        writes += changes.len() as u64;

        // A handler that has gone away no longer wants its result.
        store
            .commit(changes)
            .into_iter()
            .zip(replies)
            .for_each(|(result, reply)| drop(reply.send(result)));

        if batches.is_multiple_of(200) {
            let average = writes as f64 / batches as f64;
            println!("{writes} writes in {batches} batches, {average:.1} writes per batch");
        }
    }
}

/// Hands a write to the committer and waits for its result.
async fn submit(committer: &Sender<Request>, write: Write) -> Result<Written, ApiError> {
    let (reply, result) = oneshot::channel();
    let stopped = || ApiError::Unavailable("the committer has stopped".into());

    if committer
        .send(Request { write, reply })
        .is_err()
    {
        return Err(stopped());
    }

    let Ok(outcome) = result.await else {
        return Err(stopped());
    };

    Ok(outcome?)
}

async fn put_key(
    State(committer): State<Sender<Request>>,
    Path(key): Path<String>,
    headers: HeaderMap,
    body: Bytes,
) -> Result<Response, ApiError> {
    valid_key(&key)?;

    let write = Write {
        change: Change::Put {
            key,
            value: body.to_vec(),
        },
        condition: write_condition(&headers)?,
    };

    let written = submit(&committer, write).await?;
    let status = if written.created {
        StatusCode::CREATED
    } else {
        StatusCode::NO_CONTENT
    };

    Ok((status, [(ETAG, etag(written.version))]).into_response())
}

async fn delete_key(
    State(committer): State<Sender<Request>>,
    Path(key): Path<String>,
    headers: HeaderMap,
) -> Result<StatusCode, ApiError> {
    valid_key(&key)?;

    let write = Write {
        change: Change::Delete { key },
        condition: write_condition(&headers)?,
    };

    submit(&committer, write).await?;

    Ok(StatusCode::NO_CONTENT)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let (store, applied, discarded) = Store::open(&options.data, true)?;

    println!("recovered {applied} records, cut {discarded} bytes from the end of the log");

    let (committer, requests) = mpsc::channel();
    thread::spawn(move || run_committer(store, requests, options.batch));

    let routes = Router::new()
        .route("/v1/keys/{key}", put(put_key).delete(delete_key))
        .layer(DefaultBodyLimit::max(MAX_VALUE))
        .with_state(committer);

    let listener = TcpListener::bind(options.listen).await?;
    println!("listening on {}", listener.local_addr()?);

    axum::serve(listener, routes).await?;

    Ok(())
}

Listing 6.11 The complete kvbench/src/main.rs. kvbench/src/main.rs

//! Command kvbench writes keys to a server from several clients at once and
//! prints the number of writes per second. Every acknowledged key is added
//! to a file, in the format that `kvctl verify` reads.
//!
//! Run it from code/ with:
//!
//!     cargo run --release -p kvbench -- --server http://127.0.0.1:7001 --clients 64 --writes 200 --acked acked.txt

use std::error::Error;
use std::fs;
use std::path::PathBuf;
use std::thread;
use std::time::Instant;

use clap::Parser;
use ureq::Agent;

#[derive(Parser)]
struct Options {
    #[arg(long)]
    server: String,

    /// The number of clients that write at the same time.
    #[arg(long, default_value_t = 8)]
    clients: u32,

    /// The number of writes each client makes.
    #[arg(long, default_value_t = 200)]
    writes: u32,

    #[arg(long)]
    acked: PathBuf,
}

/// The writes of one client. It returns the keys that were acknowledged.
fn client(server: &str, number: u32, writes: u32) -> Vec<String> {
    let agent = Agent::new_with_defaults();
    let mut acknowledged = Vec::new();

    for write in 1..=writes {
        let key = format!("key-{number:03}-{write:06}");
        let address = format!("{server}/v1/keys/{key}");

        if agent
            .put(&address)
            .send(format!("value of {key}"))
            .is_ok()
        {
            acknowledged.push(key);
        }
    }

    acknowledged
}

fn main() -> Result<(), Box<dyn Error>> {
    let options = Options::parse();
    let (server, writes) = (options.server.as_str(), options.writes);
    let started = Instant::now();

    let acknowledged: Vec<String> = thread::scope(|scope| {
        let clients: Vec<_> = (1..=options.clients)
            .map(|number| scope.spawn(move || client(server, number, writes)))
            .collect();

        clients
            .into_iter()
            .flat_map(|client| client.join().unwrap_or_default())
            .collect()
    });

    let seconds = started.elapsed().as_secs_f64();
    let rate = acknowledged.len() as f64 / seconds;

    fs::write(&options.acked, acknowledged.join("\n") + "\n")?;

    println!(
        "{} clients: {} writes acknowledged in {seconds:.2} s, {rate:.0} writes per second",
        options.clients,
        acknowledged.len()
    );

    Ok(())
}

Further reading

• D. DeWitt and others, “Implementation Techniques for Main Memory Database Systems”, SIGMOD 1984, which introduced group commit.
• J. Gray and A. Reuter, Transaction Processing: Concepts and Techniques, Morgan Kaufmann, 1992.

Rate Limiter: System Design

A service that accepts requests from the outside world has no say in how many of them arrive. Its capacity is fixed by the machines it runs on, while the demand placed on it is decided by other people’s programs, and whenever demand exceeds capacity for more than a moment the service fails in a way that does not distinguish between the guilty and the innocent: the requests of a well-behaved client wait in the same queue as the requests of the client that caused the overload, and they time out together. A rate limiter gives the service back some of the control it lacks. It attaches an allowance to each caller, counts what each caller sends, and refuses whatever exceeds the allowance before the service has spent any work on it, so that a problem which would have been shared by everybody becomes a refusal addressed to the caller responsible.

Stated like this, the idea seems to leave little to design, and yet 2 parts of it turn out to be subtle. The first is the meaning of the allowance itself, because a phrase such as “5 requests per second” admits several readings that differ sharply in what they let through, and each reading leads to a different algorithm. The second is the counting, which is trivial on 1 machine and becomes a small distributed systems problem as soon as the limiter runs on several machines that each see only a part of a caller’s traffic. This chapter takes the 2 in that order. It examines 5 algorithms by following the same 11 requests through each of them, and then builds the system around the one we choose.

Why a service limits its callers

Consider an API server that can answer 1000 requests per second and serves 200 clients, each of which normally sends a few requests per second. One day a client is released with a defect in its error handling, so that a failed call is retried immediately and without pause, and this single client begins to send 5000 requests per second. The server has no means of setting these requests aside, since it cannot know who sent a request until it has accepted the connection and read the request, and by then the cost has been paid. Its queue lengthens and the response time of every client grows with it. Some of the other clients, finding their calls slow, give up and try again, which adds to the queue the very requests that were already in it. Within a minute or so the 199 clients that did nothing wrong are receiving errors on account of the 1 that did, and the operators of the service are looking at a failure whose cause lies in a program they do not own and cannot fix.

With a rate limiter in front of the server, the same defect has a very different outcome. The limiter recognises the caller, finds that it has exhausted its allowance, and answers with a short refusal that costs a small fraction of what a real response would have cost. The defective client receives a great many refusals, which is an accurate reflection of its behaviour, and the remaining clients are served as usual and never learn that anything was wrong.

Protection against overload is the best known reason for limiting callers, and there are 2 others that a design usually has to serve at the same time. One is fairness: a large customer running a nightly export may stay well within what the service can bear and still consume so much of it that smaller customers are crowded out, and an allowance per customer is the simplest way of dividing capacity among them. The other is the control of cost and abuse, where the concern is what each individual request makes possible. If every call costs money to answer, an unlimited caller is an unlimited bill, and if every call is a guess at a password, a login form that permits 5 attempts per minute for an account turns guessing into a hopeless undertaking, whereas a form without a limit will patiently evaluate millions of guesses a day. In this last case the limit is a security control as much as a protection, a distinction that matters later when we decide how the limiter should behave when it breaks.

It is worth separating the rate limiter from a neighbouring mechanism with which it is often confused. Load shedding is what a service does when it is close to overload and drops a share of its incoming requests, without regard to who sent them, in order to keep answering the rest. A rate limiter acts earlier and on different grounds, since it refuses a particular caller for exceeding that caller’s own allowance, and it does so even when the service has capacity to spare. The 2 complement each other, and large systems generally have both.

What “5 requests per second” can mean

The phrase “each client may send 5 requests per second” has the appearance of a complete specification, and the easiest way to see that it is incomplete is to imagine 3 clients and to ask of each whether it has kept to the limit.

Nobody would dispute that client A is within the limit. Client B has also sent 5 requests in a second, and in fact far fewer than 5 per second on average, although the server received all 5 at the same instant and had to work on them concurrently. Client C sent 5 requests in the first second of the clock and 5 in the next, so that each calendar second contains exactly the permitted number, and nevertheless the server received 10 requests within 200 ms.

Clients B and C expose 2 questions on which the phrase is silent. The first concerns the burst, by which we mean the number of requests that may arrive together. A limit of 5 per second does not say whether the 5 may come in the same instant or must be spread across the second, and there is no universally right answer, because services differ in what they can tolerate. A web API usually ought to permit short bursts, since that is how honest clients behave: a page is opened and issues 6 calls at once, and then nothing happens while a person reads. A service that feeds a device or a downstream system able to handle only one thing after another may need its requests evenly spaced, and would be harmed by exactly the burst that the web API welcomes.

The second question concerns the second itself. “Per second” may refer to the seconds of the clock, each beginning when the previous one ends, or it may refer to every span of 1000 ms regardless of where the span begins. Client C satisfies the first reading and violates the second, and the difference between the 2 readings is a factor of 2 in the load the server may have to carry.

Each of the 5 algorithms that follow amounts to a particular pair of answers to these 2 questions. They are best understood as different definitions of the limit that happen to be cheap to compute, and the reason no one of them has displaced the others is that the right definition depends on what is being protected.

A limit also depends on a third thing, which the phrase names without defining, namely the client. The limiter must count requests under some identity, and the identity under which requests are added together is called the key of the limit. It may be an API key, a user id, an IP address, or some combination of these. The choice of key deserves a section of its own after the algorithms, because a poorly chosen key will undo the work of a well chosen algorithm.

The traffic used in this chapter

So that the algorithms can be compared on equal terms, every one of them is given the same requests from a single client under the same limit of 5 requests per second. The client sends 11 requests within 650 ms, at the following times in milliseconds.

By any reading of the limit this client is sending too much, since it issues 11 requests in well under a second where 5 are permitted. The times have been chosen so that the requests lie across the 1 second mark of the clock, with 5 before it and 6 after it, because that is the arrangement on which the algorithms disagree most. One would expect a sound limiter to forward about 5 of the 11 and perhaps a few more as time goes by. The 5 algorithms forward 10, 5, 6, 8 and 9 of them respectively, and the reasons for these differences are the substance of the next 5 sections.

Each algorithm is accompanied by a picture that proceeds in steps, one step for each arriving request. The Next button advances to the following request and shows what the limiter has stored and what it decides. In these pictures a green dot marks a request that was forwarded to the API and a red cross marks a request that was refused.

Algorithm 1: the fixed window counter

The most direct way of enforcing 5 requests per second is to divide time into windows of 1 second, aligned with the clock, and to count the requests that arrive in the current window. The limiter then needs to remember only a single number for each client. Every forwarded request increases the number by 1, a request that finds the number already at 5 is refused, and when the clock passes into the next window the number begins again at 0.

At the end of the sequence 10 of the 11 requests have been forwarded. Requests 1 to 5 arrived between 600 and 950 ms and were counted in window 0, which they filled. At 1000 ms the clock moved into window 1, where the count stood at 0 again, so that requests 6 to 10 were forwarded as readily as the first 5 had been, and only request 11 met a full count.

The server therefore received 10 requests within 600 ms from a client whose limit is 5 per second, and it is important to see that the algorithm did precisely what it was defined to do. Each window contains exactly 5 forwarded requests. The rule is expressed in terms of the seconds of the clock, the client’s burst happened to fall across the boundary between 2 of them, and the rule has nothing to say about spans of time that do not coincide with its windows.

This behaviour is the defining weakness of the fixed window, and its extent can be stated exactly: a client can send up to twice the limit within an arbitrarily short span of time, by sending one full allowance at the end of a window and another at the start of the next. No ill intent is needed for this to occur. A great many programs do their work on a schedule, waking at the start of each minute or each hour, and a limit that is reset on the minute or the hour will meet such a program exactly at its boundary, day after day.

The synchronised reset has a second consequence that is less often noticed. Because the counts of all clients return to 0 at the same instant, all the clients that had been refused and were retrying succeed together at the start of the next window, and the server experiences a surge of traffic at every boundary that the limiter itself has organised.

These properties make the fixed window a poor choice for protecting a server against overload, since the boundary admits twice the load that the limit appears to promise. They matter much less when the window is long and the purpose is to meter usage, and this is where the algorithm is properly at home. Daily and monthly quotas are almost always fixed windows, for the good reason that customers readily understand a quota that “resets at midnight”, and a doubling of the daily quota around midnight is spread over enough time to be harmless. The cost is as low as a limiter’s can be: a count and the number of the window to which it belongs.

Algorithm 2: the sliding window log

The fixed window goes wrong because its windows are fastened to the clock, and the obvious remedy is to fasten the window to the request: when a request arrives, the limiter should look back exactly 1 second from that moment and count what it forwarded during that span, wherever the span happens to begin.

To do this the limiter has to know when each recent request arrived, and so it keeps a log, a list of the arrival times of the requests it has forwarded. On each new request it first discards the times that lie more than 1 second in the past. If fewer than 5 times then remain, the request is forwarded and its own arrival time is appended to the log, and otherwise the request is refused.

This time only 5 requests are forwarded. From 1000 ms onwards each arriving request looks back over the preceding second and finds the 5 entries from 600 to 950 ms still within it, and the fact that the clock has passed the 1 second mark is of no significance to the rule. The oldest entry, 600, leaves the log at 1600 ms, and until then nothing more can pass.

The sliding window log is exact in a sense that none of the other algorithms can match. Whichever span of 1 second one chooses to examine, beginning at any instant whatever, it contains at most 5 forwarded requests, and this is surely what most people have in mind when they say “5 per second”.

Exactness is paid for in memory. Where the fixed window stores 1 number for a client, the log stores a timestamp for every forwarded request up to the limit. For a limit of 5 per second that is 5 timestamps and of no concern. For a limit of 10000 requests per hour it is as many as 10000 timestamps for a single client, about 80 KB, and for a million active clients about 80 GB. The cost thus rises in proportion to the size of the limit, with the awkward result that the most generous limits are the most expensive to enforce.

The description above contains a decision that is easily overlooked, in the words “the requests it has forwarded”. One has to decide whether a refused request should also be entered in the log. In the version shown it is left out, so that a client which keeps on sending is served again as soon as its old entries expire. Some implementations record refused requests as well, and under that rule a client that never stops sending is never served again, since its own refused requests keep the log permanently full. That is a much harsher policy, and occasionally it is the one intended, for instance as a defence against password guessing. What must be avoided is arriving at it unintentionally, which is what happens in an implementation that appends every request to the log first and checks the count afterwards.

The log is therefore the natural choice for limits that are small and must be strict, such as 5 login attempts per minute or 3 password resets per hour, where its memory cost is negligible and its exactness is valuable. It is a poor choice for large limits applied to a large number of clients.

Algorithm 3: the sliding window counter

We now have one algorithm that is cheap and wrong at the boundary and another that is exact and expensive, and it is natural to look for something between them. The sliding window counter keeps the 2 counts that a fixed window limiter would have kept for 2 consecutive windows, and from them it estimates what the log would have reported.

The limiter stores the count of the current window and the count of the window before it. When a request arrives, the second that precedes it overlaps both windows: it includes the whole of the current window up to now, together with the final portion of the previous window. If the current window began 250 ms ago, the preceding second extends 750 ms back into the previous window and so covers 75% of it.

The limiter cannot know how the requests of the previous window were distributed within that window, having kept only their number, and so it makes the simplest assumption available, which is that they were spread evenly. On that assumption 75% of the window contains 75% of its requests, and the number of requests in the preceding second is estimated as

estimate = previous count × share of the previous window still covered + current count

A request is forwarded when the estimate, with the request itself included, does not exceed the limit.

Requests 6 to 10 are refused, as they were by the log. Immediately after the boundary the preceding second still covers nearly all of the previous window, so that nearly all of its 5 requests are counted, and the burst that the fixed window admitted across the boundary does not get through.

Request 11, at 1250 ms, is forwarded, and it is in this decision that the estimate shows what it is. The limiter calculates 5 × 0.75 + 0 = 3.75 and concludes that there is room for another request. In reality all 5 of the earlier requests arrived between 600 and 950 ms and therefore lie entirely within the preceding second, so that the true count is 5, and the log, which knew this, refused the same request. The estimate came out too low because the requests were concentrated at the end of their window, while the formula took them to be evenly spread.

The error can equally fall on the other side. When the requests of the previous window were concentrated at its beginning, the formula continues to count some of them after they have become more than 1 second old, and the limiter then refuses a request that the log would have forwarded. For traffic that is reasonably smooth the error is small in either direction, typically a few percent of the limit, and the worst case is the kind of burst at the very end of a window that we constructed deliberately. Even in the worst case the client gains a fraction of one allowance, where the fixed window would have given it a whole additional allowance.

In exchange for this modest inaccuracy the algorithm needs 2 counts per client whatever the size of the limit. For a limit of 10000 per hour that is 2 numbers where the log requires up to 10000 timestamps, and this trade explains why the sliding window counter is so often chosen for public APIs with large limits and many clients. It should not be used where a small limit has to be enforced exactly.

Algorithm 4: the token bucket

The 3 algorithms considered so far all count requests within a window, and they treat the burst only by implication. The remaining 2 start from a different picture, in which the burst is represented explicitly.

Imagine a bucket that holds tokens, each token being a permission to send 1 request. Tokens are added to the bucket at a steady rate, which in our case is 1 every 200 ms or 5 per second. The bucket has a capacity, here 5 tokens, and a token that arrives when the bucket is already full is discarded. A request is forwarded if it can remove 1 token from the bucket, and it is refused if the bucket is empty.

The 2 numbers that describe a token bucket correspond directly to the 2 questions that a limit has to answer. The rate determines how many requests per second a client can go on sending indefinitely, since over a long period it cannot spend tokens faster than they arrive. The capacity determines how many requests a client can send in the same instant after a period of quiet, since a client that has been silent finds the bucket full and is free to empty it at once. After such a burst the bucket is empty and the client is held to the rate until it pauses long enough for the bucket to fill again.

That the rate and the capacity can be chosen independently is the particular strength of this algorithm. An API whose clients typically open a page with 20 calls and then fall silent can be given a rate of 5 per second and a capacity of 20, which serves those clients well and still bounds what they can sustain. A window counter has a single number at its disposal and cannot express such a limit.

Here 8 requests are forwarded. The first 5 consume the contents of the full bucket. While they are arriving, and in the 300 ms that follow, 650 ms elapse in all, during which 3 further tokens are added, and requests 6, 7 and 10 each find one of them.

This result repays some attention, because it regularly surprises people who have configured a token bucket and then measured it. The limit is described as “5 per second”, and the limiter forwarded 8 requests in 600 ms. The explanation is that a token bucket with rate r and capacity b forwards at most b + r × t requests in any span of t seconds, namely the contents of the bucket together with whatever arrives during the span. Over 1 second, with the numbers used here, this comes to 5 + 5 = 10. Whoever sets the capacity equal to the rate and expects a strict limit of 5 in every second will therefore observe up to twice that number in the second following a quiet period. If the server truly cannot bear more than 5 requests in any second, then the capacity must be made smaller, or the rate lower, or the algorithm of the next section is the one to use.

How the bucket is stored

The picture suggests a timer that drops a token into every bucket every 200 ms. For a million clients that would mean 5 million updates per second, almost all of them applied to the buckets of clients that are sending nothing at all, and no practical limiter works in this way.

The timer can be dispensed with entirely. The limiter stores 2 numbers for each client: the number of tokens in the bucket, and the time at which that number was last brought up to date. When a request arrives, the limiter works out how much time has passed since that moment, adds the tokens that would have arrived in the interval, and caps the result at the capacity. It then removes 1 token if one is available and stores the new number together with the present time. The steps of the picture show this arithmetic in captions such as “50 ms have passed, which adds 0.25 tokens”.

The same captions show why the number of tokens has to be a fraction. A request that arrives 50 ms after its predecessor has earned a quarter of a token, and if that quarter were rounded down to nothing on every request, a client sending every 50 ms would never accumulate a token at all.

Storing the bucket in this way has a welcome consequence for memory. A bucket that has not been touched for capacity ÷ rate seconds, which is 1 second in our example, must be full, and a full bucket is indistinguishable from a bucket that has never been used. The limiter may therefore delete the entry of any client that has been silent for that long and create a full bucket when the client next appears, so that memory is occupied only by the clients that are active at the moment.

With 2 numbers of state, a few arithmetic operations per request and a burst that the operator sets directly, the token bucket is the usual choice for the general protection of an API. It is unsuitable only where the traffic reaching the service must be evenly spaced.

Algorithm 5: the leaky bucket

The token bucket passes a burst through to the service, within the bound set by its capacity. The leaky bucket does the opposite: it accepts a burst and releases it slowly.

The name comes from a bucket with a small hole in its base. However quickly water is poured in, it leaves through the hole at a constant rate, and if water continues to be poured in faster than it leaks, the bucket eventually fills and the excess runs over the side.

Applied to requests, the bucket is a queue. An arriving request joins the back of the queue, and requests leave the front of it at a fixed pace, 1 every 200 ms, to be forwarded to the API. The queue has a fixed number of places, 5 in this example, and a request that arrives when all of them are occupied is refused.

The leaky bucket accepts 9 of the 11 requests, more than any algorithm except the fixed window, and yet it places a lighter burden on the API than any of the others. The lower axis of the picture explains the apparent paradox. The requests arrived bunched together within 650 ms, and they departed at intervals of 200 ms over a period of 1.6 seconds, so that the API at no time received 2 requests less than 200 ms apart.

This is what the leaky bucket is for. The other 4 algorithms decide only whether a request is forwarded, and a request that is forwarded goes on its way at the instant of its arrival. The leaky bucket decides in addition when the request is forwarded, and in doing so it alters the shape of the traffic as well as its volume.

The smoothing has a price, and it is the client who pays it. A request that joins the queue behind 4 others waits 800 ms before it so much as reaches the API. For a person waiting on a web page such a delay is worse than a prompt refusal followed by a retry. For a background job that sends email through a provider which accepts exactly 5 messages per second it is ideal, since nothing is refused, nothing has to be retried, and the provider’s limit is observed to the letter.

A queue of waiting requests also holds resources, because each waiting request keeps a connection open and occupies memory in the limiter. For this reason the leaky bucket is found more often inside a system, between a service and something slower that it feeds, than at the public edge of an API.

The leaky bucket as a meter

The literature uses the name “leaky bucket” for a second algorithm as well, and the 2 are frequently confused. In this other version nothing waits. The bucket has a level, which rises by 1 with each accepted request and falls at a constant rate, and a request that would raise the level above the capacity is refused, while every accepted request is forwarded immediately.

This “leaky bucket as a meter” makes exactly the same decisions as a token bucket. The level of the water is the number of tokens missing from a full token bucket, and the one drains at the rate at which the other fills. When a document speaks of a leaky bucket and nothing is being queued, what it describes is a token bucket seen upside down, and everything said in the previous section applies to it, the burst included.

How the queue is stored

The queueing version, too, needs less state than its picture suggests. Because requests depart at a fixed pace, the limiter does not have to consult a list of them in order to reach a decision. It is enough to keep 1 number, the time of the next free turn to depart. An arriving request is assigned that turn, and the number is advanced by 200 ms. If the turn lies 1 second or more in the future, then 5 requests must already be waiting, and the new arrival is refused. The captions of the picture report this number at each step, as in “the next free turn to leave is at 1400 ms”.

Deciding from a single stored time in this manner is an old technique. It was standardised in the 1990s for telephone and data networks under the name generic cell rate algorithm, usually abbreviated GCRA, and it is used in present-day rate limiters for the same reason it was attractive then, which is that it requires 1 number of state per client.

The algorithms compared

The table below collects the results for the 11 requests of this chapter under the limit of 5 per second.

The simulation in the book’s repository sends 3 further kinds of traffic through the same 5 algorithms. In its results the column “most in any second” gives the largest number of requests that reached the API within any span of 1000 ms, which is the quantity the server actually has to withstand.

The first block of this table carries a warning for anyone who tests a limiter. A single burst does not distinguish the algorithms at all, since every one of them forwards 5 requests out of 20. Their differences emerge only when traffic continues for a while or crosses a boundary, and a limiter that has passed a quick test with one burst may therefore still behave quite differently from what its author expects once it is in production.

The second and third blocks show, as measurements, the 2 effects described earlier. The fixed window alone admits 10 requests in a second when they are arranged around the boundary. The token bucket admits 9 in the first second of sustained traffic, which is its full bucket of 5 and the 4 tokens that arrived meanwhile. In the third block the sliding window counter forwards 13 requests where the exact log forwards 15, its estimate having erred on the strict side on this occasion.

The properties of the 5 algorithms can be summarised as follows.

In choosing among them it is usually enough to consider what the limit is for. If whatever stands behind the limiter can work only at a fixed pace, the leaky bucket is the appropriate choice, with the understanding that requests will wait. If the limit is small and has to be exact, as with a handful of attempts per minute on something sensitive, the sliding window log should be used, its memory cost being negligible at that size. If the limit is a long quota that customers read about in a price list, a fixed window that resets at a predictable time is both the simplest and the easiest to explain. For the general protection of an API, which is the commonest case, the choice lies between the token bucket, when clients are bursty and a bounded burst does no harm, and the sliding window counter, when the limit is meant to hold over every window to within a small error.

The design in the remainder of this chapter uses a token bucket with a rate of 5 per second and a capacity of 10 for each client. This is the most widespread arrangement at an API gateway, being cheap, well suited to the way real clients behave, and equipped with a burst that the operator controls directly.

Use cases

The limiter has 2 kinds of user. Clients encounter it on every call they make, usually without noticing, and operators set the limits it enforces.

Functional requirements

Non-functional requirements

Non-functional requirement 2 calls for some justification, since in it we deliberately choose the weaker of the consistency models introduced in chapter 1. The limiter is a protective layer, and the consequences of its being slightly wrong are mild: if a limit of 100 requests per second occasionally admits 103, the API is still protected and no data anywhere is incorrect. To make every count exact, all the limiter nodes would have to agree on a client’s count before any of them answered a request, and agreement of that kind is what chapters 3 and 4 obtain by means of a leader and a majority, at the cost of a disk write and a network round trip. Incurring that cost on every call to the API would make the protection more expensive than much of what it protects. We therefore accept counts that are slightly wrong for short periods, and concentrate on bounding how wrong they can be.

Non-functional requirement 1 is likewise a decision that could have gone the other way, and for certain limits it does. The section on failures takes it up again.

Core entities

The 2 entities differ in character, and the difference shapes the design. A Limit is configuration. It changes when an operator changes it, perhaps a few times a day, it must survive restarts, and it is read enormously more often than it is written, so each limiter node keeps a copy of the limits in memory and refreshes it periodically. A Bucket is running state that changes with every request and can be lost without real harm, since a bucket that has been lost is created again full, and the only effect is that 1 client is granted 1 additional burst.

Capacity estimate

Arithmetic ahead. The buckets of 1 million clients need about 64 MB of memory. They fit on 1 node, and we run 3 for availability.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Memory per bucket	client id 40 bytes + tokens 8 + time 8 + overhead 8	64 bytes
Memory for all clients	1000000 × 64 bytes	64 MB
Checks per second	1 per request	50000
The same clients with a sliding window log and a limit of 10000 per hour	1000000 × 10000 × 8 bytes	80 GB

The last row of the calculation is included for contrast. It shows that the choice of algorithm can change the memory required by a factor of more than 1000.

The API

A rate limiter offers clients no API of its own. It makes itself known through the responses of the API it protects, by means of 1 status code and a small number of headers.

A request within the limit

GET /v1/orders HTTP/1.1
Host: api.example
Authorization: Bearer client-42

HTTP/1.1 200 OK
RateLimit-Limit: 5
RateLimit-Remaining: 3
RateLimit-Reset: 1

The 3 headers inform the client of its rate, of the number of requests it could still send at this moment, and of the number of seconds after which its allowance will be full again, so that a client which reads them can slow down before it is ever refused. The names of such headers differ from one API to another, and an API should document the ones it sends.

A request over the limit

GET /v1/orders HTTP/1.1
Host: api.example
Authorization: Bearer client-42

HTTP/1.1 429 Too Many Requests
Retry-After: 2
Content-Type: application/problem+json

{
  "type": "/problems/rate-limited",
  "title": "Too many requests",
  "status": 429,
  "detail": "5 requests per second are allowed"
}

429 Too Many Requests is the status code that HTTP provides for this situation, and Retry-After is the standard header for stating how many seconds the client should wait.

An operator sets a limit

PUT /v1/limits/client-42 HTTP/1.1
Host: limiter.internal
Content-Type: application/json

{ "rate": 5, "capacity": 10 }

HTTP/1.1 200 OK
Content-Type: application/json

{ "client": "client-42", "rate": 5, "capacity": 10 }

PUT replaces the whole limit of the named client, and repeating the request leaves the limit as it was after the first.

High-level design

One request, step by step

The steps follow the arrows of figure 7.12.

1. A client sends GET /v1/orders with its credentials. The gateway verifies the credentials and obtains the client id, client-42, from them.
2. The gateway asks the rate limiter whether client-42 may make 1 request to this route now.
3. The limiter finds the limit of client-42 in its own memory. It then asks the counter store to bring the client’s bucket up to date for the time that has passed and to remove 1 token, as a single operation, and the store replies with the decision and the number of tokens remaining.
4. If a token was removed, the gateway forwards the request to an API server and adds the RateLimit headers to the response. If the bucket was empty, the gateway answers 429 itself, and the API servers never see the request.

The order of the first 2 steps is deliberate. The gateway authenticates the caller before consulting the limiter because the limiter needs to know whose request it is counting. It follows that requests which fail authentication must be limited in some other way, usually by IP address, since otherwise an attacker could submit login attempts without limit and never come within reach of a limit that is keyed by client.

Where the limiter runs

The design places the limiter at the gateway. A limiter can in fact run in any of 3 places, and the 3 protect against different things.

These are complements, and a mature system uses all of them. A carefully written client library paces its own calls with a leaky bucket so that it seldom sees a 429. The gateway enforces the limit regardless, because a service cannot make its safety depend on the good conduct of its clients. And an expensive operation deep inside a service may carry a tighter limit of its own, based on knowledge that the gateway does not have.

Whose requests are counted together

The key of a limit determines which requests are added up, and a badly chosen key causes the limiter to penalise the wrong people while leaving the intended target untouched.

The IP address is the only identity available before a caller has authenticated, and it is a poor one in both directions. A university, a company or a mobile network may place thousands of people behind a single address, all of whom then share one allowance and are refused together when any one of them is busy. An attacker in control of 10000 addresses, on the other hand, enjoys 10000 allowances. A limit by IP address is therefore best kept generous and regarded as a coarse defence for unauthenticated traffic.

Once the caller is known, an API key or a user id is a far better key, since it identifies the party actually responsible for the traffic. Combining it with the route gives finer control, at the cost of 1 bucket for every pair of client and route. A limit keyed on nothing at all, which counts every request to a service, protects the service’s total capacity and provides no fairness whatever, because a single client is free to use it up.

In practice several limits are applied to each request, and the request is forwarded only if every one of them permits it. A representative set is shown below.

Layered limits introduce a subtlety of their own. Suppose a request is admitted by the second limit and then refused by the third. A token has already been removed from the client’s general bucket on behalf of a request that was never served, and a client that keeps running into the limit on exports will find that its general allowance drains away as well. The correct procedure is to examine all the applicable limits first and to remove tokens only when every one of them has room, which with a shared store means that all the buckets concerned are checked and updated within the same single operation.

Counting on many machines

Everything said so far has assumed that 1 limiter sees all the requests of a client. At 50000 requests per second, and for the sake of availability, there will be several limiter nodes behind a load balancer, and the requests of client-42 will be distributed among them. The question of where the buckets are to be kept has 3 answers in common use.

Each node keeps its own buckets

The simplest arrangement is for every limiter node to keep buckets in its own memory and to consult nobody. Nothing could be faster, and there is no dependency that might fail.

The arrangement is also wrong by a constant factor. With 3 nodes and traffic spread evenly among them, each node sees a third of the client’s requests and allows the full limit, so that the client obtains 3 times its allowance. One might give each node a third of the limit, but this is correct only for as long as the traffic is in fact evenly spread, which a load balancer does not undertake to ensure for any individual client: a client that keeps 1 connection open sends everything to 1 node and receives a third of what it is entitled to. The size of the error also changes whenever a node is added or removed.

The load balancer can be instructed to send all the requests of a client to the same node, for instance by hashing the client id as chapter 1 hashes keys. Local buckets are then exact. The weakness shows itself when a node is added or fails, since the clients that are moved to another node begin there with a fresh and full bucket. For a coarse protective limit this is often acceptable, and a number of large systems operate in exactly this way.

A shared counter store

In the second arrangement all the limiter nodes keep the buckets in 1 shared in-memory store, so that a request works on the same bucket whichever node it reaches. This is the arrangement our design adopts.

It brings with it a problem that did not exist on a single machine. Updating a token bucket consists of 3 steps: reading the bucket, computing the new number of tokens, and writing the result back. When 2 limiter nodes carry out these steps for the same client at the same time, the steps of one may be interleaved with the steps of the other.

Each node saw 1 token and forwarded a request, and the bucket lost only 1 token. No error is reported anywhere. The limit has been exceeded, and it will be exceeded more often the more traffic the client sends, so that the fault is at its worst in exactly the circumstances for which the limiter was installed.

The remedy is to make the 3 steps indivisible. The limiter sends the store a single request whose meaning is “bring this bucket up to date, remove 1 token if there is one, and report the outcome”, and the store carries out the whole of it before attending to any other request for the same bucket. In-memory stores provide for this in 2 ways: by executing a small script inside the store as one unit, or by offering a compare-and-set operation that writes a value only if it has not changed since it was read, as described in chapter 1. The fixed window is the easy case here, since its entire update is “add 1 and return the new value”, which every store of this kind offers as a single operation.

The same reasoning extends to time. If every limiter node supplies its own clock reading with its request, then 2 nodes whose clocks differ by 20 ms will cause the recorded time of a bucket to jump backwards and forwards, and the refill calculation will sometimes be presented with a negative interval and sometimes credit the same 20 ms twice. This is avoided by letting the store apply its own clock to every bucket, so that all the updates of a bucket are measured against a single clock.

What the shared store costs is 1 network round trip per request, well under 1 ms within a data centre, together with a new dependency, since the limiter can now do its work properly only while the store is reachable.

Local tokens obtained in batches

The third arrangement retains the shared store and avoids consulting it for every request. A limiter node asks the store for a batch of tokens on behalf of a client, say 10, and serves the next 10 requests of that client from its own memory, returning to the store only when the batch is spent.

The number of calls to the store is thereby divided by the batch size, and the price is a loss of accuracy at the margins. Tokens lying unused in the memory of one node are unavailable to the others, so that a client may be refused at one node while another still holds tokens for it, and a node that crashes takes its unspent tokens with it. The larger the batch, the fewer the calls and the coarser the count.

For this design we choose the shared store. A single in-memory store handles 50000 checks per second comfortably, and non-functional requirement 0 leaves room for the round trip.

What a refused caller should be told

A refusal is a message addressed to a program, and whether the limiter reduces the load on the system or multiplies it depends on what that program does next.

Consider a client that responds to 429 by retrying at once. It is refused again, and retries again, and the limiter finds itself answering many times more requests from this client than it did before the limit was reached. Each refusal is cheap, but the total is not negligible, and none of it is of any use. The Retry-After header exists to prevent this, by telling the client the earliest moment at which a further attempt can succeed. With a token bucket that moment is easily calculated, since the limiter knows how many tokens are lacking and how quickly they arrive.

A second effect arises when many clients are involved. Suppose that 1000 clients are refused within the same second, perhaps because a brief outage caused all of them to retry together, and that each is told Retry-After: 2. Exactly 2 seconds later 1000 requests arrive at the same instant. This is known as a thundering herd, and the remedy lies with the clients, which should wait for the stated time and then for an additional random interval, called jitter, so that their retries are spread out. Retries, lengthening pauses and jitter are treated in detail in chapter 13.

The limiter’s own boundary can also be made less abrupt. Some systems maintain 2 thresholds, adding a warning header to responses once the lower one is passed and refusing only at the higher. An operator who is about to introduce a new limit can first run it in a mode that merely records which requests it would have refused, and after a week the record will show whether the limit would have harmed legitimate clients, before any client has actually been turned away.

When the limiter itself fails

The limiter depends on the counter store, and the store may become unreachable. Each request then confronts the limiter with a decision for which no good answer exists: it cannot count, and it must nevertheless either forward the request or refuse it. To forward it is to fail open, which keeps the API available and leaves it unprotected for the duration. To refuse it is to fail closed, which keeps the API protected and makes it unavailable to everyone.

For the general protection of an API we fail open. The limiter exists in order to keep the API available, and if a fault in the limiter’s own store were to make the API unavailable, the limiter would have brought about the very outage it was installed to prevent. It should also be remembered that at most moments no client is misbehaving, so that an hour without limits will usually pass without incident.

Failing open need not mean having no limit at all. When the store cannot be reached, each limiter node can fall back on buckets in its own memory, in the manner of the first arrangement above, with every node enforcing its share of each limit. The count is coarser than usual, and a client that goes badly wrong is still contained.

For some limits the decision must be reversed. A limit on login attempts, on password resets, or on calls each of which costs money is a control on security or expenditure, and to forward requests without limit in those cases is worse than to refuse them. Such limits fail closed, and a sound design names them one by one, since a single switch for the whole limiter cannot serve both kinds.

The store itself can be run as a leader with followers, as in chapter 3, so that the loss of 1 machine does not make it unreachable. After a change of leader the new leader may lack the most recent updates, because an in-memory store commonly replicates without waiting for a majority to confirm. For buckets this is harmless, amounting to a few extra tokens for a few clients, and it is the eventual consistency of non-functional requirement 2 as it appears in practice.

The design, deployed

At 50000 requests per second, no component of the design is a single machine. Clients reach 1 of 3 zones, each with a pair of load balancers in front of 10 API gateways. Every gateway asks a limiter node in its own zone, and the 9 limiter nodes are interchangeable, since none of them holds a bucket. All of them read and update the buckets in a single counter store, which runs as a leader with 2 followers in the other zones. Only requests that pass reach the 100 API servers, which are the reason for the whole arrangement and by far the largest tier.

How real systems do it

The algorithms of this chapter are all in production somewhere, and the published accounts agree closely with what was said of each.

Stripe has described 4 limiters that it runs in front of its API. The most important is a request rate limiter for each user, built as a token bucket whose state is held in an in-memory store, as in this chapter. Beside it is a limiter on the number of requests a user may have in progress at once, which catches expensive requests that a rate alone would miss. The other 2 are load shedders, which act on the state of the whole system and reserve capacity for critical requests, and which are the load shedding that the opening section distinguished from rate limiting.

Cloudflare chose the sliding window counter for the limiter at its edge and published a measurement of its accuracy. Over 400 million requests from 270000 sources, the estimate from 2 counters wrongly allowed or refused 0.003% of requests, with an average difference of 6% between the estimated and the true rate. This supports the statement above that the error is small for real traffic, and it should be read with the caution that it describes 1 network’s traffic.

The generic cell rate algorithm comes from the standards for ATM networks of the 1990s, and it is the basis of several widely used limiter libraries. Envoy, a proxy commonly used as a gateway, offers both arrangements described under counting on many machines: a token bucket local to each proxy, and a call to a shared rate limit service, and the 2 can be combined, with the local one as a first line of defence in front of the shared one. The response headers shown in the API section follow a draft of the IETF, which has changed between versions and is not yet a standard, whereas the status 429 and the header Retry-After are standardised.

One technique in common use is missing from this chapter. A well-behaved client can limit itself adaptively, lowering its own rate when it sees refusals and raising it slowly when it does not, which the AWS client libraries do with a token bucket for retries. Chapter 13 describes the same idea from the side of the caller.

Mistakes to avoid

Failures

Trade-offs

The simulation

References

• P. Tarjan, “Scaling your API with rate limiters”, Stripe engineering blog, 2017.
• J. Desgats, “How we built rate limiting capable of scaling to millions of domains”, Cloudflare blog, 2017.
• ATM Forum, Traffic Management Specification Version 4.0, 1996, for the generic cell rate algorithm.
• RFC 6585, Additional HTTP Status Codes, 2012, for status 429.
• RFC 9110, HTTP Semantics, 2022, for Retry-After.
• IETF HTTP API working group, “RateLimit header fields for HTTP”, Internet-Draft.
• Envoy proxy documentation, “Global rate limiting” and “Local rate limiting”.

Terms introduced in this chapter

Unique ID Generator: System Design

Almost every record that a system stores needs a name by which the rest of the system can refer to it. An order is mentioned in an invoice, a message in a reply, a user in a thousand other rows, and each of these references works only if the name it uses belongs to exactly 1 record and will never belong to another. Such a name is an identifier, or id, and for as long as a system consists of a single database the question of where ids come from hardly arises, because the database keeps a counter and hands out the next number with every insert.

The question becomes interesting when records are created on many machines at once. The counter that served 1 database so well now has to be shared, and a shared counter is a place where every creation of every record in the system must queue, and a machine whose failure stops all of them. This chapter is about producing identifiers without that shared point. We examine 4 ways of doing it, of which the last, an id composed of a timestamp, a node number and a counter, is the one we then design in full, and we give particular attention to the ways in which each scheme can quietly produce the same id twice.

What an identifier has to do

Uniqueness is the one property that cannot be compromised, and it is worth being precise about its scope. An id has to differ from every other id of the same kind that has ever been issued or ever will be, on any machine, including machines that have since been replaced and machines that were restarted a moment ago and have forgotten what they did before. Most of the difficulties in this chapter come from that last clause, since it is easy to design a generator that never repeats itself while it is running and considerably harder to design one that does not repeat itself after it has lost its memory.

Several further properties are desirable, and the 4 schemes differ mainly in which of them they provide. An id should be small, because it is stored not only in its own record but in every record that refers to it and in every index over those references, so that the difference between 8 bytes and 16 is multiplied many times over. It should be obtainable without a network call, since a call adds latency to every insert and a dependency that may be unavailable. And it is very useful if ids increase with time, so that an id issued later is larger than one issued earlier.

This last property deserves an explanation, because its value is not obvious until one considers how a database stores an index. An index keeps its entries in sorted order, divided among pages of fixed size, and an insert has to write the page into which the new entry falls. If new ids are always larger than existing ones, every insert falls on the last page, which is therefore the only page that needs to be in memory, and the index grows by appending. If new ids are scattered at random across the whole range, each insert falls on a different page, nearly every page of the index is being modified at any moment, and once the index has outgrown memory each insert costs a read from disk.

Time-ordered ids bring smaller conveniences as well. Sorting by id is sorting by age, so that “the newest 20 orders” can be read from the end of the primary index without a separate index on the creation time, and an engineer looking at 2 ids in a log can tell at a glance which came first.

A counter in 1 database

The traditional source of ids is a counter kept by the database, usually called an auto-increment column or a sequence. Its ids are as small as ids can be, they are strictly increasing, and uniqueness is guaranteed by the simplest possible argument, namely that 1 component gives out each number once.

The cost lies entirely in the phrase “1 component”. Every insert anywhere in the system must reach that database, which bounds the rate at which the whole system can create records by what 1 machine can do, and adds a network round trip for every service that is not in the same data centre. If the database is unavailable, nothing can be created at all. A system that has divided its data among many shards, as the key-value store of chapter 1 does, would be reintroducing through its ids the single point of failure that sharding was meant to remove.

A well-known refinement keeps the counter and removes some of the dependence on 1 machine. 2 databases each maintain a counter, the first giving out odd numbers and the second even numbers, so that either can serve requests when the other is down and their ids never coincide. The idea extends to n databases, each stepping by n from a different starting point. Its weakness is rigidity. The number of databases is built into every id that has been issued, and to add a third database to a pair one must change the step on machines that are in service, without ever letting 2 of them issue the same number during the change. The ids are also no longer ordered in time, since one database may be far ahead of the other.

Random identifiers

At the opposite extreme is the idea of using no coordination whatsoever. Each machine picks a number at random from a range so large that 2 machines are extremely unlikely ever to pick the same one. The best-known form is the universally unique identifier, or UUID, a 128-bit value, and in its version 4 all but 6 of those bits are random, which leaves 122 random bits.

Whether “extremely unlikely” is unlikely enough can be calculated. Among n ids drawn at random from 2 to the power 122 possible values, the probability that at least 2 are equal is approximately n squared divided by 2 to the power 123. A system that generates a million ids every second for 30 years produces about 10 to the power 15 ids, and the probability of a single collision among all of them comes to roughly 1 in 10 million. For nearly every purpose this is as good as certainty, provided the random numbers are truly random. That proviso is the practical risk of the scheme: a generator seeded from the clock, or 2 virtual machines cloned from the same image with the same internal state, will produce identical “random” ids with a probability that has nothing to do with the calculation above.

Random ids have 2 real costs. They occupy 16 bytes where a counter needs 8, and that is paid again in every reference and every index. And they are scattered uniformly over their range, which is the worst case for index locality shown in the figure above. Many teams have adopted random UUIDs as primary keys while their tables were small and have discovered the second cost only when the index no longer fitted in memory and insert throughput fell sharply.

The response to this has been to give up some random bits in exchange for order. Version 7 of the UUID standard places a 48-bit timestamp in milliseconds at the front and fills the rest with random bits, so that ids generated later sort after ids generated earlier, to the precision of a millisecond, while 2 ids from the same millisecond are distinguished by 74 random bits. This keeps the great advantage of random ids, which is that a machine needs nothing from anybody in order to make one, and removes the scattered inserts. What it cannot remove is the size, and if ids must fit in 64 bits, as they must wherever they are to be stored as an ordinary integer column, there is not enough room for both a useful timestamp and enough random bits to make collisions negligible.

Blocks of numbers

A third scheme returns to the counter and reduces the cost of sharing it. A coordinator holds a single durable counter. A node that needs ids asks the coordinator for a block of them, 1000 at once for example, upon which the coordinator advances its counter by 1000 and the node gives out the numbers of its block one by one from memory. The node contacts the coordinator again only when the block is used up.

Uniqueness follows from the fact that no number belongs to 2 blocks, and the coordinator is consulted once per 1000 ids, so that its load and its influence on latency are a thousandth of what they were with a plain shared counter. If the coordinator fails, every node can continue until its current block is exhausted, which with a suitable block size gives operators ample time to restore it.

2 properties are lost. The ids contain gaps, because a node that crashes or is restarted discards the unused remainder of its block, and anyone who assumed that ids are consecutive, for instance in order to count records by subtracting one id from another, will be misled. And the ids are no longer ordered in time across nodes, since a slow node may still be giving out numbers from an old block long after a busy node has moved on to much larger ones. Where neither property is needed, this is a simple and dependable scheme, and it yields the smallest ids of any distributed method.

Time, node number and counter

The fourth scheme obtains uniqueness without any communication while a node is running, keeps the id within 64 bits, and preserves the order of time. It does so by composing each id from 3 numbers that the node already has at hand: the current time in milliseconds, the node’s own number, and a counter of the ids the node has made in the current millisecond. The arrangement became widely known through a generator that a large social network published in 2010 under the name Snowflake, and ids of this form are often called by that name.

The argument for uniqueness considers the ways in which 2 ids might be related, and shows that in each case they differ in at least 1 field. If they were made by different nodes, they differ in the node number, provided every node has a number of its own. If they were made by the same node in different milliseconds, they differ in the time, provided the node’s clock does not return to a millisecond it has already used. And if they were made by the same node in the same millisecond, they differ in the counter, provided the node does not issue more ids in a millisecond than the counter can count. The 3 provisos are the whole of the engineering problem. Each corresponds to a way in which the scheme fails if it is neglected, and the 3 sections after the next deal with them in turn.

The order of the fields within the id is what makes ids sortable by time. Because the time occupies the most significant bits, comparing 2 ids as integers compares their times first, and only when the times are equal does the comparison fall through to the node number and the counter.

The sizes of the fields are a budget of 63 bits to be divided, and the division shown is the common one. 41 bits of milliseconds last for 69 years, counted from a starting date that the service chooses for itself, and choosing a recent date in place of the conventional 1970 gives the scheme its full 69 years from the day it goes into service. 10 bits allow 1024 nodes. 12 bits allow each node 4096 ids per millisecond, or about 4 million per second. A system with few nodes and a very high rate might prefer 8 bits of node and 14 of counter, and a system spread over many small machines the reverse, and the only firm rule is that the division must be fixed before the first id is issued, since ids made under 2 different divisions cannot safely coexist.

When a millisecond is full

A node that has issued 4096 ids within 1 millisecond has no counter values left for that millisecond. The correct response is to wait until the clock has advanced and to continue with a counter of 0, which delays the caller by less than a millisecond. In the simulation for this chapter, 3 nodes are each asked for 5000 ids in every one of 100 milliseconds, and each makes exactly 4096 per millisecond, so that 1228800 ids are made in all, every one of them different, and 271200 requests have to wait.

The response to avoid is letting the counter overflow. An overflowing counter carries into the adjacent field, which in this layout is the node number, and the node then issues ids that appear to come from its neighbour and may coincide with ids that the neighbour has really made.

The clock

The second proviso, that a node’s clock never returns to a millisecond it has already used, is not one that a computer’s clock satisfies without help. The clock of a machine drifts, by an amount that is typically some tens of milliseconds a day, and a time service corrects it periodically by comparison with better clocks. When the machine’s clock is found to be ahead, the correction may set it back, and for a brief period afterwards the clock shows times that it has shown before.

The defence shown in these steps is for the node to remember the time of the last id it issued and to refuse to issue an id for any earlier time. A step backwards of a few milliseconds is then waited out. A step of seconds or minutes indicates that something is wrong with the machine or its time service, and it is better for the node to report itself unhealthy, so that callers go elsewhere and an operator is alerted, than to stall silently for as long as the step lasts.

This defence depends on the node’s memory of its last id, and that memory is lost when the node restarts. Consider a node that issues an id at a time its clock calls 5000, crashes, and is running again 20 ms later with a clock that has meanwhile been corrected backwards by 50 ms. The restarted node remembers nothing, reads 4970 from its clock, and issues ids for milliseconds that its former self has already used. 2 precautions are taken against this. The node can record on disk, every few seconds, a time somewhat in the future before which it promises not to issue ids after a restart, and on starting it waits until its clock has passed the recorded time. Alternatively, or in addition, a starting node can wait for longer than the largest clock correction that is considered possible before it issues its first id. Both cost a short delay at startup and nothing afterwards.

It is tempting to avoid the problem by using the machine’s monotonic clock, which the operating system guarantees never to run backwards. That clock, however, measures time from an arbitrary starting point, commonly the moment the machine was switched on, and its readings are therefore meaningless on any other machine and after any reboot. It can serve to detect that the wall clock has jumped, by comparing the progress of the 2, and it cannot replace the wall clock as the source of the timestamp.

The node number

The first proviso requires that no 2 nodes that are making ids ever hold the same number. Of the 3 provisos it is the one whose violation is most damaging, because 2 nodes with the same number and reasonably accurate clocks produce identical ids continuously, and nothing in an id or in either node can reveal that this is happening. The duplicates surface later and elsewhere, as rejected inserts if the ids are primary keys with a uniqueness constraint, or as 2 different records answering to 1 name if they are not.

Node numbers therefore need an authority. The usual arrangement is the one of our design, in which a starting node obtains its number from a coordinator of the kind described in chapter 1, which records the numbers in use and gives each to 1 node only. The alternatives are to write the numbers into the configuration of each machine, which is workable for a handful of machines administered by hand and dangerous wherever machines are created from a common image, or to derive the number from something assumed to be unique to the machine, such as part of its network address, which holds only as long as the assumption does.

Reusing a number requires care for the same reason that restarting a node does. When a node has been retired and its number is given to a new node, the new node must not issue ids for any millisecond that the old one used, and since the 2 clocks may disagree, the coordinator should withhold a released number for longer than the largest clock difference that is thought possible. With 1024 numbers available and far fewer nodes in service, there is rarely any pressure to reuse a number quickly.

What an id reveals

An id that is handed to the outside world, in a URL or an API response, tells its recipient whatever its construction encodes, and it is as well to decide in advance whether that is acceptable.

An id of the time, node and counter form discloses exactly when the record was created, to the millisecond, to anyone who knows the layout, and the layout of such ids is common knowledge. For a public post this hardly matters, while for a medical record or an account it may. Sequential ids from a counter disclose something of commercial interest: a competitor who places an order today and another a week later, and subtracts the 2 order numbers, learns how many orders the business took in that week.

Ordered ids are also guessable. Given one valid id, neighbouring values are likely to be valid as well, and a service that returns a record to anyone presenting its id, without checking that the caller is entitled to see it, can be read from end to end by counting. The proper remedy is the authorisation check, and an id should never be relied upon as a secret. Where an unguessable reference is truly required, as in a link for resetting a password, it should be a separate random token, kept apart from the id of the record.

The order of ids is approximate

Since the time is the leading field, it is natural to read the order of 2 ids as the order in which 2 events occurred, and within a single node that reading is correct. Between nodes it holds only to the accuracy of their clocks. If the clock of node 1 is 15 ms ahead of the clock of node 2, an id that node 2 makes now is smaller than an id that node 1 made 10 ms ago, although it was made later.

Ids of this kind are accordingly described as roughly sorted: any 2 ids made further apart in time than the largest difference between clocks are in the right order, and ids made closer together than that may be in either order. This is entirely adequate for the purposes mentioned at the start of the chapter, such as index locality and listing recent records. It is inadequate as evidence that one event caused or preceded another. A system that needs to know the true order of 2 writes, for example in order to decide which of them should prevail, must obtain that order from a mechanism that establishes it, such as the sequence numbers that a leader assigns in chapter 3, and must not infer it from identifiers.

The 4 schemes compared

The row on silent failure explains why the last scheme, for all its advantages, asks for more care than the others. The counter and the block schemes derive uniqueness from a single durable number and cannot produce a duplicate without losing that number. The schemes that need no coordination purchase their independence with assumptions about the environment, and when an assumption fails, no component is in a position to notice.

Our design uses time, node number and counter, because the requirements below ask for 64-bit ids, rough ordering by time, and continued operation when every other component is down, and no other scheme provides all 3.

Use cases

Services ask for ids, at a very high rate and with no interest in how they are made. Operators have a single duty, which is to see that every node receives a number of its own.

Functional requirements

Non-functional requirements

The consistency models of chapter 1 do not apply to this design in the ordinary sense, because the nodes share no data that could be consistent or inconsistent. The uniqueness of ids follows from their layout. The only fact that nodes must agree on is which number belongs to which node, and that fact is established once, at startup, by a coordinator that is strongly consistent.

Core entities

Capacity estimate

Arithmetic ahead. 1 node can make 4096 ids per millisecond, and the 41 bits of time last 69 years.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Ids per node per millisecond	2 to the power 12	4096
Ids per node per second	4096 × 1000	4096000
Nodes	2 to the power 10	1024
Years of time	2 to the power 41 ms ÷ (1000 × 60 × 60 × 24 × 365)	69.7

The API

The service has a single operation. Callers that need ids in very large numbers often link the generator into their own process as a library, each such process being a node with its own number, and thereby avoid the network altogether.

Get an id

POST /v1/ids HTTP/1.1
Host: ids.example

HTTP/1.1 201 Created
Content-Type: application/json

{ "id": "8388636672" }

We send the id as a string, because JSON numbers lose precision above 2 to the power 53.

The id is returned as a string for a reason that has caught many APIs unprepared. JSON itself places no limit on the size of a number, but the most widely used JSON parsers, those of web browsers, read every number as a 64-bit floating-point value, which represents integers exactly only up to 2 to the power 53. A larger id is silently rounded to a nearby value, so that 2 different ids can arrive in the browser as the same number.

High-level design

One request, step by step

The steps follow the arrows of figure 8.8.

1. When an id node starts, it asks the coordinator for a node number. The coordinator gives out each number once. This is the only time a node needs anyone else.
2. A service sends POST /v1/ids to any id node. The node reads its clock, takes the next counter value for that millisecond, and joins time, node number and counter into 1 number.

It is the absence of arrows between the id nodes that characterises this design. Once a node holds its number, it needs nothing from the coordinator or from any other node, and the availability of the service as a whole is that of whichever single node a caller manages to reach.

The design, deployed

The id nodes are spread over 3 zones and placed behind a load balancer only as a convenience for callers, since any node can answer any request. Each zone is given a range of node numbers, so that the number in an id also says where the id was made, which helps when a duplicate has to be traced. The coordinator is consulted when a node starts and at no other time. Callers with very high rates do not use the network at all: they link the generator into their own processes, each of which then holds a node number of its own.

How real systems do it

The scheme of time, node number and counter is the one that Twitter announced in June 2010 under the name Snowflake, with exactly the division used here: 1 unused bit, 41 bits of milliseconds, 10 bits for the machine and 12 for a sequence. Twitter adopted it when it moved its tweets from a single database to a sharded store and could no longer rely on the database’s counter. The layout has been widely copied with different divisions. Instagram described a variant in 2012 with 41 bits of time, 13 bits for the logical shard and 10 for a sequence, generated inside the database itself, so that no separate service is needed. Discord uses the original layout with its own starting date.

The counter in a pair of databases, stepping by 2, is the scheme that Flickr published as its ticket servers. Blocks of numbers handed out by a coordinator are how many database sequences are cached in practice.

Version 7 of the UUID, with a millisecond timestamp in its first 48 bits, was standardised in RFC 9562 in 2024, and it exists because of the problem of index locality described at the start of this chapter.

There is an important case in which the advice of this chapter is reversed. In a store that partitions its data by ranges of keys, as Spanner, CockroachDB and Bigtable do, keys that increase with time send every insert to the last range, and that 1 range and the machines holding it become a bottleneck however large the cluster. The documentation of such stores warns against time-ordered primary keys for exactly this reason, and recommends keys whose leading bits are spread out, by a hash or by reversing the bits of a sequence. The locality that helps an index on 1 machine becomes a hot spot on many, and which of the 2 effects governs depends on how the store beneath partitions its keys.

Mistakes to avoid

Failures

Trade-offs

The simulation

References

• R. King, “Announcing Snowflake”, Twitter engineering blog, 2010.
• Instagram engineering blog, “Sharding & IDs at Instagram”, 2012.
• K. Elliott-McCrea, “Ticket Servers: Distributed Unique Primary Keys on the Cheap”, Flickr code blog, 2010.
• RFC 9562, Universally Unique IDentifiers (UUIDs), 2024.
• Google Cloud Spanner documentation, “Schema design best practices”, on avoiding monotonically increasing keys.

Terms introduced in this chapter

Distributed Cache: System Design

A database answers a question by doing work: it locates pages on disk, follows an index, perhaps joins several tables and sorts the result. When the same question is asked again a moment later, it does all of that work again, although nothing has changed and the answer will be the same. A cache is a component that remembers recent answers, keeps them in memory, and returns them without consulting the database, so that a question which took 20 ms the first time takes less than 1 ms on every subsequent occasion and places no load on the database at all.

That description contains the whole appeal of caching and conceals nearly all of its difficulty. The remembered answer is a copy, and from the moment it is made the original is free to change without the copy’s knowledge. A cache also has less room than the data it copies from, so that it must continually decide what to forget. And because a system that has a cache soon comes to rely on it, the moments at which the cache cannot help, when an entry has just expired or a cache node has just been lost, are precisely the moments at which the database receives a load it has not carried for months. This chapter treats each of these in turn: what to keep, how a copy and its original drift apart and how far that can be controlled, and how a cache fails and takes other things with it.

What a cache is worth

The fraction of reads that a cache is able to answer is called its hit ratio, a read answered from the cache being a hit and a read that has to go on to the database a miss. Almost everything about the value of a cache follows from this single number, and the relationship is less linear than intuition suggests.

Suppose that a read from the cache takes 1 ms and a read from the database 20 ms. The average read then costs the hit ratio times 1 ms plus the miss ratio times 20 ms.

A cache that answers half of all reads has barely halved the average, since the slow half still dominates. At 90% the average has fallen to 2.9 ms, and at 99% to 1.19 ms. The more important effect, however, is on the database. Between a hit ratio of 90% and one of 99%, the reads reaching the database fall from 10% of the total to 1%, which is a tenfold reduction, and it is usually this reduction, more than the gain in latency, that a cache is installed to obtain. A database that could not possibly serve the whole read load directly is able to serve the 1% that the cache passes on.

The same arithmetic contains a warning that the rest of the chapter develops. A system whose database is sized for 1% of the read load is a system that cannot survive without its cache. If the hit ratio falls from 99% to 90%, the load on the database multiplies by 10, and if the cache is emptied altogether it multiplies by 100.

Whether a high hit ratio is attainable depends on the traffic. Caching works because requests are unevenly distributed, a small number of keys being asked for again and again while the great majority are asked for rarely. The simulation for this chapter sends a million requests for 100000 keys, with popularity concentrated on a few, to a cache of various sizes.

A cache holding a quarter of the keys answers nearly 3 quarters of the requests. With traffic more sharply concentrated than this, which is common, the same hit ratio would be reached with a much smaller cache, and with requests spread evenly over the keys a cache of 25% would answer only 25%. The distribution of real traffic should therefore be measured before a cache is sized, and no figure quoted from elsewhere, including the figures above, is a substitute.

Arranging a cache and a database

There are 4 established ways of placing a cache beside a database, which differ in who is responsible for filling the cache and in what order the 2 are written.

In the first and commonest arrangement, called cache-aside, the application manages everything. To read, it asks the cache, and on a miss it reads the database and stores the result in the cache for the benefit of later readers. To write, it updates the database and then removes the corresponding entry from the cache, so that the next reader will miss and fetch the new value. The cache is a passive store that knows nothing of the database. The great merit of this arrangement is that the cache is optional: if it is unreachable, the application reads the database directly and continues to work, only more slowly.

Read-through moves the same logic into the cache. The application asks the cache for a key, and the cache, on a miss, fetches the value from the database itself. The behaviour is that of cache-aside, with the advantage that the fetching code exists in 1 place, where every application shares it, and the disadvantage that the cache must be taught how to query the database.

In write-through, the application writes to the cache, and the cache writes to the database before confirming. Every write thus refreshes the cache, which consequently always holds the latest value of whatever has been written through it. The cost is that every write pays for 2 stores, and that entries are cached on writing whether or not anybody will read them.

Write-behind is write-through with the database write postponed. The cache confirms the write at once and passes it to the database later, usually in batches. Writes become very fast, and a burst of them is smoothed into a steady stream. The price is durability: writes that the cache has confirmed and has not yet passed on are lost if the cache node fails, which is the kind of loss that chapter 2 went to such lengths to prevent. Write-behind is appropriate for data whose occasional loss is tolerable, such as counters of page views, and for little else.

The design of this chapter uses cache-aside, because requirement by requirement it is the arrangement in which a failure of the cache does the least harm.

What to remove when memory is full

In steady operation a cache is always full, since it has less memory than there is data and something is inserted on every miss. Each insertion therefore requires an eviction, the removal of an existing entry, and the rule by which the victim is chosen determines the hit ratio that a cache of a given size achieves. The ideal rule would remove the entry that will not be needed for the longest time to come. As the future is unknown, practical rules estimate it from the past, and they differ in which aspect of the past they consult.

Least recently used

The rule known as least recently used, or LRU, removes the entry that has gone longest without being read or written. Its justification is the observation that requests come in clusters, so that something used a moment ago is likely to be used again soon, and something untouched for a long time is likely to remain so. The cache keeps its entries in order of last use, moves an entry to the front whenever it is used, and evicts from the back.

The final step exhibits the characteristic weakness of the rule. A had been requested 3 times in succession and was plainly the most popular key, and yet a single pass over 3 other keys, each requested once, was enough to push it out. In practice this happens when a job scans a large number of keys once, for a report or a backup: the scan passes through the cache, evicting the truly popular entries in favour of entries that will never be requested again, and the hit ratio collapses until ordinary traffic has restored the popular entries.

Least frequently used

Least frequently used, or LFU, consults how often an entry has been used and disregards when. Every entry carries a count of its uses, and the entry with the lowest count is evicted.

On this sequence the rule does better, since the count of 3 protects A from the single requests for B, C and D, and the last request is a hit. A scan can no longer displace popular entries, because the scanned keys never accumulate a count.

LFU has the opposite weakness. A count, once earned, lasts indefinitely, so that an entry which was requested 10000 times last month and has not been requested since continues to occupy its place, while a newly popular entry begins at 1 and is in danger of eviction before it has had time to prove itself. Practical versions therefore let the counts decay, halving them at intervals, so that popularity has to be maintained to be kept.

Other rules

First in, first out evicts the entry that was inserted longest ago, without regard to use. It needs no bookkeeping on reads, which makes it the cheapest rule, and it evicts popular and unpopular entries alike, which generally makes it the least effective.

Evicting an entry chosen at random sounds like no rule at all and performs surprisingly close to LRU on many workloads. A refinement that several in-memory stores have adopted is to pick a handful of entries at random and evict the least recently used among them. This approximates LRU closely while avoiding the ordered list that true LRU must update on every single read, an update which becomes a point of contention when many threads read the cache at once.

The most effective rules in current use combine recency and frequency. One widely adopted approach admits a new entry to the main part of the cache only if an approximate count of its past requests exceeds that of the entry it would displace, so that keys seen once, whether from a scan or from the long tail of unpopular keys, are turned away at the door. The details are beyond this chapter, and the principle is the one the 2 pictures above have already shown, namely that recency and frequency each catch what the other misses.

Our design uses least recently used, as most caches do. It is simple, it suits traffic in which popularity changes from hour to hour, and its weakness can be avoided by keeping scans away from the cache.

How a copy becomes stale

A cached value is stale when the database holds a newer one. Some staleness is inherent in caching and has to be accepted, and the purpose of the design is to bound it and to make sure that it is never permanent.

Under cache-aside, staleness is addressed by the rule that a writer, having updated the database, removes the entry from the cache. It may be asked why the writer does not place the new value in the cache directly, which would spare the next reader a miss. The reason is that 2 writers can then interfere. If writer 1 sets the database to 11 and writer 2 sets it to 12, and their subsequent writes to the cache happen to arrive in the opposite order, the database ends at 12 and the cache at 11. Removing the entry is immune to this, because removing something twice, in either order, leaves it removed.

The order of the 2 steps of a write also matters, the database being written before the entry is removed. If the entry were removed first, a reader arriving between the removal and the database write would miss, read the old value from the database, and put it back into the cache, where it would remain after the write had completed.

Even with both precautions observed, a stale value can become established, through an interleaving of a reader and a writer that is rare and entirely possible.

What makes this race instructive is that neither request did anything wrong. The reader followed the rule for reads and the writer the rule for writes, and the result is a cache that disagrees with the database and will go on disagreeing, since no further event is scheduled to correct it. The race needs a read to be delayed, between fetching from the database and storing in the cache, for longer than an entire write takes, which is unusual, and on a busy system unusual events occur many times a day.

There are 3 defences, of increasing strength. The first is to give every entry a time limit, discussed in the next section, so that even an entry that should have been removed disappears eventually. The second is for the writer to remove the entry a second time after a short delay, which catches a reader that stored its stale value just after the first removal. The third makes the reader’s store conditional. On a miss the cache hands the reader a token, the removal of an entry invalidates any token outstanding for it, and a store that presents an invalidated token is rejected. In the steps above, request 2’s removal would have invalidated the token of request 1, whose attempt to store 10 would then have failed, leaving the cache empty and correct.

Time limits

Every entry in a cache should carry a time to live, or TTL, after which the cache discards it regardless of anything else. The TTL is what converts every failure of invalidation, whether the race just described, a removal that was lost because the cache was briefly unreachable, or a code path that updates the database and forgets the cache, from a permanent error into a temporary one. It is the single mechanism that bounds staleness unconditionally, and for that reason it should be present even where explicit removal is believed to be complete.

Choosing the value is a judgement about the data. A long TTL gives a higher hit ratio and allows stale values to live longer, and a short one the reverse. The question to ask of each kind of data is how long a wrong answer could be shown before somebody is harmed. A product description may tolerate an hour, the number of items in stock a few seconds, and an account balance nothing at all, which is another way of saying that it should not be cached in this manner.

3 ways in which a cache fails under load

The failures considered so far concern correctness. The 3 considered next concern load, and each of them has brought down systems whose caches were functioning exactly as designed.

The stampede

When a popular entry expires or is removed, every request that arrives before it has been restored is a miss. For an entry that is read 500 times per second and takes 100 ms to recompute, some 50 requests will arrive during the recomputation, and under plain cache-aside every one of them queries the database for the same value.

If the query is expensive, the database slows down under 50 simultaneous copies of it, the recomputation takes longer, still more requests arrive during it, and the situation feeds on itself. This is called a stampede.

The principal remedy is to ensure that only 1 request recomputes a given entry. The first request to miss takes a short-lived lock on the key and proceeds to the database, and the others, finding the lock taken, wait briefly and read the cache again, by which time the value is usually there. A complementary technique avoids the miss altogether by refreshing popular entries shortly before they expire: a request that finds an entry close to its expiry may, with a probability that rises as the expiry approaches, recompute it early while all other requests continue to be served the existing value. A third approach is to serve the expired value for a short grace period while a single request fetches a fresh one in the background, which trades a little staleness for the complete absence of a stampede.

Penetration by keys that do not exist

A cache stores answers, and a request for something that does not exist has, on the face of it, no answer to store. Consequently every request for a nonexistent key misses the cache and reaches the database, which searches and finds nothing, and the next request for the same key does the same. An attacker who requests random ids, or merely a defective client that keeps asking for a deleted record, thus obtains unimpeded access to the database, the cache contributing nothing. This is known as cache penetration.

The straightforward remedy is to cache the absence. When the database reports that a key does not exist, the application stores a marker to that effect under a short TTL, and subsequent requests for the key are answered from the cache. Where the set of possible nonexistent keys is very large, as with random ids, the markers themselves could fill the cache, and a more economical structure is placed in front: a Bloom filter, which is a compact summary of the set of keys that do exist. It can state with certainty that a key is absent, and it occasionally reports an absent key as possibly present, in which case the request proceeds to the cache and the database as usual. A filter covering a billion keys occupies little more than a gigabyte.

The avalanche

The third failure arises when a large part of the cache becomes empty at once. This happens when many entries were loaded together and given the same TTL, for example by a job that warms the cache at startup, so that they all expire in the same instant. It happens on a larger scale when a cache node fails or is restarted. In either case the hit ratio falls abruptly, and by the arithmetic of the first section a fall from 99% to 90% multiplies the load on the database by 10.

Against synchronised expiry the remedy is to add a random amount to each TTL, so that entries loaded together expire over a spread of time. Against the loss of a node, the remedies are structural and are the subject of the next section. In both cases the database should be defended independently, by a limit on the number of concurrent queries that the application will send to it, so that an empty cache results in slow responses and some refusals while the database itself stays up. The rate limiter of chapter 7 and load shedding serve exactly this purpose.

A cache on many nodes

With 100 GB of entries and nodes of 16 GB, the cache has to be spread over several machines, and each key must be assigned to one of them in a way that every client computes identically. This is the problem that chapter 1 solved for the key-value store, and the solution is the same: a consistent hash ring on which each node owns many small ranges, so that adding or removing a node moves only the keys of that node.

The argument for consistent hashing is, if anything, stronger for a cache than for a store. A key that moves to a different node is, from the cache’s point of view, a key that has been forgotten, since the new node does not have it. Were keys assigned by taking a hash modulo the number of nodes, the addition of a single node would move nearly all keys, which is to say that it would empty the cache, with the consequences just described. On the ring, adding an eighth node to 7 moves about an eighth of the keys.

In this design the ring lives in a library inside each service, and the cache nodes are unaware of one another. A node holds entries, evicts and expires them, and that is all. There is no leader and no replication between nodes in the basic design, on the reasoning that every entry is a copy of something held durably elsewhere, so that the loss of a node loses nothing that cannot be fetched again.

That reasoning is sound with respect to data and incomplete with respect to load. The loss of 1 node in 8 turns an eighth of all reads into misses until those entries have been fetched again, and the database must be able to bear this. Where it cannot, each entry can be written to 2 nodes, the next one clockwise on the ring serving as a standby, at the cost of half the cache’s capacity. A related precaution concerns a node that returns after a restart with its memory empty: its entries can be loaded gradually by admitting only a fraction of its normal traffic at first, so that its misses arrive at the database as a trickle.

A last difficulty on many nodes is the hot key, a single entry so popular that the 1 node holding it is overloaded while the others are idle. No arrangement of the ring helps, since the ring spreads keys and this is 1 key. The usual answers are to keep a copy of such entries in the memory of each application process for a second or 2, in front of the distributed cache, or to store the entry under several derived keys on different nodes and have readers choose among them at random.

Use cases

Services read through the cache and inform it when the data behind an entry has changed. Operators add and remove nodes.

Functional requirements

Non-functional requirements

A cache is eventually consistent by its nature, and non-functional requirement 2 says so openly. For some period after a write, a reader may still receive the previous value, and the TTL of the entry is the bound on that period. The design does not attempt to remove this window. It attempts to make its length a deliberate choice for each kind of data, and to ensure, by means of the TTL, that no failure can extend it indefinitely.

Core entities

Capacity estimate

Arithmetic ahead. 100 GB at 16 GB of memory per node needs 7 nodes, and we run 8.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Memory needed	given	100 GB
Usable memory per node	16 GB × 0.9	14.4 GB
Nodes	100 ÷ 14.4	6.9, so 7
Reads per node	200000 ÷ 8	25000 per second

The API

Values are opaque bytes, as in the key-value store of chapter 1, and the time limit of an entry travels in a header.

Store a value for 60 seconds

PUT /v1/entries/user-42 HTTP/1.1
Host: cache.example
Cache-Control: max-age=60
Content-Length: 9

ada, 1815

HTTP/1.1 204 No Content

After 60 seconds the entry is gone, whether or not it was read.

Read a value that is not cached

GET /v1/entries/user-43 HTTP/1.1
Host: cache.example

HTTP/1.1 404 Not Found

Misses are ordinary answers: the caller reads the database and stores the result.

A miss is reported with 404, since from the point of view of HTTP the resource does not exist at that address. It is an ordinary outcome and no error, and the caller responds to it by reading the database and storing the result.

High-level design

One request, step by step

The steps follow the arrows of figure 9.8.

1. A service asks the cache library for user-42.
2. The library hashes the key on its ring and asks the 1 cache node that owns it. That node answers with the value, or with 404.
3. On 404, the service reads user-42 from the database, which is slower.
4. The service stores the value in the cache with a time limit, so that the next reader gets a hit.

A write follows the complementary path. The service updates the database first, and when the database has confirmed, it sends DELETE for the key to the cache node that owns it. If that DELETE fails because the node is unreachable, the entry survives with the old value until its TTL expires, which is the case the TTL exists for.

The design, deployed

The deployment has a feature that distinguishes it from most others in this book, which is the absence of a load balancer. Each of the 400 service instances contains the cache library with its copy of the ring, and sends every request directly to the 1 cache node that owns the key. The 8 nodes are spread over 3 zones. With a hit ratio of 99%, about 2000 of the 200000 reads per second go on to the database, whose followers serve them. The figure makes the dependence plain: the database tier is sized for the traffic that passes the cache, and would receive 100 times as much if the cache tier were lost.

How real systems do it

The most thorough public account of a cache at scale is Facebook’s paper on its use of memcached, and nearly every difficulty in this chapter appears in it with the remedy that Facebook adopted.

The stale value left by an overlapping read and write, and the stampede on a missing entry, are both solved there by 1 mechanism, the lease. On a miss, the cache gives the client a token that entitles it to fill the entry. A delete invalidates any outstanding token, so that a late fill with an old value is rejected, which is the conditional store described above. And the cache issues a token for a given key only at intervals, telling other clients that miss in the meantime to wait briefly and try again, which prevents the stampede. The paper reports that leases reduced the peak rate of database queries for a set of keys prone to stampedes from 17000 per second to 1300.

The paper also describes what it does about the avalanche. When a cache server fails, its keys are not moved to the remaining servers, which might then be overloaded in turn. Requests for them go to a small set of idle machines kept for the purpose, called the gutter. Invalidation is driven from the database’s own log of committed changes, which is more dependable than relying on every application to issue its deletes, and a cluster that starts empty is warmed by reading from a neighbouring cluster’s cache before it is allowed to reach the database.

On eviction, the rule of sampling a few entries and removing the least recently used among them is what Redis does by default, with 5 samples. The admission policy that compares the estimated frequency of a newcomer with that of the entry it would replace is TinyLFU, implemented in the Caffeine library. A study of the cache clusters at Twitter found, among other things, that for many workloads at realistic cache sizes plain first in, first out performed about as well as LRU, which has since led to a family of simple queue-based policies. The technique of refreshing an entry early with a probability that rises towards its expiry has been analysed formally and shown to be optimal in a certain sense.

Mistakes to avoid

Failures

Trade-offs

The simulation

References

• R. Nishtala and others, “Scaling Memcache at Facebook”, NSDI 2013.
• G. Einziger, R. Friedman and B. Manes, “TinyLFU: A Highly Efficient Cache Admission Policy”, ACM Transactions on Storage, 2017.
• J. Yang, Y. Yue and K. Rashmi, “A Large Scale Analysis of Hundreds of In-memory Cache Clusters at Twitter”, OSDI 2020.
• A. Vattani, F. Chierichetti and K. Lowenstein, “Optimal Probabilistic Cache Stampede Prevention”, VLDB 2015.
• Redis documentation, “Key eviction”, on approximated LRU.
• B. Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors”, Communications of the ACM, 1970.

Terms introduced in this chapter

Distributed Lock: System Design

Some operations must be performed by 1 party only. A nightly billing run that charges every customer should execute once, on 1 machine, a file that is being rewritten should not be rewritten by 2 processes whose changes would interleave, and of the replicas of a database exactly 1 should accept writes. Within a single process this need is met by a lock, an object that 1 thread holds while it works and that makes every other thread wait, and the operating system guarantees that the lock behaves as advertised.

A distributed lock attempts to offer the same service to processes on different machines, and it has to do so under conditions that a lock inside 1 process is spared. The holder of a distributed lock may crash without releasing it. It may be cut off from the network and continue working in the belief that it still holds the lock. It may stop for a while, through no fault of its own, and resume later with no awareness that time has passed. In each case the other participants can observe only that the holder has fallen silent, and they cannot tell which of these things has happened. The central result of this chapter is that under these conditions a lock service cannot, by itself, guarantee that only 1 process acts at any moment, however carefully it is built, and that the guarantee has to be completed by the resource which the lock protects.

What a lock is for

Locks are taken for 2 quite different reasons, and it is necessary to know which one applies before deciding how much machinery a lock deserves.

In the first case the lock serves efficiency. Its purpose is to avoid doing the same work twice: sending a reminder email from 2 machines, or having 5 servers recompute the same expensive report. If the lock occasionally fails and 2 processes do proceed together, the consequence is some wasted effort or a duplicated email, which is regrettable and nothing more. A lock of this kind may be simple and may be wrong once in a long while.

In the second case the lock serves correctness. Here 2 processes acting together would corrupt something: a file written by both, an account debited twice, 2 database nodes both accepting writes as leader. When such a lock fails, the result is lost or damaged data, and “once in a long while” is not an acceptable frequency.

Much of the confusion surrounding distributed locks arises from applying a mechanism that is adequate for the first purpose to the second. The simple designs described in the next 2 sections are perfectly serviceable for efficiency. The remainder of the chapter concerns what must be added for correctness, and it turns out to be a good deal.

A lock as an entry in a store

The basic construction requires nothing more than a store that all the parties can reach and that offers 1 atomic operation: create this key, provided it does not already exist. To acquire the lock named nightly-billing, a worker asks the store to create that key with the worker’s own name as its value. If the creation succeeds, the worker holds the lock. If the key already exists, another worker holds it, and the creation fails. To release the lock, the holder deletes the key.

The atomicity of “create if absent” is what makes this a lock. Were the worker to read the key first and create it afterwards, in 2 separate operations, 2 workers could both read that the key was absent and both proceed to create it, which is the same race that chapter 7 found between 2 rate limiter nodes sharing a bucket.

This lock has an obvious and fatal defect. If the worker that holds it crashes, or loses its network connection, the key is never deleted, and the lock remains taken for ever. Every other worker waits indefinitely on a holder that no longer exists.

Leases

The remedy is to let the lock expire. The key is created with a time limit, 10 seconds for example, and the store deletes it automatically when that time has passed. A lock that expires in this way is called a lease. A worker that needs the lock for longer than the lease must renew it periodically, by asking the store to extend the time limit while the worker’s name is still on the key, and a worker that crashes ceases to renew, so that its lease runs out and the lock becomes available to others.

The lease repairs the defect of the previous section and in doing so introduces the difficulty that occupies the rest of this chapter. The lock can now pass to a second worker without the first having agreed to give it up. Whenever a lease ends, there are 2 possibilities which the store cannot distinguish: the holder is dead, in which case handing the lock to someone else is exactly right, or the holder is alive and has merely been prevented from renewing, in which case there will shortly be 2 workers that each consider themselves the holder.

The duration of the lease expresses a trade between these cases. A short lease means that a crashed holder is replaced quickly and that a slow or briefly disconnected holder is more often presumed dead. A long lease means the opposite. No duration makes the second case impossible.

The holder that pauses

It might seem that a careful worker could protect itself against the second case. It knows when its lease ends, so it could check the clock before each step of its work and stop if the lease has run out.

This fails, and the reason is fundamental. No amount of careful programming removes it. Between the moment at which the worker checks its lease and the moment at which the effect of its next action reaches the protected resource, an arbitrary amount of time can pass without the worker being aware of it. The process may be suspended by its runtime in order to reclaim memory, which in some languages can take seconds. The virtual machine in which it runs may be paused by its host, for instance while being moved to another physical machine. The operating system may be short of memory and spend a long time reading the process’s pages back from disk. Or the worker’s write may be on its way through a congested network, delayed in a buffer for longer than the lease has left to run. In every one of these cases the worker performed its check correctly, found the lease valid, and acted in good faith, and the action took effect after the lease had ended and the lock had been given to someone else.

Nor can the worker rely on its clock to notice. A lease is measured on the clock of the lock service, while the worker reads its own, and chapter 8 has already shown that the clocks of 2 machines differ and that either may be corrected at any moment.

The conclusion has to be stated without qualification. A worker can never be certain, at the instant its action takes effect, that it still holds the lock. Any design in which correctness depends on that certainty is unsound, and the degree of unsoundness is merely a question of how often pauses of sufficient length occur.

Fencing tokens

Since the lock service cannot prevent a late holder from acting, and the holder cannot detect its own lateness, the only remaining party that can intervene is the resource being protected. It needs some means of recognising that a request comes from a holder whose lease has been superseded, and the lock service can supply that means at very small cost.

Each time the service grants the lock, it also issues a number that is larger than any it has issued before. This number is the fencing token. The holder attaches its token to every request it sends to the protected resource, and the resource remembers the highest token it has seen and refuses any request that carries a lower one.

The picture follows the scenario of the previous section to its end. Worker A was suspended, its lease expired, B was granted the lock with token 2 and wrote, and A then resumed and attempted to write with token 1. The storage refused, on the simple ground that 1 is less than 2. Nothing in this depends on clocks, on the length of the pause, or on A’s noticing anything.

The token changes the nature of what the lock service provides. Without tokens, the service is asked to guarantee that 2 workers never act together, which it cannot. With tokens, it is asked only to issue increasing numbers and to grant the lock to 1 worker per number, which it can, and the exclusion is enforced at the 1 place where it can be enforced, namely the resource itself.

There is a requirement hidden in this arrangement, and it is the reason fencing is used less often than it should be. The protected resource must take part. It has to accept a token with each request, store the highest, and compare. A database can do this readily, by keeping the token in a column and making each write conditional on the token’s being no lower than the stored one, which is the compare-and-set operation of chapter 1. A resource that cannot be made to check tokens, such as a third-party API that merely accepts requests, cannot be protected for correctness by any distributed lock at all, and in that case one must either accept the risk or redesign the operation so that performing it twice does no harm.

Releasing and renewing

Releasing a lock looks like the simplest of the operations and conceals a well-known error. A worker that releases by deleting the key, without regard to its contents, may delete a lock that it no longer holds.

The correct release is conditional: delete the key only if its value is still my name. The comparison and the deletion have to be a single atomic operation in the store, for the usual reason that the lock might change hands between a separate read and delete. The same condition applies to renewal, which must extend the lease only if the caller is still the holder.

Renewal raises a question of timing. A worker normally renews at intervals of about a third of the lease, so that 2 consecutive renewals may be lost before the lease is in danger. The renewing is commonly done by a separate thread, so that the work itself need not be interrupted, and this has a hazard of its own. If the working thread becomes stuck while the renewing thread continues happily, the lock is held for ever by a worker that is accomplishing nothing, which is the original defect returned by another route. A sound implementation ties the 2 together, for example by renewing only while the work reports progress.

The lock service must not lose locks

Everything so far has treated the store that holds the lock as reliable, and a lock service on a single machine is as reliable as that machine. For availability the service is therefore replicated, and how it is replicated matters a great deal.

Many in-memory stores replicate in the manner that chapter 3 called asynchronous: the leader confirms a write to the client at once and forwards it to its followers afterwards. For a cache this is a sensible choice. For a lock it is a flaw.

The lock service is the 1 component in this design that requires strong consistency in the full sense of chapter 1. A granted lock has to survive the failure of any single node, which means that it has to be stored on a majority of nodes before it is confirmed, exactly as chapter 3 stores a record and chapter 4 elects a leader. Token numbers must likewise come from the replicated log, so that a new leader continues from the highest token ever issued and does not start again from a lower one. In practice this means building the lock service on a consensus system of the kind developed in chapters 3 and 4, and the coordinator of chapter 1 is such a system and commonly offers locks as one of its services.

An alternative that is sometimes proposed avoids consensus by using several independent stores. The worker attempts to create the lock on each of 5 unrelated nodes and regards itself as the holder if it succeeds on at least 3 within a short time. This is more dependable than a single node, since no 1 failure loses the lock. Its safety, however, rests on assumptions about time: that the clocks of the 5 nodes advance at nearly the same rate, and that the worker is not paused between acquiring the lock and using it. The previous sections have shown that neither assumption can be relied upon, and so this approach is suitable for locks taken for efficiency and should be combined with fencing, like any other, when correctness is at stake. It cannot itself supply fencing tokens, since 5 independent nodes have no common counter.

Waiting for a lock

A worker that fails to acquire a lock generally wishes to obtain it as soon as it is free. The crude method is to try again at intervals, which either wastes requests if the interval is short or wastes time if it is long.

Lock services therefore let a worker watch a key and be notified when it is deleted. The naive use of this facility has a defect at scale. If 100 workers are all watching the same lock, its release wakes all 100, all of them attempt to acquire it, 1 succeeds, and 99 resume watching, so that every release costs 100 notifications and 100 attempts. This is the thundering herd that chapter 7 met among retrying clients.

The established remedy arranges the waiters in a queue. Each worker, on arriving, creates an entry of its own that the service numbers in order of arrival. The worker whose entry bears the lowest number holds the lock. Every other worker watches only the entry immediately ahead of its own. When the holder finishes and deletes its entry, exactly 1 worker is notified, the next in line, and it becomes the holder. The entries are tied to the sessions of their workers, so that the entry of a worker that crashes is removed by the service and the queue closes up behind it. As well as avoiding the herd, this arrangement is fair, since the lock is granted in the order in which it was requested.

When a lock is the wrong tool

A distributed lock used for correctness requires a consensus system, fencing tokens, and the cooperation of the protected resource. It is worth asking, before paying that price, whether the problem can be solved without mutual exclusion.

Often it can. If the operation can be made idempotent, so that performing it twice has the same effect as performing it once, then 2 workers doing it together is harmless and no lock is needed: chapter 19 uses this approach for payments. If the operation is an update to a single record, a conditional write expresses “change this only if nobody else has changed it since I read it” directly, and the loser of a race reads again and retries, which is the compare-and-set of chapter 1 and is called optimistic concurrency. And if work can be partitioned so that each item has exactly 1 owner, as the message queue of chapter 11 assigns each partition to 1 consumer, then exclusion is achieved by the assignment and need not be negotiated for each item.

It will be observed that fencing is itself a conditional write, and this is no coincidence. In a correct design the resource’s own check does the real work of protection, and the lock in front of it serves chiefly to keep workers from wasting effort on attempts that would be refused. Seen in this light, a lock is an optimisation layered over a conditional write, and a design that has the conditional write may find that it needs the lock less than it supposed.

Use cases

The lock service has a single kind of user, a worker that must be alone during some critical step.

Functional requirements

Non-functional requirements

This is one of the few designs in the book that demands strong consistency without reservation. A cache may be stale and a rate limiter may be off by a few requests, but a lock service that grants 1 lock to 2 holders has failed at the only thing it is for. The lock and its token are therefore stored on a majority of the service’s nodes before a grant is confirmed, and when a majority cannot be reached the service refuses to grant locks at all, preferring unavailability to error.

Core entities

Capacity estimate

Arithmetic ahead. A lock is about 100 bytes, so a million locks fit in 100 MB, and 3 small nodes are enough.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Memory per lock	name 50 bytes + holder 30 + token 8 + expiry 8	about 100 bytes
Memory for 1 million locks	1000000 × 100 bytes	100 MB

The API

A lock is a resource with a name. Acquiring it creates the resource, renewing it replaces the expiry, and releasing it deletes the resource.

Acquire a lock for 10 seconds

POST /v1/locks/nightly-billing HTTP/1.1
Host: locks.example
Content-Type: application/json

{ "holder": "worker-a", "seconds": 10 }

HTTP/1.1 201 Created
Content-Type: application/json

{ "token": 1, "expires_in": 10 }

Workers have to pass the token on to whatever the lock protects.

Acquire a lock that is held

POST /v1/locks/nightly-billing HTTP/1.1
Host: locks.example
Content-Type: application/json

{ "holder": "worker-b", "seconds": 10 }

HTTP/1.1 409 Conflict
Content-Type: application/problem+json

{
  "type": "/problems/lock-held",
  "title": "The lock is held",
  "status": 409,
  "detail": "held for 9 more seconds"
}

409 Conflict is the status for a request that is valid but cannot be carried out in the current state.

High-level design

One request, step by step

The steps follow the arrows of figure 10.6.

1. A worker sends POST /v1/locks/nightly-billing. The lock service, a group of 3 nodes that agree by majority, records the worker as holder for 10 seconds and returns token 1.
2. The worker does its protected work and sends token 1 with every write to the storage. The storage compares the token with the highest it has accepted.

The scenario, simulated

The simulation for this chapter plays out the paused holder with and without help from the storage. Its output for the fenced case reads as follows.

At the end the storage holds what worker B wrote.

The design, deployed

The lock service is small. It has 3 nodes, 1 in each zone, which form a single consensus group of the kind built in chapter 4, and workers reach whichever of them is the leader. It needs no more machines than this, because a well-designed system takes locks rarely and holds them for long periods, so that the rate of requests is low. What the figure is chiefly meant to show is the second arrow. The workers write to the protected storage directly, and the lock service is not on that path at all, which is why the last line of defence has to be in the storage.

How real systems do it

The lock service of this chapter is modelled on Chubby, which Google described in 2006 and which is the origin of most of the vocabulary. Chubby is a small replicated store whose locks are leases renewed by periodic messages, and its paper makes 2 observations that bear directly on this chapter. It introduces the sequencer, a token obtained with a lock that the holder passes to the servers it uses and that they check, which is the fencing token under its original name. And it reports that the service was designed for coarse locks held for hours or days, chiefly for electing leaders, and that using it for fine-grained locking is discouraged.

ZooKeeper provides the queue of waiters described above as a standard recipe. Each waiter creates a node that is numbered in sequence and tied to its session, and watches only the node immediately before its own. In etcd, a lock is a key attached to a lease, and the revision number at which the key was created, which increases with every change to the store, serves as a fencing token.

The approach of acquiring a lock on a majority of independent in-memory stores is the Redlock algorithm proposed by the author of Redis. Martin Kleppmann’s analysis of it in 2016 set out the argument of this chapter: that a pause of the process or a jump of a clock can leave 2 clients each believing that it holds the lock, that no lease-based lock can prevent this, and that safety therefore requires fencing, which Redlock does not provide since it has no source of increasing numbers. The author of Redis replied in defence of the algorithm, and the exchange remains the clearest public discussion of what a distributed lock can and cannot promise.

Mistakes to avoid

Failures

Trade-offs

The simulation

References

• M. Burrows, “The Chubby Lock Service for Loosely-Coupled Distributed Systems”, OSDI 2006.
• P. Hunt and others, “ZooKeeper: Wait-free Coordination for Internet-scale Systems”, USENIX ATC 2010.
• M. Kleppmann, “How to do distributed locking”, 2016.
• S. Sanfilippo, “Is Redlock safe?”, 2016.
• C. Gray and D. Cheriton, “Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency”, SOSP 1989.

Terms introduced in this chapter

Message Queue: System Design

When one service needs another to do something, the direct course is to call it and wait for the answer. An order service that has accepted an order calls the billing service to issue an invoice, then the warehouse to reserve the goods, then the mail service to send a confirmation, and replies to the customer when all 3 have answered. This works, and it binds the 4 services together more tightly than is good for any of them. The order service is now as slow as the slowest of the 3 and as available as all of them together, so that an outage of the mail service, which is surely the least important of the 3, prevents customers from placing orders.

A message queue loosens this binding. The order service records the fact that an order was created by writing a short message to the queue, and its part is then finished. Billing, the warehouse and the mail service each read the message when they are able to, at their own pace, and a service that is down for an hour reads it an hour late and is otherwise none the worse. The sender and the receivers need not be running at the same time, need not be equally fast, and need not know of one another’s existence.

What the queue takes on in exchange is a set of promises that are easy to state and hard to keep: that a message once accepted is not lost, that it reaches every party entitled to it, that related messages arrive in the order in which they were sent, and that all of this continues to hold when machines fail in the middle of things. This chapter designs a queue of the kind that keeps its messages in a log, and most of it is devoted to stating exactly what can and what cannot be promised about delivery.

What a queue changes

A queue placed between 2 services alters their relationship in 3 respects, each of which is a reason for introducing one.

It separates them in time. The sender, called the producer, hands its message to the queue and proceeds without waiting for anybody to act on it, and the receiver, called the consumer, takes the message whenever it is ready. Neither depends on the other being available at that moment, and the availability of the producer is no longer limited by that of its consumers.

It absorbs differences in speed. If orders arrive in a burst of 1000 per second for a minute while the billing service can process 200 per second, direct calls would overwhelm billing or force the order service to refuse customers. With a queue between them, the burst accumulates as a backlog of 48000 messages, which billing works through in the following 4 minutes. The queue has levelled the load, turning a peak that billing could not meet into a delay that it can.

And it permits several independent receivers. The order service publishes a single message, “order 7 created”, and has no need to know who is interested. A new service, written next year to compute sales statistics, can begin reading the same messages without any change to the producer.

These benefits come at a definite cost, which should be acknowledged at the outset. The producer no longer learns whether its message was acted upon, or when. Errors that a direct call would have reported immediately now occur later, in another service, and have to be found there. And the system as a whole has become eventually consistent in the sense of chapter 1: for some time after an order is accepted, no invoice exists for it, and every part of the system has to be designed in the knowledge that this interval exists and is sometimes long.

2 kinds of queue

The word “queue” is applied to 2 designs that share the vocabulary of producers and consumers and differ in almost everything else.

The older design, which we may call a task queue, treats a message as a piece of work to be done once. Consumers take messages from the front, and each message is given to exactly 1 of them. When the consumer reports that it has finished, by sending an acknowledgement, the queue deletes the message. If the consumer fails to acknowledge within a set time, the queue assumes it has crashed and makes the message available to another. The queue holds only what has not yet been done, and a message that has been processed is gone.

The newer design is a log. Messages are appended to the end of a file and stay there, each identified by its position, which is called its offset. Reading a message does not remove it. A consumer keeps track of the offset it has reached, reads onward from there, and periodically records its position with the queue. Messages are deleted only when they exceed a configured age or the log exceeds a configured size, quite independently of whether anybody has read them.

The log has become the more common basis for communication between services, chiefly on account of the second and third rows. Because reading does not consume, any number of services can read the same messages, each at its own offset. Because messages are retained, a consumer that had a defect can be corrected and set back to reprocess the last day’s messages, and a newly written service can begin from the start of the log and build its state from history. The log is also the structure this book already knows best: it is the append-only log of chapter 2, replicated as in chapter 3, and offered to other services as a product.

The task queue remains the better choice for what its name describes, the distribution of independent jobs among a pool of workers, and it reappears in chapter 12. The remainder of this chapter concerns the log.

Partitions and order

A single log has a single end to which every message is appended, and is therefore limited to the rate at which 1 machine can write and to the capacity of 1 disk. To go beyond that, a named stream of messages, called a topic, is divided into several logs, called partitions, which can be placed on different machines and written concurrently.

Dividing the topic raises the matter of order. Within 1 partition, messages have a definite order, that of their offsets, and every consumer reads them in that order. Between partitions there is no order at all: if one message is in partition 0 and another in partition 1, nothing records which was published first, and 2 consumers may process them in either sequence.

For many messages this is of no consequence. For some it matters a great deal. “Order 7 created”, “order 7 paid” and “order 7 shipped” must be processed in that order, while the order of “order 7 paid” relative to “order 8 created” is immaterial. The requirement, in other words, is for order among messages about the same thing, and the queue provides exactly this by means of a key. The producer attaches a key to each message, here the order number, and the queue chooses the partition by hashing the key, so that all messages with the same key are appended to the same partition, where their order is preserved.

3 consequences of this arrangement tend to be discovered later than they should be. The first is that the choice of key is a design decision with lasting effects, since it fixes which messages are ordered relative to which. The second is that the number of partitions cannot be changed casually: the partition of a key is its hash modulo the number of partitions, so that adding partitions sends new messages for an existing key to a different partition from the old ones, and their order across the change is lost. The third is that a key which accounts for a large share of all messages, such as a single very large customer, makes its partition far busier than the others, and no number of additional partitions relieves it.

Consumer groups

A topic with 4 partitions can be read by 4 consumers working in parallel, and the mechanism that coordinates them is the consumer group. All instances of 1 service, for example the 4 running copies of the billing service, belong to 1 group, and the queue ensures that each partition is read by exactly 1 member of the group at any moment. Different services form different groups, and each group reads the whole topic independently, with its own offsets.

Confining each partition to 1 reader per group is what preserves order during processing. If 2 members of the same group read the same partition, one might process “order 7 paid” before the other had finished with “order 7 created”. The rule has the further consequence, shown in the fourth step, that the number of partitions is an upper limit on parallelism. A fifth consumer in a group that reads 4 partitions has nothing to do.

The redistribution of partitions when a member joins or leaves is called a rebalance, and it is a regular source of trouble in operation. The group detects a failed member by the absence of its periodic heartbeat, a short message that says only “I am alive”. While a rebalance is in progress, the partitions being moved are read by nobody. A consumer that is merely slow, taking longer over a batch of messages than the heartbeat timeout allows, is declared dead, loses its partitions, and rejoins, which triggers another rebalance, and a group can spend a large part of its time rebalancing in this way without any member having actually failed. The remedies are to send heartbeats from a thread separate from the one doing the work, and to limit the number of messages fetched at once so that a batch is always finished well within the timeout.

What “delivered” can mean

A consumer that handles a message does 2 things: it performs the work that the message calls for, and it records with the queue that the message has been dealt with, which in a log is done by committing an offset. These are separate operations on separate systems, and a consumer can crash between them. The order in which it performs them determines what the system guarantees, and there are only 2 orders to choose from.

If the consumer does the work first and commits afterwards, a crash between the 2 leaves the work done and the offset not advanced.

The replacement consumer begins from the last committed offset and repeats work that was already done. No message is ever skipped, and some are processed more than once. This guarantee is called at least once.

If the consumer commits first and does the work afterwards, a crash between the 2 leaves the offset advanced and the work not done.

Now the replacement begins after the message that was never processed, and that message is lost for good. Nothing is processed twice, and some messages are not processed at all. This guarantee is called at most once.

There is no third order, and it follows that a queue cannot on its own deliver each message exactly once. One naturally asks whether the work and the commit could be made a single atomic step, and in general they cannot, because they affect different systems: the invoice is sent by a mail server, and the offset is stored by the queue. Making 2 independent systems change together or not at all is the hard problem of distributed transactions, and for something like sending an email it is impossible, since an email cannot be unsent.

What can be achieved, and what careful systems mean when they speak of exactly-once processing, is an effect that occurs exactly once. The queue delivers at least once, and the consumer is built so that processing a message a second time changes nothing. An operation with this property is idempotent, and there are 2 standard ways of obtaining it. Some operations are idempotent by nature: “set the status of order 7 to paid” may be applied any number of times with the same result, whereas “add 1 to the count of paid orders” may not. For the others, the consumer records the identifier of every message it has processed, in the same database transaction as the effect of the message, and on receiving a message it has already recorded it does nothing. The simulation for this chapter uses this second method: a consumer crashes after acting on 6 messages and before committing, its successor is given the same 6 messages again, recognises each of them, and the total number of actions taken remains 6.

The choice between the 2 guarantees should be made by asking which failure is the more tolerable. For most business events it is plainly better to risk a duplicate, which idempotency can neutralise, than a loss, which nothing can repair, and at least once with idempotent consumers is the standard combination. At most once is appropriate where a late or repeated message is worthless or harmful, such as a reading from a sensor that will be superseded by the next reading a second later.

The producer’s side

The same difficulty exists on the producing side, where it is less often noticed. A producer sends a message and waits for the queue to confirm that it has been stored. If no confirmation arrives within the expected time, the producer cannot tell whether the message was lost on the way to the queue or stored successfully with only the confirmation lost on the way back. If it does not send again, the message may be lost, and if it does, the message may be stored twice.

The solution again consists in making the repetition harmless. Each producer is given an identity by the queue, and numbers its messages to each partition consecutively. The partition remembers the highest number it has stored for each producer and discards any message whose number it has already seen. A producer can then resend without hesitation, and the log contains each message once.

The producer also decides how much durability to wait for, by choosing what constitutes a confirmation. It may send without waiting for any confirmation, which is fastest and loses messages whenever a machine fails. It may wait until the leader of the partition has written the message, in which case a message is lost if the leader fails before its followers have copied it. Or it may wait until the message is stored on a majority of the replicas, which is the rule of chapter 3 and the only setting under which a confirmed message survives the failure of a machine. Our design uses the third.

Durability and replication

Each partition is a log, and it is kept durable and available by precisely the means that chapter 3 applied to the log of a key-value shard. One of the machines that hold a copy of the partition, the machines being called brokers, is its leader and receives all appends. The others are followers, which copy the leader’s log, and a message counts as stored when a majority holds it. If the leader fails, a follower that holds every stored message is elected in its place, by the procedure of chapter 4.

Consumers are permitted to read a message only once it is stored on a majority. The reason is the one given in chapter 3 for not applying a record before it is committed. If a consumer were allowed to read a message that existed on the leader alone, and the leader then failed, the new leader’s log would not contain that message, and the consumer would have acted on something that, as far as the queue is concerned, never happened.

The committed offsets of consumer groups are themselves data that must survive failures, since losing them would cause every group to reprocess its entire topic. They are small and frequently updated, and are kept in a replicated store of their own, shown in the design as the offset store.

Difficulties in operation

Messages that cannot be processed

Suppose a message is malformed in a way the consumer does not anticipate, so that every attempt to process it raises an error. Under at least once delivery the consumer does not commit past it, and so it receives the same message again, fails again, and continues to do so indefinitely. A single such message, known as a poison message, halts its entire partition, since no later message in the partition can be reached.

The remedy is to limit the number of attempts. After a message has failed a set number of times, 5 for example, the consumer publishes it to a separate topic reserved for failures, called a dead-letter queue, commits past it, and continues. An alert draws an engineer’s attention to the dead-letter queue, where the message can be examined and, once the defect is corrected, published again. The cost is that the failed message has been overtaken by its successors, so that the order within its key is broken, and whether that is acceptable has to be decided for each topic.

Consumers that fall behind

The difference between the offset of the newest message in a partition and the offset that a group has committed is the group’s lag. It is the most informative single measurement in the operation of a queue, because it states directly how far a consumer is behind its producers. A lag that fluctuates around a small value indicates health. A lag that rises steadily indicates that messages arrive faster than they are consumed, and since messages are retained only for a limited period, a lag that continues to rise ends with the oldest unread messages being deleted before the consumer has reached them, which is a loss of data that no setting on the producer can prevent.

Publishing together with a database write

A service very often needs to do 2 things in response to 1 request: change its own database and publish a message announcing the change. The order service inserts an order and publishes “order 7 created”. Performed as 2 separate operations, these can come apart. If the service writes the database and crashes before publishing, an order exists of which no other service will ever hear. If it publishes first and the database write then fails, other services act on an order that does not exist.

The outbox pattern makes the 2 operations one. The service writes the message, as an ordinary row, into a table of its own database called the outbox, in the same transaction as the order itself, so that the database guarantees that both are written or neither is. A separate process then reads the outbox and publishes its rows to the queue.

The relay may crash between publishing a row and marking it as published, and will in that case publish it again on restarting. The pattern therefore delivers at least once, which returns us to the idempotent consumer, and it will be noticed how many of the questions in this chapter have ended at that same answer.

Keeping the latest value only

Retaining messages for a fixed period suits a stream of events. Some topics, however, carry the current state of things, each message being the latest value for its key, such as a customer’s current address. For these, old values are of no interest and the latest must be kept however old it is. A log can be compacted for this purpose: a background process rewrites it, keeping for every key only the most recent message. A new consumer reading a compacted topic from the beginning thereby obtains the current value of every key, and can construct a complete copy of the data from the topic alone.

Use cases

Producers publish, consumers fetch and commit, and operators decide how many partitions a topic has.

Functional requirements

Non-functional requirements

The queue is strongly consistent within a partition, which means that all consumers see the same messages at the same offsets, and that a message which has been confirmed to its producer will be seen by every consumer. It makes no promise about the relative order of messages in different partitions. The system built from the queue and its consumers is eventually consistent, as the opening section explained, since a consumer acts on a message some time after it was published.

Core entities

Capacity estimate

Arithmetic ahead. 7 days of messages are 60 TB for 1 copy and 181 TB for 3, about 46 brokers with 4 TB disks.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Bytes per day	100000 × 1 KB × 86400 s	8.64 TB
7 days, 1 copy	8.64 × 7	60.5 TB
3 copies	60.5 × 3	181 TB
Brokers with 4 TB disks	181 ÷ 4	45.4, so 46

The API

Publishing creates a message within a topic. Fetching reads from a partition, starting at an offset. Committing replaces the stored offset of a group.

Publish a message

POST /v1/topics/orders/messages HTTP/1.1
Host: queue.example
Content-Type: application/json

{ "key": "order-7", "body": "paid" }

HTTP/1.1 201 Created
Content-Type: application/json

{ "partition": 1, "offset": 2 }

Its key chooses the partition, and the response says where the message now lives.

Commit an offset

PUT /v1/topics/orders/partitions/1/offsets/billing HTTP/1.1
Host: queue.example
Content-Type: application/json

{ "committed": 6 }

HTTP/1.1 204 No Content

PUT is idempotent, so committing the same offset twice is harmless.

Consumers fetch messages, and the queue does not push them. This allows each consumer to proceed at its own pace and to ask for as many messages as it can handle, where a queue that pushed would have to guess at the capacity of every consumer. So that an idle consumer does not issue requests continuously, a fetch that finds no new messages is held open by the broker for a short time and answered as soon as a message arrives.

High-level design

One request, step by step

The steps follow the arrows of figure 11.8.

1. A producer sends a message with the key order-7. A hash of the key picks partition 0, and the request goes to the broker that leads it.
2. The leader appends the message to the partition’s log and copies it to a follower, as in chapter 3. Then it answers with the offset.
3. A consumer asks the leader for messages from its committed offset onward.
4. The consumer acts on each message. When the batch is done, it stores the new offset in the offset store.

The design, deployed

The 12 brokers are divided among 3 zones, and each partition has 1 replica in each zone. Leadership is spread evenly, so that every broker leads roughly a third of the partitions it holds, and the failure of a broker moves the leadership of only those. Producers and consumers hold a list of the brokers and connect directly to the leader of each partition they use, so that here too there is no load balancer on the path of the data. The offsets of the consumer groups are kept in a small replicated store of their own.

How real systems do it

The log-structured queue of this chapter is Kafka, which LinkedIn described in 2011. The partitioned log, the offset kept by the consumer, the consumer group and the retention of messages by age all come from that design, as do the compaction of a log by key and the producer that numbers its messages so that a retry is not stored twice, which was added in 2017 together with transactions.

In 1 respect Kafka differs from this chapter, and the difference is instructive. This chapter replicates a partition as chapter 3 does, confirming a message when a majority of replicas hold it. Kafka uses a different rule. The leader keeps a list of the replicas that are fully caught up, the in-sync replicas, and confirms a message when every one of them has it. A replica that falls behind is removed from the list, and the partition continues with those that remain. With this rule, f + 1 replicas tolerate f failures, where a majority rule needs 2f + 1, which matters for a system that stores a great deal of data. The cost is that the list itself must be agreed upon by some other means, for which Kafka relied on ZooKeeper and now on a Raft-based controller of its own, and that a setting for the smallest permitted size of the list has to be chosen with care, since a list that has shrunk to the leader alone confirms messages that exist on 1 machine.

The task queue described at the start of the chapter is the model of Amazon’s SQS, whose visibility timeout is the period after which an unacknowledged message is given to another consumer, and of RabbitMQ. Pulsar separates the brokers, which serve requests, from the storage of the log, which is delegated to a separate replicated service. And the outbox pattern is commonly implemented without a relay process, by reading the database’s own log of changes and publishing from that, a technique called change data capture.

Mistakes to avoid

Failures

Trade-offs

The simulation

References

• J. Kreps, N. Narkhede and J. Rao, “Kafka: a Distributed Messaging System for Log Processing”, NetDB 2011.
• J. Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction”, LinkedIn engineering blog, 2013.
• Apache Kafka documentation, “Replication”, on in-sync replicas.
• Apache Kafka, KIP-98, “Exactly Once Delivery and Transactional Messaging”, 2017.
• Amazon SQS documentation, “Visibility timeout”.
• C. Richardson, Microservices Patterns, Manning, 2018, for the transactional outbox.

Terms introduced in this chapter

Job Scheduler: System Design

A good deal of the work that a system does is not done while anybody waits for it. A user uploads a video and receives an immediate confirmation, while the conversion of the video into several resolutions takes place over the following minutes. An account is closed today, and its data is to be erased in 30 days. A report on yesterday’s sales is to be compiled every night at 2 o’clock. In each case a piece of work, which we shall call a job, has to be remembered now and carried out later, by some machine other than the one that asked for it, and it has to be carried out even if that machine, or the one that accepted the request, fails in the meantime.

A job scheduler is the service that takes responsibility for such work. It accepts jobs, stores them durably, hands them to workers when their time has come, and keeps track of what became of them. The essential difficulty is that the scheduler can never observe a job being executed. It gives a job to a worker and thereafter knows only what the worker reports, and a worker that has stopped reporting may have crashed, may be slow, or may have completed the job and failed only in delivering the news. Everything in the design follows from the need to make progress in spite of this uncertainty without either losing jobs or running them more often than can be tolerated.

The life of a job

Every job in the scheduler is in exactly 1 of a small number of states, and the whole behaviour of the system can be described by the permitted movements between them.

A newly submitted job is waiting. When a worker takes it, the job becomes running, and from that state there are 3 ways out. If the worker reports success, the job has succeeded and nothing more will happen to it. If the worker reports failure, or reports nothing before a deadline, the job returns to waiting in order to be tried again. And if it has already been tried a fixed number of times without success, it becomes dead, which means that the scheduler has given up on it and that a person will have to look at it.

The value of describing the system in this way is that every question about its correctness becomes a question about a transition. Jobs are not lost if every job is always in 1 of the states and no transition discards a job. A job is not run by 2 workers at once if the transition from waiting to running can be made by 1 worker only. The 3 sections that follow take the transitions out of the running state one by one, beginning with the least obvious.

Lending a job: the lease

Suppose that a worker takes a job and the scheduler marks it as running and does nothing further. If the worker completes the job, all is well. If the worker crashes, the job remains marked as running indefinitely, and since only waiting jobs are handed out, it will never be given to anyone else. The job has in effect been lost, although it is still in the store.

The scheduler therefore does not give a job to a worker outright. It lends it for a limited period, and the loan is called a lease. The worker must report the result before the lease ends. If the period passes without a report, the scheduler concludes that the worker is not going to finish and returns the job to the waiting state, from which another worker will take it. Chapter 10 used a lease for the same reason, to recover a lock from a holder that has vanished.

The simulation for this chapter runs this scenario with 6 jobs and 3 workers, 1 of which crashes 2 seconds into its first job. All 6 jobs are completed. The job held by the crashed worker is recovered when its lease expires at 3 seconds and is finished by another worker at 6, and the record shows 2 attempts for that job and 1 for each of the others.

The length of the lease involves the compromise that chapter 10 described. If it is short, a crashed worker’s job is recovered quickly, and a worker that is merely slow has its job taken away and given to another while it is still working. If it is long, slow workers are safe and a crashed worker’s job sits idle for a long time before anyone notices. For jobs whose duration is predictable, the lease is set comfortably above the usual duration. For jobs that may take minutes or hours, a fixed lease of sufficient length would delay recovery intolerably, and the worker takes a short lease and renews it at intervals for as long as it is making progress, as with the renewal of a lock.

A job may run twice

The lease brings with it a consequence that has to be faced squarely. The expiry of a lease does not prove that the worker has stopped. The worker may be running slowly, or may have been paused in the manner described in chapter 10, or may have finished the job and been unable to report because of a fault in the network. In each of these cases the scheduler gives the job to a second worker while the first is still executing it or has already executed it, and the job is performed twice.

This is the same situation that chapter 11 called delivery at least once, and it is unavoidable for the same reason. The scheduler must choose between reissuing a job whose worker has fallen silent, which risks a second execution, and not reissuing it, which risks no execution at all, and since a job that is never run is nearly always the worse outcome, schedulers reissue. It follows that a scheduler guarantees that a job will be run at least once, and cannot guarantee that it will be run exactly once.

The responsibility for making this harmless lies with the author of the job. A job should be idempotent, which is to say that running it twice should leave the world in the same condition as running it once. Some jobs have this property without effort: resizing an image and storing the result under a fixed name can be repeated indefinitely. Others must be given it. A job that charges a customer has to carry a unique reference that the payment system uses to recognise a repeated charge, as chapter 19 describes, and a job that sends an email should record that it has done so and check the record before sending.

One further precaution limits the damage that a late worker can do. When a worker reports a result, it presents the number of the lease under which it was working, and the scheduler accepts the report only if that lease is still the current one for the job. The slow worker of the scenario above, reporting success under lease 771 after the job has been reissued under lease 772, is told that its lease is no longer valid and its report is ignored. The lease number plays the part of the fencing token of chapter 10: it cannot prevent the late worker from having done the work, and it does prevent the late worker from overwriting the state recorded by its successor.

Giving 1 job to 1 worker

The transition from waiting to running must be made by 1 worker only, and when many workers ask for jobs at the same time this requires care.

The natural way of writing the operation is in 2 steps: find a job that is waiting, then mark it as running. Between the 2 steps there is a gap, and a second worker’s request can fall into it.

The remedy, shown in the last step, is to make the transition a single conditional operation in the job store, of the form “set this job to running, under my lease, if it is still waiting”. The store performs each such operation in its entirety before beginning the next, so that when 2 workers attempt it on the same job, the second finds the condition no longer true and is refused. This is the compare-and-set of chapter 1 once more.

A conditional change is correct and, with many workers, not yet efficient. If 50 idle workers all identify the oldest waiting job and all attempt to take it, 1 succeeds and 49 have wasted a round trip and must try again, upon which they all converge on the next job. Databases used as job stores offer a remedy in the form of a query that selects waiting rows while passing over any row that another transaction has locked, so that each worker is given a different job at the first attempt. Where no such facility exists, workers can be made to choose at random among the first few waiting jobs, which reduces the collisions considerably.

Retries and jobs that never succeed

A job may fail for reasons that will have passed in a moment, such as a brief network fault or a service that was restarting, and for such failures the right response is to try again. The scheduler counts the attempts made on each job and, after a failure, returns the job to the queue.

It should not be returned for immediate execution. If a job has failed because some service it depends upon is overloaded, then retrying at once adds to the overload, and thousands of jobs doing the same can prevent the service from recovering at all. The job is therefore made to wait before its next attempt, and to wait longer after each successive failure: 1 second, then 2, then 4, and so on, which is called exponential backoff. A random amount is added to each pause, so that jobs which failed together do not all return together. Chapter 13 examines these pauses in detail.

Some jobs fail for reasons that will never pass. The input is malformed, or the code has a defect that this particular input exposes, and the hundredth attempt will fail exactly as the first did. Retrying such a job for ever would waste capacity and, worse, would conceal the problem. The number of attempts is therefore limited, and a job that exhausts its attempts is moved to the dead state, where it is kept with its last error for an engineer to examine. This is the dead-letter queue of chapter 11 under another name. An alert on the number of dead jobs is among the most useful that a scheduler can provide, since a sudden increase usually means that a defect has been deployed.

It is helpful if workers can distinguish the 2 kinds of failure where the cause is evident. A job that fails because its input cannot be parsed will not improve with repetition, and a worker that reports it as permanently failed spares the system the remaining attempts.

Jobs that are due later

A job may be submitted with a time before which it is not to run. The scheduler must then, at every moment, be able to find the jobs that have become due, and it must do so without examining all the jobs that are not.

The straightforward method is to store the due time of each job and to keep the waiting jobs in an index ordered by that time. The scheduler asks the store, every second or so, for waiting jobs whose time is not later than the present, and since the index is ordered, the store answers by reading from its beginning and stopping at the first job that is not yet due. The cost is proportional to the number of jobs that are due, however many millions are waiting for later.

2 limitations of this method should be understood. The first is precision: a job may start up to a polling interval late, which is of no importance for a job due in 30 days and matters for one that is meant to fire at a precise second. The second is that all scheduler nodes query the same index, which becomes a point of contention at high rates. A common refinement divides responsibility. Jobs due in the distant future remain in the durable store, and a scheduler node periodically loads those due within the next minute into its own memory, where it keeps them in a structure ordered by time and can release each at its exact moment. If that node fails, nothing is lost, since the jobs are still in the store and another node loads them.

Recurring jobs

A recurring job is a rule, such as “every day at 02:00”, from which a new job is to be created at each of the times the rule names. Creating those jobs is itself a piece of work that must be done exactly once per occasion, and with several scheduler nodes running for the sake of availability, the obvious implementation does it once per node.

One remedy is to elect 1 scheduler node as the only one that evaluates schedules, using the lock service of chapter 10. This works, and it inherits the weakness of every lock, namely that for a short time during a change of leader there may be 2 nodes that each believe themselves elected. The more dependable remedy does not rely on there being a single creator. Each run of a recurring job is given a name constructed from the rule and the scheduled time, and the job is created only if no job of that name exists. Any number of nodes may then attempt the creation, and the store admits the first and rejects the rest. The creation has been made idempotent, which, as so often in this book, is more reliable than trying to ensure that it is attempted only once.

The opposite hazard is the run that is missed. If every scheduler node is down from 01:55 to 02:10, the question arises of what should happen at 02:10 to the run that was due at 02:00. There is no universally right answer. A nightly report should probably be run late, since a late report is better than none. A job that sends a “good morning” notification should probably be skipped if it is already afternoon. And a job due every minute that has missed 3 hours should certainly not be run 180 times in succession. The policy for missed runs, whether to run once on recovery, to run every missed occasion, or to skip, has to be stated for each recurring job, and a scheduler that leaves it unstated has made the choice on its users’ behalf.

Schedules expressed in local time have a further peculiarity that causes a failure once or twice a year. On the night when clocks are set forward, the time 02:30 does not occur, and a job scheduled for it is either skipped or run at an unexpected hour. On the night when clocks are set back, 02:30 occurs twice, and the job may run twice. Schedules are best expressed in coordinated universal time, which has no such discontinuities, and converted for display only.

Priorities and fairness

When more jobs are waiting than the workers can take, the order in which jobs are handed out becomes a matter of policy. The simplest policy is first come, first served, and 2 situations call for something more.

Some jobs are more urgent than others. An email carrying a password reset should not wait behind 100000 jobs that regenerate thumbnails. Jobs are accordingly given a priority, and the scheduler hands out the highest priority first. Strict priority has a well-known defect, which is that a continuous supply of high-priority jobs prevents low-priority jobs from ever running, a condition called starvation. It is countered by letting the effective priority of a job rise with the time it has waited, or by reserving a share of the workers for each level.

Fairness among customers presents a similar problem. If 1 customer submits a million jobs and they are served in order of arrival, every other customer’s jobs wait behind all of them. The remedy is to keep a separate queue for each customer and to take from the queues in rotation, so that a customer who submits a great deal delays chiefly its own jobs. A limit on the number of jobs of 1 customer that may be running at once serves the same end.

Use cases

Developers submit jobs and ask about their state. Workers take jobs and report results.

Functional requirements

Non-functional requirements

The state of a job has to be strongly consistent. Were 2 scheduler nodes to hold different views of whether a job is waiting or running, each could hand it to a worker, and the lease, on which all recovery depends, would mean nothing. The job store is therefore replicated in the manner of chapters 3 and 4, and a transition is acknowledged only when a majority has recorded it. What remains eventual is the relationship between the store and the world: a job recorded as running may in fact have finished a moment ago, and the design accepts this by allowing a job to run more than once.

Core entities

Capacity estimate

Arithmetic ahead. 10 million jobs per day are 116 per second on average, which 1 replicated job store handles.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Jobs per second, average	10000000 ÷ 86400	116
Jobs per second, at a peak of 10 times the average	116 × 10	1160
Storage for 30 days	10000000 × 30 × 1 KB	300 GB

The API

A worker takes a job by creating a lease, which is why the request is a POST to the collection of leases, and it reports by writing the result that belongs to its lease, which is a PUT to an address that names the job and the lease.

Submit a job

POST /v1/jobs HTTP/1.1
Host: jobs.example
Content-Type: application/json

{ "kind": "resize-image", "payload": { "image": "cat.jpg" }, "run_at": "2026-10-14T09:00:00Z" }

HTTP/1.1 201 Created
Location: /v1/jobs/8231
Content-Type: application/json

{ "id": 8231, "state": "waiting", "attempts": 0 }

Location gives the URL of the new job, as HTTP expects after a 201.

Take a job: create a lease

POST /v1/leases HTTP/1.1
Host: jobs.example
Content-Type: application/json

{ "worker": "worker-2", "seconds": 30 }

HTTP/1.1 201 Created
Location: /v1/leases/771
Content-Type: application/json

{ "lease": 771, "job": { "id": 8231, "kind": "resize-image", "payload": { "image": "cat.jpg" } }, "expires_in": 30 }

A worker takes a job by creating a lease, so the path names a thing and the method says what happens to it. The scheduler picks the job. When no job is waiting, the answer is 204 No Content.

Report the result

PUT /v1/jobs/8231/results/771 HTTP/1.1
Host: jobs.example
Content-Type: application/json

{ "outcome": "succeeded" }

HTTP/1.1 204 No Content

The lease number proves that this worker still holds the job. A report with an expired lease is answered 409 Conflict.

Workers ask the scheduler for jobs, and the scheduler does not push jobs to workers. A worker that asks is by that fact ready to work, and one that is overloaded or shutting down merely stops asking, so that no separate mechanism is needed for the scheduler to learn the capacity of each worker.

High-level design

One request, step by step

The steps follow the arrows of figure 12.7.

1. A developer sends POST /v1/jobs. The scheduler checks the request and gives the job an id.
2. The scheduler writes the job to the job store in the state waiting. The store keeps it on a majority of its nodes, as in chapter 3.
3. A worker sends POST /v1/leases. The scheduler picks a waiting job, marks it running with this worker and a deadline, and returns it.
4. Every few seconds the reclaimer looks for running jobs whose deadline has passed and sets them back to waiting.

The design, deployed

The scheduler nodes hold no state and stand behind a pair of load balancers, through which both submitters and workers reach them. All state is in the job store, a leader with 2 followers in 3 zones. The reclaimer, which returns jobs with expired leases to the queue, runs as 2 instances of which 1 is active, chosen through the lock service of chapter 10, and since reclaiming is itself a conditional write, a moment in which both believe themselves active does no harm. The 200 workers are the only tier whose size follows the amount of work, and it can be changed freely.

How real systems do it

Google has published an account of its distributed cron service that agrees with this chapter on the essentials and is frank about the same limits. The service keeps its state in a store replicated by Paxos, and only the leader launches jobs. The authors state that the system cannot guarantee that a job is launched exactly once, since the leader may fail between launching and recording the launch, and that the owners of each job must therefore decide whether it is worse for the job to be skipped or to run twice. They recommend making launches idempotent, and naming each launch after its scheduled time so that a repeat can be recognised, as this chapter does.

The remedy for many workers competing for the same waiting job, a query that passes over rows locked by other transactions, is the clause SKIP LOCKED, which PostgreSQL has offered since version 9.5 and which is the basis of several job queues built on that database. For keeping very large numbers of timers in memory, the classical structure is the timing wheel, in which timers are placed in slots by their expiry time and a pointer advances 1 slot at each tick, so that starting and stopping a timer take constant time.

A more recent development goes beyond anything in this chapter. Systems such as Temporal, which descends from Uber’s Cadence, record every step of a long-running piece of work in a durable history, and recover a failed worker by replaying that history to the point it had reached. A job in such a system is a program that appears to run for days without interruption, and the leases, retries and idempotency of this chapter are handled beneath it. The underlying guarantee is unchanged: each step runs at least once, and the steps must tolerate repetition.

Mistakes to avoid

Failures

Trade-offs

The simulation

References

• S. Davidovic and K. Guliani, “Reliable Cron across the Planet”, ACM Queue, 2015.
• PostgreSQL documentation, “SELECT”, on FOR UPDATE SKIP LOCKED.
• G. Varghese and A. Lauck, “Hashed and Hierarchical Timing Wheels”, SOSP 1987.
• Temporal documentation, “Workflow execution” and “Event history”.

Terms introduced in this chapter

Notification Service: System Design

Nearly every service in a large system has occasion to tell a person something. The order service wishes to say that a parcel has been shipped, the security service that a login was attempted from a new device, the billing service that a payment has failed. If each of these services dealt with email, text messages and push notifications for itself, each would need to know the user’s preferred channel and language, to hold credentials for the outside companies that carry the messages, to handle their failures, and to refrain from disturbing the user at 3 in the morning. A notification service gathers all of this into 1 place. Other services tell it what should be said and to whom, and it sees to the saying.

The peculiar character of this system comes from the fact that its most important work is done by parties it does not control. An email is delivered by an email provider, a text message by a telephone network, a push notification by the company that makes the phone’s operating system. These providers are slow at some times and unavailable at others, they impose limits on how fast they may be called, and they reply to a request in ways that leave real doubt about what has happened. The design of a notification service is therefore, to a large extent, a study in calling an unreliable dependency well: when to try again and how long to wait, when to stop trying, how to avoid sending the same message twice, and how to avoid making a struggling provider’s condition worse.

The path of a notification

A notification passes through a sequence of decisions before anything is sent, and laying them out in order shows where the service’s responsibilities lie.

A calling service submits a request that names a user, a kind of message and the data to be inserted into it, for example “tell user 42 that order 7 has shipped”. The notification service first establishes that it has not already received this request. It then consults the user’s preferences to learn whether the user wishes to receive messages of this kind at all and by which channel. It finds the address for that channel, which may be an email address, a telephone number, or the device token by which a push provider identifies a particular installation of an app on a particular phone. It selects the template for this kind of message in the user’s language and fills in the data. Only then is there something to send, and it is placed in a queue from which a pool of senders takes messages and hands them to the appropriate provider.

The division into a part that accepts requests and a part that sends them, with a queue between, is the central structural decision, and it follows from the first section of chapter 11. A provider may take several seconds to answer, or may not answer at all. If the calling service had to wait for the provider, an outage at a text message provider would make the order service slow, which is exactly the coupling a notification service exists to remove. The service therefore answers the caller as soon as the request has been stored durably, with the status 202 Accepted, whose meaning in HTTP is precisely “received, and not yet acted upon”, and does the sending afterwards.

The same request twice

The calling service is itself a distributed program and will sometimes send the same request more than once. It may time out while waiting for the 202 and try again, or it may be a consumer of a message queue that, as chapter 11 explained, delivers its messages at least once. From the user’s side the result of treating both requests as new would be 2 identical text messages, which is a visible defect and, for some kinds of message, an alarming one.

The notification service cannot recognise a repeat by comparing contents, since 2 separate and legitimate notifications may have identical text. The caller must therefore say which requests are the same, and does so by attaching an idempotency key, a name of the caller’s own choosing that identifies the event being reported, such as order-7-shipped. The service records every key it has accepted, and when a request arrives with a key that is already recorded, it returns the original response and sends nothing.

The recording of the key and the placing of the message in the queue must succeed or fail together. If the key were recorded and the service then crashed before queueing the message, the caller’s retry would be dismissed as a repeat and the notification would never be sent. This is the problem of writing to a database and publishing a message as 1 step, and the outbox of chapter 11 solves it here as it did there. Keys need not be kept for ever. A retry arrives within seconds or minutes of the original, and a key that is retained for a day covers every realistic case.

Failures that pass and failures that do not

When a sender hands a message to a provider, the provider’s reply falls into 1 of 3 classes, and the proper response is different for each.

The provider may accept the message. The sender records the fact and is finished.

The provider may refuse the message for a reason that is permanent. The telephone number does not exist, the email address is malformed, or the device token belongs to an app that the user has since removed. Sending the same request again will yield the same refusal, so the sender must not retry. It should record the failure and, in the case of an invalid address or token, mark it so that no further notifications are attempted there. Continuing to send to dead addresses is more than wasteful: providers of email in particular observe what proportion of a sender’s messages are undeliverable and begin to treat a sender with a high proportion as a source of unwanted mail, to the detriment of its deliverable messages.

Or the provider may fail in a way that is probably temporary. It answers that it is overloaded, or that the sender has exceeded its permitted rate, or it does not answer within the time allowed. These are the cases for trying again, and they include the 1 that is most awkward. When a request times out, the sender does not know whether the provider never received it or received it, sent the message, and lost only the reply. A retry in the second case sends the user a duplicate. Where the provider’s API accepts an idempotency key of its own, the sender should supply one, so that the provider can recognise the retry, and where it does not, an occasional duplicate after a timeout is a risk that has to be accepted, since the alternative is an occasional message that is never sent.

Retrying with pauses that double

Having decided to retry, the sender must decide when. The answer that comes first to mind, immediately, is the worst available.

A provider that reports overload is receiving more requests than it can handle. A sender that responds by repeating its request at once has increased the load on a system that has just said it has too much, and if a thousand senders do the same, the provider is held in its overloaded condition by the very traffic that its failures generate. Systems have been kept down for hours in this way after the original cause of the trouble had gone.

The sender should therefore wait, and the question of how long is settled by the observation that each successive failure is evidence that the previous wait was too short. The wait is accordingly doubled after every failure: 1 second, then 2, then 4, then 8. This is exponential backoff. It has the property that a brief disturbance is overcome quickly, by the first or second retry, while in a prolonged outage the sender’s requests become rapidly less frequent and soon impose almost no load.

The simulation for this chapter sends 5 notifications to a provider that fails a set number of times for each. One succeeds at the first try. One fails twice and is sent on the third try, 3 seconds after the first, and one fails 4 times and is sent on the fifth, after 15 seconds, as in the picture. One fails every time and is set aside after 5 tries. The remaining request carries a key that has been seen before and causes nothing to be sent.

2 limits complete the scheme. The pause is capped at some maximum, a minute for instance, so that it does not grow to absurd lengths. And the number of tries is bounded, so that a notification which can never be sent leaves the system. What should be done with a notification that has exhausted its tries depends on its kind. A code for logging in is worthless after a few minutes and may be dropped, whereas a notice that a payment has failed should be set aside, in the manner of the dead-letter queue of chapter 11, and looked at by a person.

Why the pauses are made random

Doubling the pause protects a provider from a single sender. It does not protect the provider from many senders that failed at the same moment, which is the usual situation, since a provider’s difficulty affects all of its callers together.

Suppose that 1000 notifications fail in the same second. All 1000 wait 1 second and return together, and if the provider has not recovered they fail together, wait 2 seconds, and return together again. The doubling has spaced the waves further apart and has done nothing to diminish them: each wave strikes the provider with the full 1000 requests, and a provider that is on the point of recovering is knocked down afresh by each.

The remedy is to make each sender’s pause random, which is called jitter. In the simplest effective form, where the plain scheme would wait 4 seconds, the sender waits for a time chosen at random between 0 and 4 seconds. The 1000 retries are then spread evenly over the interval and reach the provider as a steady flow of about 250 per second. The average wait has been halved, and the peak load on the provider has been reduced by a far larger factor. This thundering herd and its cure appeared in chapter 7 from the other side, as the advice a rate limiter gives its clients.

The circuit breaker

Backoff governs the retries of a single notification. It leaves untouched a further problem, which arises when a provider is not merely slow but down. Every new notification that arrives is then tried for the first time, and each such first try waits for the full length of the timeout before failing. Senders are occupied with calls that cannot succeed, the queue lengthens behind them, and notifications for other channels, whose providers are perfectly healthy, are delayed because all the senders are waiting on the provider that is not.

A circuit breaker is a small mechanism that remembers the recent behaviour of a dependency and, when the dependency is plainly failing, stops calling it for a time. It takes its name from the electrical device that cuts off a circuit in which too much current is flowing.

The breaker has 3 states. While it is closed, calls pass through to the provider and the breaker counts consecutive failures. When the count reaches a threshold, the breaker becomes open, and in that state it rejects every call at once without contacting the provider. After a set interval it becomes half open and admits a single trial call. If the trial succeeds, the breaker closes and normal operation resumes, and if the trial fails, the breaker opens again for another interval.

2 benefits follow. The senders fail fast, in microseconds, where they would otherwise have waited out a timeout of many seconds, and so remain free to serve the other channels. And the failing provider is relieved of all traffic but the occasional trial, which gives it the best chance of recovering. A notification rejected by an open breaker has not been lost. It returns to the queue to be tried later, or is directed to another provider.

Providers’ limits and a second provider

Providers restrict the rate at which they may be called, and a sender that exceeds the permitted rate is refused, usually with the status 429 that chapter 7 described. A well-behaved notification service does not discover the limit by running into it. It paces its own calls to each provider slightly below that provider’s limit, using the leaky bucket of chapter 7, which is the algorithm designed for exactly this purpose of feeding a consumer at a steady rate. Notifications that arrive faster than the provider will take them wait in the queue, which is what the queue is for.

For channels on which delivery matters greatly, such as the text messages that carry login codes, a service commonly holds accounts with 2 providers. When the circuit breaker for the first is open, messages are directed to the second. This raises the availability of the channel considerably and brings a small complication with it: a message that timed out at the first provider and was then sent through the second may arrive twice, since the 2 providers know nothing of each other and cannot recognise the duplicate.

Sent is not delivered

When a provider accepts a message, it has undertaken to try to deliver it, and nothing more. The email may yet be rejected by the recipient’s mail server, the text message may wait for a phone that is switched off, and the push notification may be addressed to an app that was removed last week. None of this is known at the moment the provider replies to the sender.

Providers report the later fate of a message after the event, by calling an address that the notification service has registered with them. Such a call from a provider back to its customer is known as a webhook. The service must be prepared to receive these reports at any time, in any order, and more than once, and uses them to bring the recorded state of each notification up to date. They are also the principal means by which the service learns that an address has gone bad, so that it can stop sending to it.

The practical consequence is that a notification service should be careful in what it claims. It can truthfully state that a message was handed to a provider. Whether the message was delivered is known only later and only for some channels, and whether it was read is hardly ever known. A system that requires certainty that a person has seen something, such as an alert to an engineer on call, cannot rely on a notification alone. It must require an explicit acknowledgement from the person and escalate to someone else when none arrives.

Preferences, quiet hours and priorities

Not every notification that is requested ought to be sent, and the service is the place where this is decided.

Users choose which kinds of message they wish to receive and by which channels, and in many countries the law requires that every promotional message offer a means of refusing further ones and that the refusal be honoured. The check against preferences belongs in the notification service itself, where no calling service can omit it by oversight. Users may also set hours during which they are not to be disturbed, and a message that falls within them is held and released when they end, unless it is of a kind, such as a security alert, that overrides them.

A limit on frequency protects users from the combined enthusiasm of many calling services, each of which considers its own messages important. A user who receives 30 notifications in a day will turn all of them off, the important ones included, and so the service limits the number of non-urgent notifications per user per day and discards or combines the excess.

Notifications differ enormously in urgency. A login code is useless unless it arrives within seconds, and a weekly digest may arrive at any time today. If both wait in the same queue, a campaign that addresses 10 million users will delay every login code until it has been worked through. The service therefore keeps separate queues for separate priorities, each with senders of its own, so that the bulk traffic cannot occupy the capacity on which urgent traffic depends.

Sending to very many users

A single request sometimes asks that millions of users be notified, of a change to the terms of service for example. Treated naively as 1 unit of work, it would occupy a sender for hours and would have to begin again from the start if that sender failed.

The request is fanned out. A first stage divides the list of recipients into batches of perhaps 1000 and places 1 message per batch in the queue, and a second stage expands each batch into individual notifications. The work is thereby spread across all the senders, a failure affects 1 batch only, and the progress of the whole is the number of batches completed. Each individual notification is given an idempotency key derived from the campaign and the user, so that a batch which is processed twice does not send anything twice. Such campaigns are sent at low priority and paced over minutes or hours, both to respect the providers’ limits and because a notification that brings a million users to the app within the same minute is an overload of one’s own making.

Use cases

Other services ask for notifications to be sent. Users decide which they are willing to receive.

Functional requirements

Non-functional requirements

The guarantee offered to callers is that an accepted request leads to at least 1 attempt to hand the message to a provider, and that a request repeated with the same key leads to no more than the first one did. The record of keys must be strongly consistent, since 2 nodes that disagreed about whether a key had been seen would between them send a duplicate. The state of a notification, by contrast, is eventually consistent with reality, since delivery is learned of after the fact, from the provider’s reports.

Core entities

Capacity estimate

Arithmetic ahead. 50 million notifications per day are 580 per second on average, and the queue absorbs peaks of 10 times that.

Skip to the high-level design

Show the calculation

Quantity	Calculation	Result
Per second, average	50000000 ÷ 86400	579
Per second, peak	579 × 10	5790
Senders, at 50 sends per second each	5790 ÷ 50	116