5. Kubernetes in a new cloud account

So far our cluster has existed before each chapter began. In this chapter, we will look at where a cluster comes from. We start with an account at a provider of machines, which holds nothing, and end with a set of machines that the API server knows as nodes and the scheduler can place GPU pods on.

Two separate things have to happen. Something has to ask the provider for machines and keep asking until the right number exist, and each machine, once it boots, has to prove to the cluster that it is allowed to join. We will take the second first, because we can watch it on the cluster we already have. Then we will write provisioner, in Go and in Rust, a model of the controller that asks for machines, and use it to study the one failure that makes this job harder than it looks: a request that succeeds while its reply is lost.

What the account gives us

A provider of machines, whether a public cloud or the team that racks hardware in a company’s own data center, offers the same small interface. We can ask for a machine of a given model, list the machines we have, and give one back. The account also has a quota for each model: the largest number of machines of that model it may hold at once.

Our fleet from chapter 4 fits this picture exactly, and its quotas are physical. There are five Jetson AGX Thors on the shelf and two DGX Sparks on the desk. A request for a sixth Thor fails for the same reason a request for a sixth GPU instance fails in a cloud account whose quota is five: the provider has no more to give us.

A new machine is a computer with an operating system and a network address. It knows nothing about our cluster, and our cluster knows nothing about it.

How a machine becomes a node

A machine becomes a node when the kubelet on it registers with the API server. From chapter 1 we know that every request to the API server is an HTTP request, and the API server requires each one to identify its sender. A kubelet identifies itself with a client certificate: a file that states a name, here the node’s name, and carries a signature from the cluster’s certificate authority (CA), a key pair the cluster created for itself when it was first set up. The API server trusts any certificate that its CA signed.

A new machine has no such certificate, and it cannot be handed the CA’s private key, since any holder of that key can impersonate every node in the cluster. The machine is given something much weaker instead, and trades it for a certificate. Figure 5.1 shows the exchange.

The weaker thing is a bootstrap token: a short string that the API server accepts for a limited time and for one purpose. Our kind cluster was created with kubeadm, the standard tool for setting up a cluster, and kubeadm can print the command a new machine would run to join. We have cut the two secret values from the output:

$ docker exec gpu-lab-control-plane kubeadm token create --ttl 10m --print-join-command
kubeadm join gpu-lab-control-plane:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

The command carries three facts, and each one answers a question the new machine has.

• gpu-lab-control-plane:6443 is where the API server listens.
• The token is what the machine will show to prove it was invited. We asked for it to expire after ten minutes.
• The hash is a fingerprint of the cluster’s CA public key. The machine uses it in the other direction: before it sends the token anywhere, it checks that the server it reached presents a certificate signed by the CA with this fingerprint. A machine without the hash could be tricked into sending its token to an impostor.

With the token accepted, the kubelet generates a key pair of its own and submits a CertificateSigningRequest, a Kubernetes object that says “please sign a certificate that names me system:node:thor-3”. The token is permitted to do only this. A controller in the control plane checks that the request comes from a bootstrap token and asks for a node identity, approves it, and signs it with the CA. The kubelet downloads the certificate and from then on uses it for every request. The private key was generated on the machine and has stayed there.

With its certificate, the kubelet creates the two objects we already know: a Node object, and the Lease from chapter 3 that it will renew every ten seconds.

What a node reports about itself

The Node object is where the kubelet tells the scheduler what the machine has. Here is the capacity our Orin NX reports, with the huge-page lines removed:

$ kubectl get node gpu-lab-control-plane -o jsonpath='{.status.capacity}'
{"cpu":"8","ephemeral-storage":"1966327652352","memory":"16031232Ki","nvidia.com/gpu":"1","pods":"110"}

Eight CPU cores, 16,031,232 KiB of memory, which is the board’s 15.3 GiB, and one nvidia.com/gpu, reported by the device plugin from Appendix A. The scheduler subtracts what running pods have requested from these numbers to find out what is free.

The Node object also carries labels: pairs of a key and a value that describe the machine.

$ kubectl get node gpu-lab-control-plane -o jsonpath='{.metadata.labels}'
{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64",
"kubernetes.io/hostname":"gpu-lab-control-plane","kubernetes.io/os":"linux",
"node-role.kubernetes.io/control-plane":""}

The kubelet sets the architecture, the operating system, and the host name by itself. Nothing here says which GPU the machine has. A count of nvidia.com/gpu: 1 is the same on an Orin NX with 16 GB and on a Thor with 128 GB, and a job that needs 60 GB of GPU memory fits on only one of them. On clusters that the GPU Operator supports, a component called GPU Feature Discovery adds labels such as nvidia.com/gpu.product and nvidia.com/gpu.memory to each node. On our fleet we would set a label by hand as each machine joins:

$ kubectl label node thor-3 gpucloud.dev/model=jetson-agx-thor

A pod then restricts itself to machines with that label through the nodeSelector field of its spec, and the scheduler considers only nodes whose labels match. Chapter 8 uses labels of this kind to place jobs.

Asking for machines

Now for the first half: getting the machines to exist. We describe what we want as pools. A pool is a model and a count: five Thors, two Sparks. The controller’s job is to make the account hold exactly that.

This is reconciliation again, with the provider’s interface in place of the API server. The controller compares the wanted count with the machines it has and creates the difference. The difficulty lies in a property of the create call that the calls in chapter 1 did not have. Creating a machine changes the world outside the cluster, costs money while the machine runs, and counts against a quota.

Consider what the controller knows after a create call. There are three outcomes:

1. A reply arrives with the new machine’s identifier. The machine exists.
2. A reply arrives with an error such as “quota exceeded”. No machine was created.
3. No reply arrives before the controller’s timeout.

The third outcome is the awkward one, because it covers two different situations that look identical from the controller’s side. The request may have been lost on the way, in which case nothing was created. Or the provider may have created the machine and the reply was lost on the way back. Figure 5.2 follows the second situation.

If the controller assumes nothing was created and sends another create, it may end up with two machines where it wanted one. It owns the second and has no record of the first, which runs, costs money, and uses up quota until someone finds it. If the controller assumes the machine was created, it has no identifier to manage it by.

A token that names the request

The remedy is to let the controller name each request. The controller attaches a client token to the call: a string it chose, which identifies this particular machine-to-be. The provider stores the token with the machine it creates. When a second call arrives with a token it has already seen, the provider creates nothing and returns the existing machine.

A call with this property is called idempotent: sending it twice has the same effect as sending it once. With an idempotent create, the controller needs no way to tell the two timeout situations apart. It sends the same call again, and the answer is the same machine either way.

The token has to be the same on the retry, so it cannot be random. Our controller derives it from the pool’s name and the slot it is filling: the third Thor’s token is thor-3, on every attempt. Real providers work the same way. The call that creates a virtual machine in Amazon EC2 takes a ClientToken parameter for exactly this purpose.

Building provisioner

The provider

The provider is the account. Create does three things in order. It looks for a machine that already carries the caller’s token and, if it finds one, returns it. It checks the quota. Then it creates the machine, and only after that does the run get the chance to lose the reply. That ordering is what makes the lost reply dangerous: the machine is in the list before the caller learns anything.

Listing 5.1 The provider and its create call, in Go. code/go/provisioner/main.go

// Machine is one machine the provider has created.
type Machine struct {
	// ID is the provider's identifier for the machine.
	ID string
	// Model is the kind of machine.
	Model string
	// Token is the client token of the call that created it, or "".
	Token string
}

// Errors a create call can return.
var (
	// ErrTimeout means no reply arrived. The machine may or may not exist.
	ErrTimeout = errors.New("timed out")
	// ErrQuota means the account already holds as many machines of this model
	// as it is allowed.
	ErrQuota = errors.New("quota exceeded")
)

// Provider is the account that machines are created in.
type Provider struct {
	// quota is the number of machines of each model the account may hold.
	quota map[string]int
	// machines lists every machine that exists, in creation order.
	machines []Machine
	// calls counts create calls, so the run can lose the reply to one of them.
	calls int
	// loseReply is the number of the call whose reply is lost.
	loseReply int
}

// count returns how many machines of a model exist.
func (p *Provider) count(model string) int {
	n := 0

	for _, m := range p.machines {
		if m.Model == model {
			n++
		}
	}

	return n
}

// Create makes one machine. A call that carries a token already used by an
// existing machine returns that machine and creates nothing.
func (p *Provider) Create(model, token string) (Machine, error) {
	p.calls++

	if token != "" {
		for _, m := range p.machines {
			if m.Token == token {
				return m, nil
			}
		}
	}

	if p.count(model) >= p.quota[model] {
		return Machine{}, ErrQuota
	}

	m := Machine{ID: fmt.Sprintf("m-%02d", len(p.machines)+1), Model: model, Token: token}
	p.machines = append(p.machines, m)

	if p.calls == p.loseReply {
		return Machine{}, ErrTimeout
	}

	return m, nil
}

Listing 5.2 The provider and its create call, in Rust. code/rust/provisioner/src/main.rs

/// One machine the provider has created.
#[derive(Clone)]
struct Machine {
    /// The provider's identifier for the machine.
    id: String,
    /// The kind of machine.
    model: &'static str,
    /// The client token of the call that created it, if it carried one.
    token: Option<String>,
}

/// The ways a create call can fail.
enum CreateError {
    /// No reply arrived. The machine may or may not exist.
    Timeout,
    /// The account already holds as many machines of this model as it is
    /// allowed.
    Quota,
}

impl fmt::Display for CreateError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str(match self {
            CreateError::Timeout => "timed out",
            CreateError::Quota => "quota exceeded",
        })
    }
}

/// The account that machines are created in.
struct Provider {
    /// The number of machines of each model the account may hold.
    quota: BTreeMap<&'static str, usize>,
    /// Every machine that exists, in creation order.
    machines: Vec<Machine>,
    /// Counts create calls, so the run can lose the reply to one of them.
    calls: usize,
    /// The number of the call whose reply is lost.
    lose_reply: usize,
}

impl Provider {
    /// Returns how many machines of a model exist.
    fn count(&self, model: &str) -> usize {
        self.machines.iter().filter(|m| m.model == model).count()
    }

    /// Makes one machine. A call that carries a token already used by an
    /// existing machine returns that machine and creates nothing.
    fn create(&mut self, model: &'static str, token: Option<&str>) -> Result<Machine, CreateError> {
        self.calls += 1;
        if let Some(token) = token
            && let Some(existing) = self
                .machines
                .iter()
                .find(|m| m.token.as_deref() == Some(token))
        {
            return Ok(existing.clone());
        }

        if self.count(model) >= self.quota[model] {
            return Err(CreateError::Quota);
        }

        let machine = Machine {
            id: format!("m-{:02}", self.machines.len() + 1),
            model,
            token: token.map(str::to_owned),
        };
        self.machines.push(machine.clone());

        if self.calls == self.lose_reply {
            return Err(CreateError::Timeout);
        }

        Ok(machine)
    }
}

The Rust version represents a missing token as None, and the lookup is an if let ... && let ... chain, the same construct we used in section 1.1: the first let unwraps the caller’s token, and the second searches for a machine that carries it.

The controller

Reconcile works on one pool. It names the next empty slot, builds the token from that name when tokens are switched on, and calls the provider. A failed call ends the pass for this pool, and the controller records nothing for it. The next pass starts from the same slot.

Listing 5.3 The pool and the controller's reconcile, in Go. code/go/provisioner/main.go

// Pool is a group of identical machines and the size its owner wants.
type Pool struct {
	// Name prefixes the name of each slot: thor-1, thor-2, and so on.
	Name string
	// Model is the kind of machine in the pool.
	Model string
	// Want is the number of machines the pool should have.
	Want int
}

// Controller brings each pool to its wanted size.
type Controller struct {
	provider *Provider
	// tokens says whether create calls carry a client token.
	tokens bool
	// owned maps a pool's name to the machines the controller knows it has.
	owned map[string][]Machine
}

// Reconcile fills the empty slots of one pool, in order, and stops at the
// first call that fails. It returns true when the pool is full.
func (c *Controller) Reconcile(pool Pool) bool {
	for len(c.owned[pool.Name]) < pool.Want {
		slot := fmt.Sprintf("%s-%d", pool.Name, len(c.owned[pool.Name])+1)
		token := ""

		if c.tokens {
			token = slot
		}

		m, err := c.provider.Create(pool.Model, token)

		if err != nil {
			fmt.Printf("    %-8s create failed: %v\n", slot, err)

			return false
		}

		fmt.Printf("    %-8s is machine %s\n", slot, m.ID)
		c.owned[pool.Name] = append(c.owned[pool.Name], m)
	}

	return true
}

Listing 5.4 The pool and the controller's reconcile, in Rust. code/rust/provisioner/src/main.rs

/// A group of identical machines and the size its owner wants.
struct Pool {
    /// Prefixes the name of each slot: thor-1, thor-2, and so on.
    name: &'static str,
    /// The kind of machine in the pool.
    model: &'static str,
    /// The number of machines the pool should have.
    want: usize,
}

/// Brings each pool to its wanted size.
struct Controller {
    provider: Provider,
    /// Whether create calls carry a client token.
    tokens: bool,
    /// Maps a pool's name to the machines the controller knows it has.
    owned: BTreeMap<&'static str, Vec<Machine>>,
}

impl Controller {
    /// Fills the empty slots of one pool, in order, and stops at the first
    /// call that fails. It returns true when the pool is full.
    fn reconcile(&mut self, pool: &Pool) -> bool {
        let owned = self.owned.entry(pool.name).or_default();

        while owned.len() < pool.want {
            let slot = format!("{}-{}", pool.name, owned.len() + 1);
            let token = self.tokens.then_some(slot.as_str());

            match self.provider.create(pool.model, token) {
                Ok(machine) => {
                    println!("    {slot:<8} is machine {}", machine.id);
                    owned.push(machine);
                }
                Err(e) => {
                    println!("    {slot:<8} create failed: {e}");

                    return false;
                }
            }
        }

        true
    }
}

The controller counts only what it has been told. owned grows by one machine for each successful reply, and Reconcile reads the pool’s size from it. This is where a lost reply does its damage in the first run, as the output will show.

The run

The run asks for five Thors and two Sparks, with quotas of exactly five and two, and loses the reply to the third create call. It allows three passes and then prints what the controller owns beside what the provider holds.

Listing 5.5 The pools, the three passes, and the final comparison, in Go. code/go/provisioner/main.go

// pools is the fleet we ask for: the Thors on the shelf and the two Sparks.
var pools = []Pool{
	{"thor", "Jetson AGX Thor", 5},
	{"spark", "DGX Spark", 2},
}

// run reconciles every pool for three passes and then compares what the
// controller owns with what the provider holds.
func run(tokens bool) {
	provider := &Provider{quota: map[string]int{"Jetson AGX Thor": 5, "DGX Spark": 2}, loseReply: 3}
	c := &Controller{provider: provider, tokens: tokens, owned: map[string][]Machine{}}

	for pass := 1; pass <= 3; pass++ {
		fmt.Printf("  pass %d\n", pass)
		full := true

		for _, pool := range pools {
			full = c.Reconcile(pool) && full
		}

		if full {
			fmt.Println("    every pool is full")
			break
		}
	}

	known := map[string]bool{}

	for _, pool := range pools {
		for _, m := range c.owned[pool.Name] {
			known[m.ID] = true
		}

		fmt.Printf("  %-5s wanted %d, controller owns %d, provider holds %d\n",
			pool.Name, pool.Want, len(c.owned[pool.Name]), provider.count(pool.Model))
	}

	for _, m := range provider.machines {
		if !known[m.ID] {
			fmt.Printf("  machine %s (%s) is running and belongs to no pool\n", m.ID, m.Model)
		}
	}
}

func main() {
	fmt.Println("create calls without a client token:")
	run(false)
	fmt.Println("create calls with the slot name as the client token:")
	run(true)
}

Listing 5.6 The pools, the three passes, and the final comparison, in Rust. code/rust/provisioner/src/main.rs

/// The fleet we ask for: the Thors on the shelf and the two Sparks.
const POOLS: [Pool; 2] = [
    Pool {
        name: "thor",
        model: "Jetson AGX Thor",
        want: 5,
    },
    Pool {
        name: "spark",
        model: "DGX Spark",
        want: 2,
    },
];

/// Reconciles every pool for three passes and then compares what the
/// controller owns with what the provider holds.
fn run(tokens: bool) {
    let provider = Provider {
        quota: BTreeMap::from([("Jetson AGX Thor", 5), ("DGX Spark", 2)]),
        machines: Vec::new(),
        calls: 0,
        lose_reply: 3,
    };
    let mut c = Controller {
        provider,
        tokens,
        owned: BTreeMap::new(),
    };

    for pass in 1..=3 {
        println!("  pass {pass}");
        let mut full = true;

        for pool in &POOLS {
            full = c.reconcile(pool) && full;
        }

        if full {
            println!("    every pool is full");
            break;
        }
    }

    let mut known = BTreeSet::new();

    for pool in &POOLS {
        let owned = &c.owned[pool.name];
        known.extend(owned.iter().map(|m| m.id.as_str()));
        println!(
            "  {:<5} wanted {}, controller owns {}, provider holds {}",
            pool.name,
            pool.want,
            owned.len(),
            c.provider.count(pool.model)
        );
    }

    for m in c
        .provider
        .machines
        .iter()
        .filter(|m| !known.contains(m.id.as_str()))
    {
        println!(
            "  machine {} ({}) is running and belongs to no pool",
            m.id, m.model
        );
    }
}

fn main() {
    println!("create calls without a client token:");
    run(false);
    println!("create calls with the slot name as the client token:");
    run(true);
}

$ go run ./provisioner
create calls without a client token:
  pass 1
    thor-1   is machine m-01
    thor-2   is machine m-02
    thor-3   create failed: timed out
    spark-1  is machine m-04
    spark-2  is machine m-05
  pass 2
    thor-3   is machine m-06
    thor-4   is machine m-07
    thor-5   create failed: quota exceeded
  pass 3
    thor-5   create failed: quota exceeded
  thor  wanted 5, controller owns 4, provider holds 5
  spark wanted 2, controller owns 2, provider holds 2
  machine m-03 (Jetson AGX Thor) is running and belongs to no pool
create calls with the slot name as the client token:
  pass 1
    thor-1   is machine m-01
    thor-2   is machine m-02
    thor-3   create failed: timed out
    spark-1  is machine m-04
    spark-2  is machine m-05
  pass 2
    thor-3   is machine m-03
    thor-4   is machine m-06
    thor-5   is machine m-07
    every pool is full
  thor  wanted 5, controller owns 5, provider holds 5
  spark wanted 2, controller owns 2, provider holds 2

Both runs begin the same way. The third call creates machine m-03 and its reply is lost, so the controller reports a timeout for thor-3 and moves on to the Sparks.

In the first run, pass 2 sends a fresh create for thor-3. The provider has no way to connect it with the earlier call and creates m-06. The controller fills thor-4 with m-07, and then the create for thor-5 fails: the account now holds five Thors, which is the whole quota. Pass 3 fails in the same way, and so would every later pass. The controller owns four Thors and has no way to get a fifth, while m-03 runs with no owner. The pool is one machine short and the account is full.

In the second run, pass 2 sends the create for thor-3 again with the token thor-3. The provider finds m-03 under that token and returns it. The controller then creates thor-4 and thor-5, and every pool is full after two passes, with nothing left over.

Animation 5.1 replays both runs for the pool of Thors.

The same design as Kubernetes objects

Cluster API is the Kubernetes project that manages clusters with Kubernetes objects. It runs in one cluster, called the management cluster, and creates others. Its objects map onto our model directly:

A Machine object is created first and the provider’s machine second, so the object’s own name can serve as the client token, and a controller that restarts finds every slot it was working on by listing Machine objects. The record of ownership lives in the API server, where the resource versions of chapter 1 protect it, and the provider’s account is reconciled towards it.

The complete program

Listing 5.7 The complete program in Go. code/go/provisioner/main.go

// Command provisioner models a controller that asks a provider for machines
// until a pool has the size its owner asked for.
//
// The provider's create call can fail in an awkward way: the machine is
// created and the reply is lost. The program runs the same controller twice.
// The first run retries a lost reply with a plain create call. The second
// sends a client token with every call, so a retry returns the machine that
// the first attempt created.
//
// Run it from code/go with:
//
//	go run ./provisioner
package main

import (
	"errors"
	"fmt"
)

// Machine is one machine the provider has created.
type Machine struct {
	// ID is the provider's identifier for the machine.
	ID string
	// Model is the kind of machine.
	Model string
	// Token is the client token of the call that created it, or "".
	Token string
}

// Errors a create call can return.
var (
	// ErrTimeout means no reply arrived. The machine may or may not exist.
	ErrTimeout = errors.New("timed out")
	// ErrQuota means the account already holds as many machines of this model
	// as it is allowed.
	ErrQuota = errors.New("quota exceeded")
)

// Provider is the account that machines are created in.
type Provider struct {
	// quota is the number of machines of each model the account may hold.
	quota map[string]int
	// machines lists every machine that exists, in creation order.
	machines []Machine
	// calls counts create calls, so the run can lose the reply to one of them.
	calls int
	// loseReply is the number of the call whose reply is lost.
	loseReply int
}

// count returns how many machines of a model exist.
func (p *Provider) count(model string) int {
	n := 0

	for _, m := range p.machines {
		if m.Model == model {
			n++
		}
	}

	return n
}

// Create makes one machine. A call that carries a token already used by an
// existing machine returns that machine and creates nothing.
func (p *Provider) Create(model, token string) (Machine, error) {
	p.calls++

	if token != "" {
		for _, m := range p.machines {
			if m.Token == token {
				return m, nil
			}
		}
	}

	if p.count(model) >= p.quota[model] {
		return Machine{}, ErrQuota
	}

	m := Machine{ID: fmt.Sprintf("m-%02d", len(p.machines)+1), Model: model, Token: token}
	p.machines = append(p.machines, m)

	if p.calls == p.loseReply {
		return Machine{}, ErrTimeout
	}

	return m, nil
}


// Pool is a group of identical machines and the size its owner wants.
type Pool struct {
	// Name prefixes the name of each slot: thor-1, thor-2, and so on.
	Name string
	// Model is the kind of machine in the pool.
	Model string
	// Want is the number of machines the pool should have.
	Want int
}

// Controller brings each pool to its wanted size.
type Controller struct {
	provider *Provider
	// tokens says whether create calls carry a client token.
	tokens bool
	// owned maps a pool's name to the machines the controller knows it has.
	owned map[string][]Machine
}

// Reconcile fills the empty slots of one pool, in order, and stops at the
// first call that fails. It returns true when the pool is full.
func (c *Controller) Reconcile(pool Pool) bool {
	for len(c.owned[pool.Name]) < pool.Want {
		slot := fmt.Sprintf("%s-%d", pool.Name, len(c.owned[pool.Name])+1)
		token := ""

		if c.tokens {
			token = slot
		}

		m, err := c.provider.Create(pool.Model, token)

		if err != nil {
			fmt.Printf("    %-8s create failed: %v\n", slot, err)

			return false
		}

		fmt.Printf("    %-8s is machine %s\n", slot, m.ID)
		c.owned[pool.Name] = append(c.owned[pool.Name], m)
	}

	return true
}


// pools is the fleet we ask for: the Thors on the shelf and the two Sparks.
var pools = []Pool{
	{"thor", "Jetson AGX Thor", 5},
	{"spark", "DGX Spark", 2},
}

// run reconciles every pool for three passes and then compares what the
// controller owns with what the provider holds.
func run(tokens bool) {
	provider := &Provider{quota: map[string]int{"Jetson AGX Thor": 5, "DGX Spark": 2}, loseReply: 3}
	c := &Controller{provider: provider, tokens: tokens, owned: map[string][]Machine{}}

	for pass := 1; pass <= 3; pass++ {
		fmt.Printf("  pass %d\n", pass)
		full := true

		for _, pool := range pools {
			full = c.Reconcile(pool) && full
		}

		if full {
			fmt.Println("    every pool is full")
			break
		}
	}

	known := map[string]bool{}

	for _, pool := range pools {
		for _, m := range c.owned[pool.Name] {
			known[m.ID] = true
		}

		fmt.Printf("  %-5s wanted %d, controller owns %d, provider holds %d\n",
			pool.Name, pool.Want, len(c.owned[pool.Name]), provider.count(pool.Model))
	}

	for _, m := range provider.machines {
		if !known[m.ID] {
			fmt.Printf("  machine %s (%s) is running and belongs to no pool\n", m.ID, m.Model)
		}
	}
}

func main() {
	fmt.Println("create calls without a client token:")
	run(false)
	fmt.Println("create calls with the slot name as the client token:")
	run(true)
}

Listing 5.8 The complete program in Rust. code/rust/provisioner/src/main.rs

//! Models a controller that asks a provider for machines until a pool has the
//! size its owner asked for.
//!
//! The provider's create call can fail in an awkward way: the machine is
//! created and the reply is lost. The program runs the same controller twice.
//! The first run retries a lost reply with a plain create call. The second
//! sends a client token with every call, so a retry returns the machine that
//! the first attempt created.
//!
//! Run it from code/rust with:
//!
//! ```text
//! cargo run -p provisioner
//! ```

use std::collections::{BTreeMap, BTreeSet};
use std::fmt;

/// One machine the provider has created.
#[derive(Clone)]
struct Machine {
    /// The provider's identifier for the machine.
    id: String,
    /// The kind of machine.
    model: &'static str,
    /// The client token of the call that created it, if it carried one.
    token: Option<String>,
}

/// The ways a create call can fail.
enum CreateError {
    /// No reply arrived. The machine may or may not exist.
    Timeout,
    /// The account already holds as many machines of this model as it is
    /// allowed.
    Quota,
}

impl fmt::Display for CreateError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str(match self {
            CreateError::Timeout => "timed out",
            CreateError::Quota => "quota exceeded",
        })
    }
}

/// The account that machines are created in.
struct Provider {
    /// The number of machines of each model the account may hold.
    quota: BTreeMap<&'static str, usize>,
    /// Every machine that exists, in creation order.
    machines: Vec<Machine>,
    /// Counts create calls, so the run can lose the reply to one of them.
    calls: usize,
    /// The number of the call whose reply is lost.
    lose_reply: usize,
}

impl Provider {
    /// Returns how many machines of a model exist.
    fn count(&self, model: &str) -> usize {
        self.machines.iter().filter(|m| m.model == model).count()
    }

    /// Makes one machine. A call that carries a token already used by an
    /// existing machine returns that machine and creates nothing.
    fn create(&mut self, model: &'static str, token: Option<&str>) -> Result<Machine, CreateError> {
        self.calls += 1;
        if let Some(token) = token
            && let Some(existing) = self
                .machines
                .iter()
                .find(|m| m.token.as_deref() == Some(token))
        {
            return Ok(existing.clone());
        }

        if self.count(model) >= self.quota[model] {
            return Err(CreateError::Quota);
        }

        let machine = Machine {
            id: format!("m-{:02}", self.machines.len() + 1),
            model,
            token: token.map(str::to_owned),
        };
        self.machines.push(machine.clone());

        if self.calls == self.lose_reply {
            return Err(CreateError::Timeout);
        }

        Ok(machine)
    }
}

/// A group of identical machines and the size its owner wants.
struct Pool {
    /// Prefixes the name of each slot: thor-1, thor-2, and so on.
    name: &'static str,
    /// The kind of machine in the pool.
    model: &'static str,
    /// The number of machines the pool should have.
    want: usize,
}

/// Brings each pool to its wanted size.
struct Controller {
    provider: Provider,
    /// Whether create calls carry a client token.
    tokens: bool,
    /// Maps a pool's name to the machines the controller knows it has.
    owned: BTreeMap<&'static str, Vec<Machine>>,
}

impl Controller {
    /// Fills the empty slots of one pool, in order, and stops at the first
    /// call that fails. It returns true when the pool is full.
    fn reconcile(&mut self, pool: &Pool) -> bool {
        let owned = self.owned.entry(pool.name).or_default();

        while owned.len() < pool.want {
            let slot = format!("{}-{}", pool.name, owned.len() + 1);
            let token = self.tokens.then_some(slot.as_str());

            match self.provider.create(pool.model, token) {
                Ok(machine) => {
                    println!("    {slot:<8} is machine {}", machine.id);
                    owned.push(machine);
                }
                Err(e) => {
                    println!("    {slot:<8} create failed: {e}");

                    return false;
                }
            }
        }

        true
    }
}

/// The fleet we ask for: the Thors on the shelf and the two Sparks.
const POOLS: [Pool; 2] = [
    Pool {
        name: "thor",
        model: "Jetson AGX Thor",
        want: 5,
    },
    Pool {
        name: "spark",
        model: "DGX Spark",
        want: 2,
    },
];

/// Reconciles every pool for three passes and then compares what the
/// controller owns with what the provider holds.
fn run(tokens: bool) {
    let provider = Provider {
        quota: BTreeMap::from([("Jetson AGX Thor", 5), ("DGX Spark", 2)]),
        machines: Vec::new(),
        calls: 0,
        lose_reply: 3,
    };
    let mut c = Controller {
        provider,
        tokens,
        owned: BTreeMap::new(),
    };

    for pass in 1..=3 {
        println!("  pass {pass}");
        let mut full = true;

        for pool in &POOLS {
            full = c.reconcile(pool) && full;
        }

        if full {
            println!("    every pool is full");
            break;
        }
    }

    let mut known = BTreeSet::new();

    for pool in &POOLS {
        let owned = &c.owned[pool.name];
        known.extend(owned.iter().map(|m| m.id.as_str()));
        println!(
            "  {:<5} wanted {}, controller owns {}, provider holds {}",
            pool.name,
            pool.want,
            owned.len(),
            c.provider.count(pool.model)
        );
    }

    for m in c
        .provider
        .machines
        .iter()
        .filter(|m| !known.contains(m.id.as_str()))
    {
        println!(
            "  machine {} ({}) is running and belongs to no pool",
            m.id, m.model
        );
    }
}

fn main() {
    println!("create calls without a client token:");
    run(false);
    println!("create calls with the slot name as the client token:");
    run(true);
}

References

• Kubernetes, TLS bootstrapping: https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/
• Kubernetes, Authenticating with bootstrap tokens: https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
• Kubernetes, kubeadm join: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/
• Amazon EC2, Ensuring idempotency in API requests: https://docs.aws.amazon.com/ec2/latest/devguide/ec2-api-idempotency.html
• The Cluster API Book: https://cluster-api.sigs.k8s.io/
• NVIDIA, GPU Feature Discovery: https://github.com/NVIDIA/k8s-device-plugin/blob/main/docs/gpu-feature-discovery/README.md