Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Four tigresses in a row joined by arrows: Hammer, the reviewer with a magnifier, Lasso and Spark

High-level design

This chapter covers

  • The components of the platform and the data passed between them
  • The memory and bandwidth limits of the Jetson AGX Thor
  • The design decisions that follow from those limits

1.1 Components

Three sources feed thor-hammer-trainer, which writes train.jsonl. train.jsonl feeds the review page and the planned chain: QLoRA training, merge and quantize, TensorRT-LLM engine, trtllm-serve. trtllm-serve feeds the planned human evaluation, whose verdicts feed the next data round.
Figure 1.1 Components and data flow. Green: built. Dashed amber: planned.
ComponentInputOutputResponsibility
thor-hammer-trainersynthetic chat set, curated sets, book repository, slop flagstrain.jsonl, preferences.jsonl, slop.jsonl, stats.mdconvert files to instruction/response pairs, drop unusable and flagged ones
Review servertrain.jsonl, slop.jsonllabels/slop_flags.jsonldisplay examples, highlight personal data, record slop flags
QLoRA trainingtrain.jsonl, base modeladapter weightsfine-tune the model
Merge, quantize, buildbase model, adapterTensorRT-LLM engineproduce a servable model
trtllm-serveengineHTTP endpoint on localhostanswer prompts
Human evaluationprompts, two endpointsverdicts.jsonlcompare base and fine-tuned answers

The components share files, not memory or APIs. train.jsonl is the only interface between data generation and training. One line of it:

{"id":"…","instruction":"What is a futex?","response":"A fast userspace mutex. ...","source":"chat","origin":"conversations.json#c1/1"}

1.2 Hardware limits

PropertyValue
Memory128 GB LPDDR5X, unified between CPU and GPU, about 115 GB usable
Memory bandwidth273 GB/s
CPU14 Arm Neoverse-V3AE cores
GPUBlackwell, 2,560 CUDA cores, 96 tensor cores, compute capability sm_110
SoftwareJetPack 7, CUDA 13, Python 3.12

In unified memory, the CPU and GPU address the same physical memory. A model loaded by the CPU is readable by the GPU without a copy.

A bar of 128 GB with about 115 GB usable. Reported peak memory: QLoRA on a 27B model about 28 GB, full fine-tuning of a 4B model about 42 GB, LoRA on a 9B model about 50 GB.
Figure 1.2 Usable memory and reported peak memory per training method. The three budgets are from the project notes, not measured here.

Consequences for the design:

  • Model size. A 9B-parameter large language model in 16-bit precision needs about 18 GB for its weights. Full fine-tuning adds gradients and optimizer state, several times that. LoRA and QLoRA train small adapter matrices only.
  • Step time. Each training step reads all weights. At 273 GB/s, reading 18 GB takes about 66 ms. Training is bound by memory bandwidth, not by compute. The 2,070 FP4 TFLOPS figure is for sparse inference and does not apply to training.
  • Software support. sm_110 is a new target. There is no 4-bit training stack in Rust and no mature FlashAttention build for it.

1.3 Design decisions

DecisionReason
Data generation and review in Rustplain file processing; one static binary
Training in Python (Unsloth, TRL)4-bit QLoRA is available there; Candle has no 4-bit training, and a 9B model does not fit otherwise
A file as the only interface between stageseither side can change independently; the file can be inspected with standard tools
Manual review before traininga fine-tuned model can reproduce training text verbatim, including personal data
Base and fine-tuned model served with the same quantizationthe comparison then measures training, not quantization
Human judgement as the evaluation metriccorrectness and clarity of an explanation cannot be checked by pattern matching

1.4 Crates and machines today

Sources feed thor-hammer-trainer, which writes data/train.jsonl, read by the reinforcer review page and by lasso. Model answers go to spark, which writes scores.
Figure 1.3 The crates and the files between them.
CrateBinaryDoes
thor-spark-safety-evalsparkscores answers for slop phrases and the five Rust rules
thor-hammer-trainerthor-hammer-trainerbuilds data/train.jsonl from licensed sources
thor-tigress-reinforcer-frontendreinforcerreview page: mark slop, flag records
thor-lasso-distillerlassoturns book sections into conversations using a teacher model
thor-tigress-agentthor-tigress-agentthe Thor Tigress Cub’s server: page, access key, web search, API

Machines

Development machineThe Thor
Used for this bookJetson Orin NX 16 GB, Ubuntu (named yahboom)Jetson AGX Thor, 128 GB
Roleediting, data builds, running agentsmodels, training, shared chat
Could beany Linux or macOS machine with SSH access to the Thor

Code is edited on the development machine and copied to the Thor with thor-sync (chapter “Access and syncing”).