The training data

2.1 Example format

An example is one instruction and one response. The training set is stored as JSONL: one JSON object per line.

2.2 Sources

The answer set is synthetic. Open-source models between roughly one and twenty billion parameters answer a fixed prompt list, and the answers are exported as a transcript. The same models write programs that break the five rules. The clever-vs-readable set keeps the readable rewrite as the chosen answer and the sloppy version as the rejected one. The corrections come from the senior engineer who reviews every sample the pipeline keeps.

2.3 Composition

A token is the unit a model reads. Counts are estimated at four characters per token. The median example is 1,494 characters. One example exceeds 8,192 tokens.

The two curated sets together are 1.7% of the tokens. Chapter 5 specifies repeating them during training.

2.4 Exclusion rules

From assistant messages, only text blocks are kept. Thinking, tool calls and tool results are dropped.

anti_ai_slop.md is excluded because it consists of the phrases the model must not produce.

2.5 Preference pairs

A preference pair is one prompt with a chosen and a rejected answer. Training on such pairs (DPO, direct preference optimization) moves the model toward the chosen style and away from the rejected one.

The clever-vs-readable set contains 37 pairs: chosen is the readable rewrite, rejected is the clever original. Both versions of every entry compile, run and produce the same output (verification_report.txt: 37/37 pass). They are written unchanged to data/preferences.jsonl in the conversational format of TRL’s DPO trainer:

{"prompt":[{"role":"system",...},{"role":"user",...}],"chosen":[{"role":"assistant",...}],"rejected":[{"role":"assistant",...}],"meta":{...}}

The system message of the SFT records is dropped from train.jsonl. It is identical in all 74 records, and the training script supplies one system prompt per run.

2.6 Slop flags

A reviewer flags examples on the review page at two levels:

Span categories are the sections of anti_ai_slop.md:

One line per flagged example in labels/slop_flags.jsonl:

{"id":"2e6e9c9623674c82","note":"opener and sign-off","spans":[{"field":"response","text":"Great question!","category":"flattery_filler_opener"}]}

A span stores the selected text, not character offsets, so it stays valid if whitespace in the source changes.

On the next build, flagged examples are:

• removed from train.jsonl (SkipReason::FlaggedSlop)
• written to data/slop.jsonl with the note and spans, as a set of negative examples
• counted per span category in stats.md

labels/ is outside the generated data/ folder and is committed. Flags stay attached as long as the example text, and therefore its id, does not change.

2.7 Personal data in the current set

The review server highlights 20 examples. Most matches are placeholders in code, such as user@example.com. The rest:

These must be removed or redacted before Stage 2. Names and personal details have no fixed pattern and are not flagged; the chat source requires reading.

2.8 The current build

bash train/fetch_corpus.sh          # clone the open books and docs
thor-hammer-trainer data            # build data/train.jsonl and data/stats.md

• Book and doc sections are passages: plain text with no made-up question. Questions get written later by a teacher model (lasso).
• Licences: only MIT, Apache-2.0, BSD, CC BY and CC0. One restrictive licence file rules a source out.
• Cap: each source gives at most 400,000 tokens, so a few large doc sites don’t drown out the Rust books.
• Flags: a record flagged on the review page leaves the next build, but only if a person set the flag.

Not done yet: dropping chat answers that fail spark.