Low-level design: the data generator

thor-hammer-trainer is a Rust binary of about 1,300 lines, tests and doc comments included. A full build of the training set takes under 0.3 s.

3.1 Modules

Readers do not depend on each other. A new source is one new module and one call in main.rs.

3.2 Shared types

Example is one line of train.jsonl and derives Serialize:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct Example {
    pub instruction: String,
    pub response: String,
    pub source: Source,
    pub origin: String,
}
}

SkipReason records why an input was excluded. Readers push reasons into a Vec<SkipReason>; report.rs counts them.

#![allow(unused)]
fn main() {
pub enum SkipReason {
    SafetyFlag,
    AboutImage,
    EmptyTurn,
    NotesFile,
    NoHeaderComment,
    TooShort,
    Duplicate,
}
}

DataError covers failures that stop the build: I/O errors and invalid JSON.

3.3 Chat pairing

The transcript is deserialized with serde into types that declare only the fields the generator reads. Message content blocks are distinguished by their type field:

#![allow(unused)]
fn main() {
#[derive(Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
enum Content {
    Text { text: String },
    Flag {},
    #[serde(other)]
    Hidden,
}
}

#[serde(other)] maps every unlisted block type, including thinking, tool_use and tool_result, to Hidden. Hidden blocks are never read.

Pairing is one pass over the messages with one match on the sender and the state of the last pair:

#![allow(unused)]
fn main() {
fn question_answer_pairs(messages: &[Message]) -> Vec<QuestionAnswer> {
    let mut pairs: Vec<QuestionAnswer> = Vec::new();
    for message in messages {
        match (&message.sender, pairs.last_mut()) {
            (Sender::Human, Some(open_pair)) if !open_pair.is_answered() => {
                open_pair.add_question(message)
            }
            (Sender::Human, Some(..) | None) => pairs.push(QuestionAnswer::asked_in(message)),
            (Sender::Assistant, Some(current_pair)) => current_pair.add_answer(message),
            (Sender::Assistant, None) => {}
        }
    }
    pairs
}
}

A pair becomes an example or a skip reason:

A conversation containing a Flag block is excluded before pairing results are used, with one SkipReason::SafetyFlag per turn.

3.4 Chapter splitting

Shell listings in the corpus contain # comment lines. The splitter tracks code fences and classifies each line:

#![allow(unused)]
fn main() {
fn line_kind<'a>(line: &'a str, heading_markers: &[&str], inside_code_block: bool) -> LineKind<'a> {
    let is_code_fence =
        line.trim_start().starts_with("```") || line.trim_start().starts_with("~~~");
    let heading = heading_markers
        .iter()
        .find_map(|marker| line.strip_prefix(marker));
    match (is_code_fence, inside_code_block, heading) {
        (true, ..) => LineKind::CodeFence,
        (false, false, Some(heading)) => LineKind::Heading(heading),
        (false, true, ..) | (false, false, None) => LineKind::Text,
    }
}
}

Split depth. Chapters are split at # and ##. A section over 12,000 characters (about 3,000 tokens) is split again at ###. Measured on the corpus: 4 of 674 ## sections exceed the limit, and no ### subsection of those 4 does. No deeper splitting is implemented.

Instruction text. Section numbers such as 13.1 are removed from the heading, then:

#![allow(unused)]
fn main() {
fn instruction(chapter: &str, heading: &str) -> String {
    match heading {
        introduction if introduction == chapter => format!("Introduce \"{chapter}\"."),
        question if question.ends_with('?') => format!("{question} (Context: {chapter}.)"),
        topic => format!("Explain \"{topic}\" from \"{chapter}\"."),
    }
}
}

WARNING: All 690 book instructions follow these three templates. A model can learn the template wording. Stage 3 is planned to replace them with questions written by a teacher model.

3.5 Source files

#![allow(unused)]
fn main() {
const LANGUAGES: [Language; 4] = [
    Language { extension: "rs", name: "Rust", code_block_tag: "rust", header_marker: "//!" },
    Language { extension: "go", name: "Go", code_block_tag: "go", header_marker: "//" },
    Language { extension: "cu", name: "CUDA", code_block_tag: "cuda", header_marker: "//" },
    Language { extension: "py", name: "Python", code_block_tag: "python", header_marker: "#" },
];
}

The leading comment lines, without markers, form the instruction Write a <language> program for this: <header>. The file in a code block forms the response. Files without a header comment produce SkipReason::NoHeaderComment.

3.6 Directory walk

classify maps each path to a CorpusFile:

3.7 Filtering and output

#![allow(unused)]
fn main() {
fn rejection(example: &Example, seen_texts: &mut HashSet<String>) -> Option<SkipReason> {
    if example.char_count() < MIN_EXAMPLE_CHARS {
        return Some(SkipReason::TooShort);
    }
    let is_first_copy = seen_texts.insert(example.dedup_key());
    match is_first_copy {
        true => None,
        false => Some(SkipReason::Duplicate),
    }
}
}

• Minimum length: 200 characters, instruction and response combined.
• Duplicate key: instruction and response with whitespace runs collapsed to one space.
• Order of reading: readability, chat, book, code. The first copy of a duplicate is kept.

Output: train.jsonl via serde_json::to_string per example; stats.md with per-source counts, the number of examples over 8,192 tokens, and one line per SkipReason.

3.8 Clever-vs-readable set

SFT records are deserialized into typed messages; the user and assistant contents become the example:

#![allow(unused)]
fn main() {
let (Some(instruction), Some(response)) = (record.content_of(Role::User), record.content_of(Role::Assistant))
else {
    skip_reasons.push(SkipReason::EmptyTurn);
    continue;
};
}

DPO records are deserialized into PreferencePair to validate their shape and serialized unchanged to preferences.jsonl.

3.9 Example ids and slop flags

Example::id is the FNV-1a 64-bit hash of dedup_key, printed as 16 hex digits. FNV-1a is used instead of std::collections::hash_map::DefaultHasher for one reason: its output is fixed by definition. The standard hasher may change between Rust releases, which would detach every stored flag.

train.jsonl lines are written from a wrapper that adds the id to the example:

#![allow(unused)]
fn main() {
#[derive(Serialize)]
struct TrainingRecord<'a> {
    id: String,
    #[serde(flatten)]
    example: &'a Example,
}
}

After filtering, slop_flags::separate_flagged splits the examples by whether their id has a flag. Flagged ones go to slop.jsonl as {id, note, spans, instruction, response, source, origin}.

Span categories are a closed enum, deserialized with serde:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Deserialize, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum SlopCategory {
    FakeImportance,
    DramaticSetup,
    EmptyDepthWords,
    FakeBalanceHedging,
    FlatteryFillerOpener,
    WrapUpRepeat,
    RhythmTrick,
    Other,
}
}

A flags file with an unknown category fails the build with a JSON error instead of being counted under a wrong name. slop_flags::span_counts counts spans per category for stats.md.

3.10 Verification

Tests cover:

• hidden blocks dropped
• image question skipped
• flagged conversation skipped
• a heading inside a code block
• an oversized section split
• readability comment removal
• short and duplicate filtering
• path classification
• clever-vs-readable parsing
• preference pair round trip
• flagged example separation with span counts
• flags without spans
• an unknown span category rejected
• a missing flags file