Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Spark the tigress beside a balance with pans labelled A and B

Human evaluation (planned)

This chapter covers

  • The blind comparison procedure
  • The verdict file format
  • Use of the results

Status: design only. No code exists for this chapter.

6.1 Procedure

A prompt list is sent to the base model and the fine-tuned model. Answers are shown as A and B in shuffled order with model names hidden. The reviewer picks A, B or tie. Verdicts are written to verdicts.jsonl and used to select data for the next round.
Figure 6.1 Blind pairwise comparison.
  1. A prompt list is written by hand. It contains prompts per book topic, prompts similar to the readability set, and prompts on topics absent from the data.
  2. Each prompt is sent to both endpoints with identical generation settings.
  3. The two answers are displayed as A and B. Order is randomized per prompt; model names are hidden.
  4. The reviewer selects A, B or tie.

The comparison page follows the review page: localhost only, text rendering only.

6.2 Verdict format

One JSON object per prompt in verdicts.jsonl:

FieldContent
promptthe prompt text
answer_a, answer_bthe two answers as displayed
a_modelbase or fine_tuned
verdicta, b or tie

6.3 Use of the results

  • Win rate: share of non-tie verdicts won by the fine-tuned model.
  • Losses by topic: prompts lost by the fine-tuned model, grouped by topic, select which sources to extend in the next data round.