claudegoodies
Skill

eval-harness-first

From wshobson

Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.

Facts

Repository
wshobson/agents
Status
Actively maintained
Last commit
Source file
plugins/llm-finetuning/skills/eval-harness-first/SKILL.md

Source preview

The instructions Claude Code reads when this skill runs.

# Eval Harness First

The Phase 0 gate for the whole plugin:
`finetuning-method-selection` and every downstream
skill assume this harness exists before a training
config gets written. The harness is not a run-end
side artifact — it is the data-curation engine. The
same labeled traces that build the goldens feed
training data, minus an explicit holdout.

**Input:** production/agent traces if they exist, or
a task spec if they don't, plus labelers willing to
grade ≥100 examples.
**Output format:** the `eval/` directory below —
goldens, graders, drift suite, and the base-model
baseline that later phases gate on.

## The Gate

No eval harness, no fine-tune. Skip to a training
config and there is nothing to measure against,
nothing to catch regressions, and no labeled data
to train on. The flywheel:

1. **Collect traces** — production/agent spans, or
   synthetic tasks if none exist yet.
2. **Error analysis** — open coding on ≥100 traces,
   axial coding into 4–8 failure buckets.
3. **One grader per bucket** — deterministic first;
   calibrated LLM-judge only for genuinely
   subjective criteria.
4. **Prioritize** by frequency × severity × value.
5. **The labeled traces feed dataset curation, minus
   an explicit holdout.** Every `eval/goldens.jsonl`
   ID stays excluded from training data by ID.
6. **Train.**
7. **Re-run the same harness** on the checkpoint —
   not a different, lo
View full source on GitHub →

Other skills