Prompt Eval Kit

Prompt Eval Kit turns ad-hoc prompt tweaking into a measurable workflow. It bootstraps a golden dataset, defines scoring rubrics, runs your prompt across a matrix of models and variants, and reports variance-adjusted win rates so you can tell a real improvement from sampling noise. It's built for prompt engineers and ML/QA folks who keep "fixing" a prompt without proof it actually got better.

When to use it

• You changed a system prompt and need to know if quality moved or you just got lucky on three cherry-picked cases.
• You're comparing two prompt variants, or the same prompt across claude-opus-4, claude-sonnet, and a cheaper fallback.
• You want a regression gate in CI so a prompt edit can't silently degrade output.
• You're handing a prompt to teammates and want a reproducible scorecard, not vibes.

How it works

The kit organizes an eval into four artifacts:

• dataset.jsonl — the golden set. Each line has an input and optional reference/expected. The skill can bootstrap candidates from examples you paste or from logged production cases, which you then trim and label.
• rubric.yaml — named criteria (e.g. correctness, format, refusal) with weights and a 0–N scale. Scoring is done by an LLM judge against these criteria, or by exact/regex/JSON-schema checks for deterministic fields.
• matrix.yaml — the cartesian product of prompt variants × models × parameters (temperature, max tokens) to evaluate.
• report.md / report.json — results: per-criterion means, variance-adjusted wins (each case run multiple times so the score carries a confidence interval, not a single draw), and a flag when two variants overlap within noise.

Run it, repeat to set a baseline, then re-run after edits to see deltas. Because each cell is sampled several times, a 2-point "win" inside the variance band is reported as inconclusive rather than a victory.

curl -fsSL --create-dirs -o ~/.claude/skills/prompt-eval-kit/SKILL.md https://claudeskil.com/skills/prompt-eval-kit/SKILL.md

# inside your project
prompt-eval init                 # scaffolds dataset.jsonl, rubric.yaml, matrix.yaml
prompt-eval run --repeats 5      # runs the matrix, 5 samples per case
prompt-eval run --baseline main  # compare current vs a saved baseline

Tips & gotchas

• Set --repeats to at least 5. A single sample per case makes the variance estimate meaningless and tiny diffs will look real.
• The judge model is a confound. Pin one judge model for the whole comparison; don't let it drift between baseline and candidate runs, or the deltas reflect the judge, not your prompt.
• Keep the golden set small and adversarial. 30–60 hard, diverse cases beat 500 easy ones — easy cases saturate at full marks and hide regressions.
• Save baselines per prompt version, not per day, so regression diffs map to actual edits.

FAQ

Does it need a paid judge to score? LLM-as-judge criteria need a model, but deterministic checks (regex, exact match, JSON schema) run free and offline — mix them in rubric.yaml.

Can I gate CI on it? Yes. prompt-eval run --baseline <ref> exits non-zero when a weighted score regresses beyond the variance band, so it drops into a pipeline step.

Why did a "better" prompt show no win? The improvement fell inside the confidence interval. Add cases or raise --repeats to tighten the band.