---
name: prompt-eval-kit
description: "Design, run, and score prompt evaluations with variance-aware benchmarks and regression tracking. Bootstraps a golden dataset, defines rubrics, runs model matrices, computes variance-adjusted wins, and writes a report."
license: MIT
---

# Prompt Eval Kit

Prompt Eval Kit turns ad-hoc prompt tweaking into a measurable workflow. It bootstraps a golden dataset, defines scoring rubrics, runs your prompt across a matrix of models and variants, and reports variance-adjusted win rates so you can tell a real improvement from sampling noise. It's built for prompt engineers and ML/QA folks who keep "fixing" a prompt without proof it actually got better.

## When to use it

- You changed a system prompt and need to know if quality moved or you just got lucky on three cherry-picked cases.
- You're comparing two prompt variants, or the same prompt across `claude-opus-4`, `claude-sonnet`, and a cheaper fallback.
- You want a regression gate in CI so a prompt edit can't silently degrade output.
- You're handing a prompt to teammates and want a reproducible scorecard, not vibes.

## How it works

The kit organizes an eval into four artifacts:

- **`dataset.jsonl`** — the golden set. Each line has an `input` and optional `reference`/`expected`. The skill can bootstrap candidates from examples you paste or from logged production cases, which you then trim and label.
- **`rubric.yaml`** — named criteria (e.g. `correctness`, `format`, `refusal`) with weights and a 0–N scale. Scoring is done by an LLM judge against these criteria, or by exact/regex/JSON-schema checks for deterministic fields.
- **`matrix.yaml`** — the cartesian product of prompt variants × models × parameters (temperature, max tokens) to evaluate.
- **`report.md` / `report.json`** — results: per-criterion means, **variance-adjusted wins** (each case run multiple times so the score carries a confidence interval, not a single draw), and a flag when two variants overlap within noise.

Run it, repeat to set a baseline, then re-run after edits to see deltas. Because each cell is sampled several times, a 2-point "win" inside the variance band is reported as inconclusive rather than a victory.

```bash
curl -fsSL --create-dirs -o ~/.claude/skills/prompt-eval-kit/SKILL.md https://claudeskil.com/skills/prompt-eval-kit/SKILL.md

# inside your project
prompt-eval init                 # scaffolds dataset.jsonl, rubric.yaml, matrix.yaml
prompt-eval run --repeats 5      # runs the matrix, 5 samples per case
prompt-eval run --baseline main  # compare current vs a saved baseline
```

## Tips & gotchas

- **Set `--repeats` to at least 5.** A single sample per case makes the variance estimate meaningless and tiny diffs will look real.
- **The judge model is a confound.** Pin one judge model for the whole comparison; don't let it drift between baseline and candidate runs, or the deltas reflect the judge, not your prompt.
- **Keep the golden set small and adversarial.** 30–60 hard, diverse cases beat 500 easy ones — easy cases saturate at full marks and hide regressions.
- **Save baselines per prompt version,** not per day, so regression diffs map to actual edits.

## FAQ

**Does it need a paid judge to score?** LLM-as-judge criteria need a model, but deterministic checks (regex, exact match, JSON schema) run free and offline — mix them in `rubric.yaml`.

**Can I gate CI on it?** Yes. `prompt-eval run --baseline <ref>` exits non-zero when a weighted score regresses beyond the variance band, so it drops into a pipeline step.

**Why did a "better" prompt show no win?** The improvement fell inside the confidence interval. Add cases or raise `--repeats` to tighten the band.