Evals Runner

Evals Runner is a harness for measuring whether your LLM features actually work — and whether a prompt tweak, model swap, or temperature change made them better or worse. It treats evaluation as code: test cases live in version control, judges are explicit prompts you can audit, and every run produces a diffable report. Built for prompt engineers and QA folks who are tired of eyeballing a handful of outputs and calling it a day.

When to use it

• You changed a system prompt and need to know if quality moved before shipping.
• You're deciding between two models (e.g. a cheaper one vs. your current default) and want a head-to-head verdict, not vibes.
• You want a CI gate that fails a PR when a known-good case regresses.
• You're tuning a RAG or extraction pipeline and need per-case pass/fail tracking over time.

How it works

You define a dataset of cases — each case has an input and optional reference/expected output — and one or more judges. A judge is an LLM-as-judge prompt that scores a single output (pointwise, e.g. a 1–5 rubric or pass/fail) or compares two outputs (pairwise battle, returning A / B / tie). Runs are keyed by a label (a git SHA, model id, or prompt version) so the reporter can diff a new run against a baseline and flag regressions — cases that previously passed and now fail, or lost a pairwise matchup.

Pairwise mode alternates which candidate is presented first to cancel position bias, and aggregates results into a win rate with tie handling. The report writes both a human-readable summary and a machine-readable artifact for CI.

Example

curl -fsSL --create-dirs -o ~/.claude/skills/evals-runner/SKILL.md https://claudeskil.com/skills/evals-runner/SKILL.md

# cases.jsonl — one case per line
# {"id": "refund-tone", "input": "I want my money back NOW", "reference": "calm, apologetic, offers next step"}

# Run a pairwise battle between two prompt versions
evals run \
  --dataset cases.jsonl \
  --candidate-a "prompts/support_v3.txt" \
  --candidate-b "prompts/support_v4.txt" \
  --judge judges/helpfulness.md \
  --baseline runs/v3-baseline.json \
  --report out/report.md

A judge prompt is just Markdown with placeholders for the input and the candidate output(s); keep the rubric and the required output format (e.g. a JSON verdict) at the bottom so the model can't drift.

Tips & gotchas

• Pin the judge model and temperature (temperature 0) separately from the model under test. If the judge moves, your scores aren't comparable across runs.
• Watch position bias in pairwise: if A wins overwhelmingly only when shown first, your judge prompt is leaking order. The alternating-order aggregate is the number to trust.
• Start with 20–40 hard cases, not 500 easy ones. Adversarial and edge cases catch regressions; happy-path cases mostly waste tokens.
• Commit the dataset and judges. A judge prompt edited mid-experiment silently invalidates your baseline comparison.

FAQ

Does it call models for me or just orchestrate? It orchestrates: you point it at candidate prompts/outputs and a judge, and it handles batching, ordering, scoring, and diffing.

Can I gate CI on it? Yes — the run exits non-zero when new regressions appear against the baseline, and the JSON report lists exactly which case ids broke.

Can I judge without a reference answer? Yes. Pointwise rubric judges and pairwise battles both work reference-free; references just give the judge a stronger anchor when you have them.