---
name: evals-runner
description: "Build, run, and report on LLM evals. Pairwise comparisons, judges, regression detection. Sets up an evaluation harness for your AI features. Defines test cases, writes LLM-as-judge prompts, runs pairwise battles between model versions, and outputs a regression-friendly report."
license: MIT
---

# Evals Runner

Evals Runner is a harness for measuring whether your LLM features actually work — and whether a prompt tweak, model swap, or temperature change made them better or worse. It treats evaluation as code: test cases live in version control, judges are explicit prompts you can audit, and every run produces a diffable report. Built for prompt engineers and QA folks who are tired of eyeballing a handful of outputs and calling it a day.

## When to use it

- You changed a system prompt and need to know if quality moved before shipping.
- You're deciding between two models (e.g. a cheaper one vs. your current default) and want a head-to-head verdict, not vibes.
- You want a CI gate that fails a PR when a known-good case regresses.
- You're tuning a RAG or extraction pipeline and need per-case pass/fail tracking over time.

## How it works

You define a **dataset** of cases — each case has an input and optional reference/expected output — and one or more **judges**. A judge is an LLM-as-judge prompt that scores a single output (pointwise, e.g. a 1–5 rubric or pass/fail) or compares two outputs (**pairwise battle**, returning A / B / tie). Runs are keyed by a label (a git SHA, model id, or prompt version) so the reporter can diff a new run against a baseline and flag **regressions** — cases that previously passed and now fail, or lost a pairwise matchup.

Pairwise mode alternates which candidate is presented first to cancel position bias, and aggregates results into a win rate with tie handling. The report writes both a human-readable summary and a machine-readable artifact for CI.

## Example

```bash
curl -fsSL --create-dirs -o ~/.claude/skills/evals-runner/SKILL.md https://claudeskil.com/skills/evals-runner/SKILL.md

# cases.jsonl — one case per line
# {"id": "refund-tone", "input": "I want my money back NOW", "reference": "calm, apologetic, offers next step"}

# Run a pairwise battle between two prompt versions
evals run \
  --dataset cases.jsonl \
  --candidate-a "prompts/support_v3.txt" \
  --candidate-b "prompts/support_v4.txt" \
  --judge judges/helpfulness.md \
  --baseline runs/v3-baseline.json \
  --report out/report.md
```

A judge prompt is just Markdown with placeholders for the input and the candidate output(s); keep the rubric and the required output format (e.g. a JSON verdict) at the bottom so the model can't drift.

## Tips & gotchas

- **Pin the judge model and temperature** (temperature 0) separately from the model under test. If the judge moves, your scores aren't comparable across runs.
- **Watch position bias** in pairwise: if A wins overwhelmingly only when shown first, your judge prompt is leaking order. The alternating-order aggregate is the number to trust.
- **Start with 20–40 hard cases**, not 500 easy ones. Adversarial and edge cases catch regressions; happy-path cases mostly waste tokens.
- **Commit the dataset and judges.** A judge prompt edited mid-experiment silently invalidates your baseline comparison.

## FAQ

**Does it call models for me or just orchestrate?** It orchestrates: you point it at candidate prompts/outputs and a judge, and it handles batching, ordering, scoring, and diffing.

**Can I gate CI on it?** Yes — the run exits non-zero when new regressions appear against the baseline, and the JSON report lists exactly which case ids broke.

**Can I judge without a reference answer?** Yes. Pointwise rubric judges and pairwise battles both work reference-free; references just give the judge a stronger anchor when you have them.