Ragas RAG Evaluation

Quantify whether your retrieval-augmented generation pipeline actually works. This skill wires up Ragas to score answers and retrieved contexts with LLM-graded metrics, and to synthesize golden test sets from your own documents — so you can catch regressions before they ship instead of eyeballing sample outputs.

When to use it

• You changed your chunking strategy, embedding model, reranker, or prompt and need a number that says it improved (or didn't).
• You have documents but no labeled QA pairs, and writing them by hand is impractical.
• A RAG answer looks plausible but you suspect it's hallucinating beyond the retrieved context.
• You want CI to fail when faithfulness or context recall drops below a threshold.

How it works

Ragas operates on an EvaluationDataset built from SingleTurnSample records, each carrying user_input, retrieved_contexts, response, and optionally reference. You pass a list of metrics to evaluate(), which calls a judge LLM (wrapped via LangchainLLMWrapper) and returns per-sample scores plus aggregates.

Core metrics in the panel:

• Faithfulness — fraction of answer claims supported by the retrieved context (hallucination detector; needs no reference).
• ResponseRelevancy — how well the answer addresses the question.
• LLMContextPrecisionWithReference — are the relevant chunks ranked high?
• ContextRecall — did retrieval surface everything the reference answer needs?
• NoiseSensitivity — how much irrelevant context corrupts the answer.

For test sets, TestsetGenerator builds a knowledge graph from your Document list and applies query synthesizers (SingleHopSpecificQuerySynthesizer, MultiHopAbstractQuerySynthesizer, etc.) to generate varied questions with ground-truth references.

Usage

curl -fsSL --create-dirs -o ~/.claude/skills/ragas-rag-evaluation/SKILL.md https://claudeskil.com/skills/ragas-rag-evaluation/SKILL.md

from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    Faithfulness, ResponseRelevancy,
    LLMContextPrecisionWithReference, ContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

dataset = EvaluationDataset.from_list(samples)  # list of dicts
judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), ResponseRelevancy(),
             LLMContextPrecisionWithReference(), ContextRecall()],
    llm=judge,
)
print(result)                 # aggregate scores
result.to_pandas().head()     # per-sample breakdown

Tips & gotchas

• Faithfulness and ResponseRelevancy run without a reference, but precision/recall metrics require it — split your metric list by what data you have.
• ResponseRelevancy needs embeddings (it generates questions from the answer and compares), so pass an embeddings model, not just an LLM.
• Wrap judge calls with RunConfig(max_workers=..., timeout=...) to control concurrency and avoid rate-limit failures on large datasets.
• Scores are stochastic — pin a low-temperature judge and average across runs when comparing pipeline versions.

FAQ

Does it lock me into OpenAI? No. Any LangChain-compatible chat model works through LangchainLLMWrapper (Anthropic, local vLLM, Azure). Set both llm and embeddings.

Can I run it offline in CI? Yes — generate the test set once, persist it, then evaluate against a static dataset. Only the judge LLM needs network access.

Why is faithfulness low but answers look correct? The judge counts claims not entailed by retrieved_contexts. Correct-but-unsupported facts (from the model's parametric memory) score as unfaithful by design.