---
name: ragas-rag-evaluation
description: "Evaluate RAG pipelines with Ragas — faithfulness, answer relevancy, context precision/recall, and test set generation. Production RAG evaluation with Ragas: build golden datasets via TestsetGenerator with knowledge-graph synthesizers, run evaluate() over EvaluationDataset, and compute the core metric panel — Faithfulness, ResponseRelevancy, LLMContextPrecisionWithReference, ContextRecall, plus NoiseSensitivity and FactualCorrectness. Wire LangchainLLMWrapper/embeddings, diagnose retrieval vs generation failures from per-sample scores, and gate CI with threshold checks."
license: MIT
---

# Ragas RAG Evaluation

Quantify whether your retrieval-augmented generation pipeline actually works. This skill wires up [Ragas](https://docs.ragas.io) to score answers and retrieved contexts with LLM-graded metrics, and to synthesize golden test sets from your own documents — so you can catch regressions before they ship instead of eyeballing sample outputs.

## When to use it

- You changed your chunking strategy, embedding model, reranker, or prompt and need a number that says it improved (or didn't).
- You have documents but no labeled QA pairs, and writing them by hand is impractical.
- A RAG answer looks plausible but you suspect it's hallucinating beyond the retrieved context.
- You want CI to fail when faithfulness or context recall drops below a threshold.

## How it works

Ragas operates on an `EvaluationDataset` built from `SingleTurnSample` records, each carrying `user_input`, `retrieved_contexts`, `response`, and optionally `reference`. You pass a list of metrics to `evaluate()`, which calls a judge LLM (wrapped via `LangchainLLMWrapper`) and returns per-sample scores plus aggregates.

Core metrics in the panel:
- **Faithfulness** — fraction of answer claims supported by the retrieved context (hallucination detector; needs no reference).
- **ResponseRelevancy** — how well the answer addresses the question.
- **LLMContextPrecisionWithReference** — are the relevant chunks ranked high?
- **ContextRecall** — did retrieval surface everything the reference answer needs?
- **NoiseSensitivity** — how much irrelevant context corrupts the answer.

For test sets, `TestsetGenerator` builds a knowledge graph from your `Document` list and applies query synthesizers (`SingleHopSpecificQuerySynthesizer`, `MultiHopAbstractQuerySynthesizer`, etc.) to generate varied questions with ground-truth references.

## Usage

```python
from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    Faithfulness, ResponseRelevancy,
    LLMContextPrecisionWithReference, ContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

dataset = EvaluationDataset.from_list(samples)  # list of dicts
judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), ResponseRelevancy(),
             LLMContextPrecisionWithReference(), ContextRecall()],
    llm=judge,
)
print(result)                 # aggregate scores
result.to_pandas().head()     # per-sample breakdown
```

## Tips & gotchas

- `Faithfulness` and `ResponseRelevancy` run without a `reference`, but precision/recall metrics require it — split your metric list by what data you have.
- `ResponseRelevancy` needs embeddings (it generates questions from the answer and compares), so pass an embeddings model, not just an LLM.
- Wrap judge calls with `RunConfig(max_workers=..., timeout=...)` to control concurrency and avoid rate-limit failures on large datasets.
- Scores are stochastic — pin a low-temperature judge and average across runs when comparing pipeline versions.

## FAQ

**Does it lock me into OpenAI?** No. Any LangChain-compatible chat model works through `LangchainLLMWrapper` (Anthropic, local vLLM, Azure). Set both `llm` and `embeddings`.

**Can I run it offline in CI?** Yes — generate the test set once, persist it, then evaluate against a static dataset. Only the judge LLM needs network access.

**Why is faithfulness low but answers look correct?** The judge counts claims not entailed by `retrieved_contexts`. Correct-but-unsupported facts (from the model's parametric memory) score as unfaithful by design.