Evaluate RAG Quality with RAGAS Metrics

RAG pipelines are hard to evaluate — how do you know if the retrieved context was relevant or if the answer is grounded? RAGAS (Retrieval-Augmented Generation Assessment) provides automated metrics using an LLM-as-judge approach. Below: evaluating a RAG pipeline with the four core RAGAS metrics: faithfulness, answer relevancy, context precision, and context recall.

When to Use This

• Evaluating RAG pipeline quality before deploying to production
• Comparing different chunking strategies, embedding models, or LLMs
• Regression testing when changing RAG components
• Establishing baseline metrics for continuous improvement

Prerequisites

• Python 3.10+
• ragas package (pip install ragas)
• langchain-openai for the evaluator LLM
• An OpenAI API key

Solution

1. Install Dependencies

pip install ragas langchain-openai datasets

2. Prepare Evaluation Dataset

from datasets import Dataset

eval_data = {
    "question": [
        "What is the Redis cache-aside pattern?",
        "How does Docker Compose define multi-container apps?",
        "What are PostgreSQL ACID compliance features?",
    ],
    "answer": [
        "The cache-aside pattern checks the cache first, and if data is missing, loads it from the database and sets it in the cache with a TTL.",
        "Docker Compose uses a YAML file to define multi-container applications, specifying services, networks, and volumes.",
        "PostgreSQL provides ACID compliance through atomic transactions, consistent constraints, isolation levels, and durable write-ahead logging.",
    ],
    "contexts": [
        [
            "Redis cache-aside: application checks cache first. On cache miss, loads from database and sets cache entry with TTL.",
        ],
        [
            "Docker Compose is a tool for defining multi-container Docker applications in a YAML file with services, networks, and volumes.",
        ],
        [
            "PostgreSQL ensures ACID compliance: atomicity via transactions, consistency via constraints, isolation via MVCC, durability via WAL.",
        ],
    ],
    "ground_truth": [
        "Cache-aside checks cache first, loads from DB on miss, sets cache with TTL.",
        "Docker Compose defines multi-container apps in YAML with services, networks, and volumes.",
        "PostgreSQL ACID: atomicity (transactions), consistency (constraints), isolation (MVCC), durability (WAL).",
    ],
}

eval_dataset = Dataset.from_dict(eval_data)

3. Run RAGAS Evaluation

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# LLM and embeddings for evaluation
eval_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
eval_embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Run evaluation with all four core metrics
results = evaluate(
    eval_dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ],
    llm=eval_llm,
    embeddings=eval_embeddings,
)

# Convert to pandas DataFrame for analysis
df = results.to_pandas()
print(df[["question", "faithfulness", "answer_relevancy", "context_precision", "context_recall"]])

4. Interpret the Metrics

def print_eval_report(results) -> None:
    """Print a formatted evaluation report."""
    df = results.to_pandas()

    print("=" * 60)
    print("RAGAS Evaluation Report")
    print("=" * 60)

    metrics = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]

    for metric in metrics:
        scores = df[metric].tolist()
        avg = sum(scores) / len(scores)
        print(f"\n{metric.replace('_', ' ').title()}:")
        print(f"  Average: {avg:.3f}")
        print(f"  Min:     {min(scores):.3f}")
        print(f"  Max:     {max(scores):.3f}")
        for i, score in enumerate(scores):
            print(f"  Q{i+1}: {score:.3f}")

    print("\n" + "=" * 60)
    aggregate_score = sum(df[m].mean() for m in metrics) / len(metrics)
    print(f"Aggregate Score: {aggregate_score:.3f}")

print_eval_report(results)

5. Evaluate Your Own RAG Pipeline

from typing import List

def evaluate_rag_pipeline(
    rag_pipeline,
    test_cases: List[dict],
) -> "Dataset":
    """Evaluate a RAG pipeline against test cases.

    Args:
        rag_pipeline: Object with .ask(query) -> {answer, sources} method.
        test_cases: List of {question, ground_truth} dicts.

    Returns:
        RAGAS evaluation results.
    """
    eval_rows = []

    for case in test_cases:
        # Run the RAG pipeline
        result = rag_pipeline.ask(case["question"])

        eval_rows.append({
            "question": case["question"],
            "answer": result["answer"],
            "contexts": [s["text"] for s in result["sources"]],
            "ground_truth": case["ground_truth"],
        })

    dataset = Dataset.from_list(eval_rows)

    return evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
        llm=eval_llm,
        embeddings=eval_embeddings,
    )

# Usage
test_cases = [
    {
        "question": "What data structures does Redis support?",
        "ground_truth": "Redis supports strings, hashes, lists, sets, sorted sets, streams, and more.",
    },
    {
        "question": "How does PostgreSQL handle isolation?",
        "ground_truth": "PostgreSQL handles isolation via MVCC (Multi-Version Concurrency Control).",
    },
]

results = evaluate_rag_pipeline(my_rag_pipeline, test_cases)
print_eval_report(results)

How It Works

1. Faithfulness — measures whether the answer is grounded in the retrieved context. If the answer contains claims not supported by the context, faithfulness drops. High faithfulness = low hallucination.
2. Answer Relevancy — measures how relevant the answer is to the question. The LLM generates potential questions from the answer and compares them to the original question using embeddings.
3. Context Precision — measures whether the retrieved context contains information needed to answer the question. High precision means the retrieval found the right documents.
4. Context Recall — measures whether all the information needed to answer the question was retrieved. High recall means no relevant information was missed.
5. LLM-as-judge — RAGAS uses an LLM to evaluate each metric, making it automated but dependent on the evaluator LLM’s quality. Use gpt-4o for best results.

Variants

Custom Metrics

from ragas.metrics import Metric

class ConcisenessMetric(Metric):
    """Custom metric: measure answer conciseness."""

    name = "conciseness"

    def _as_dict(self):
        return {"name": self.name}

    def score(self, row):
        answer = row["answer"]
        words = len(answer.split())
        # Score: 1.0 for 20-50 words, decreasing for longer answers
        if 20 <= words <= 50:
            return 1.0
        elif words < 20:
            return words / 20
        else:
            return max(0.0, 50 / words)

Batch Evaluation with CSV

import pandas as pd

def evaluate_from_csv(csv_path: str) -> pd.DataFrame:
    """Load test cases from CSV and evaluate."""
    df = pd.read_csv(csv_path)

    # CSV columns: question, answer, contexts (semicolon-separated), ground_truth
    eval_data = {
        "question": df["question"].tolist(),
        "answer": df["answer"].tolist(),
        "contexts": [c.split(";") for c in df["contexts"]],
        "ground_truth": df["ground_truth"].tolist(),
    }

    dataset = Dataset.from_dict(eval_data)
    results = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
        llm=eval_llm,
        embeddings=eval_embeddings,
    )
    return results.to_pandas()

Compare Two RAG Configurations

def compare_configs(
    pipeline_a,
    pipeline_b,
    test_cases: list[dict],
) -> pd.DataFrame:
    """Compare two RAG configurations side by side."""
    results_a = evaluate_rag_pipeline(pipeline_a, test_cases).to_pandas()
    results_b = evaluate_rag_pipeline(pipeline_b, test_cases).to_pandas()

    comparison = pd.DataFrame({
        "question": results_a["question"],
        "faithfulness_a": results_a["faithfulness"],
        "faithfulness_b": results_b["faithfulness"],
        "relevancy_a": results_a["answer_relevancy"],
        "relevancy_b": results_b["answer_relevancy"],
        "precision_a": results_a["context_precision"],
        "precision_b": results_b["context_precision"],
        "recall_a": results_a["context_recall"],
        "recall_b": results_b["context_recall"],
    })

    comparison["faithfulness_diff"] = comparison["faithfulness_b"] - comparison["faithfulness_a"]
    comparison["relevancy_diff"] = comparison["relevancy_b"] - comparison["relevancy_a"]

    return comparison

Best Practices

• For a deeper guide, see Complete Guide to LLM Evaluation.

• Use gpt-4o as evaluator — cheaper models produce inconsistent judgments

• Create diverse test cases — cover different question types (factual, comparative, procedural)

• Include ground truth from domain experts — automated ground truth generation is less reliable

• Run evaluation after every RAG change — chunking strategy, embedding model, or LLM changes affect all metrics

Common Mistakes

• Evaluating without ground truth — context recall requires ground truth answers; without it, you only get 3 of 4 metrics
• Using the same LLM for generation and evaluation — can bias results; use a different model or at least different temperature
• Too few test cases — 3-5 cases give noisy results; aim for 20+ for reliable averages
• Ignoring faithfulness — high answer relevancy with low faithfulness means confident hallucination