RAG Assessment framework for Python providing reference-free evaluation of RAG pipelines using LLM-as-a-judge, measuring context relevancy, context recall, faithfulness, and answer relevancy with automatic test data generation.
Loading more......
RAGAs (Retrieval-Augmented Generation Assessment) is a framework that provides the necessary ingredients to help you evaluate your RAG pipeline on a component level. Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications.
RAGAs started as a framework for "reference-free" evaluation, meaning instead of relying on human-annotated ground truth labels, RAGAs leverages LLMs under the hood to conduct evaluations.
Four Core Metrics:
Together, these make up the RAGAs score.
The framework provides tooling for automatic test data generation, reducing the need for manual dataset creation.
pip install ragas
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_recall,
context_precision,
)
result = evaluate(
dataset=eval_dataset,
metrics=[
context_precision,
context_recall,
faithfulness,
answer_relevancy,
],
)
ragas quickstart rag_eval # Creates RAG evaluation project template
Free and open-source framework (Apache 2.0 license). Costs only for LLM API calls used in evaluation.