Supercharge Your LLM Application Evaluations π
Objective metrics, intelligent test generation, and data-driven insights for LLM apps
Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready? We also do production-aligned test set generation.
- π― Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
- π§ͺ Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
- π Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
- π Build feedback loops: Leverage production data to continually improve your LLM applications.
Pypi:
pip install ragas
Alternatively, from source:
pip install git+https://github.com/explodinggradients/ragas
This is 5 main lines:
from ragas import evaluate
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
from langchain_openai.chat_models import ChatOpenAI
from ragas.llms import LangchainLLMWrapper
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
metrics = [LLMContextRecall(), FactualCorrectness(), Faithfulness()]
results = evaluate(dataset=eval_dataset, metrics=metrics, llm=evaluator_llm)
Find the complete RAG Evaluation Quickstart here: https://docs.ragas.io/en/latest/getstarted/rag_evaluation/
π±οΈClick to see preview of RESULTS
user_input | retrieved_contexts | response | reference | context_recall | factual_correctness | faithfulness |
---|---|---|---|---|---|---|
What are the global implications of the USA Supreme Court ruling on abortion? | "- In 2022, the USA Supreme Court ... - The ruling has created a chilling effect ..." | The global implications ... Here are some potential implications: | The global implications ... Additionally, the ruling has had an impact beyond national borders ... | 1 | 0.47 | 0.516129 |
Which companies are the main contributors to GHG emissions ... ? | "- Fossil fuel companies ... - Between 2010 and 2020, human mortality ..." | According to the Carbon Majors database ... Here are the top contributors: | According to the Carbon Majors database ... Additionally, between 2010 and 2020, human mortality ... | 1 | 0.11 | 0.172414 |
Which private companies in the Americas are the largest GHG emitters ... ? | "The private companies responsible ... The largest emitter amongst state-owned companies ..." | According to the Carbon Majors database, the largest private companies ... | The largest private companies in the Americas ... | 1 | 0.26 | 0 |
What if you don't have the data for folks asking questions when they interact with your RAG system?
Ragas can help by generating synthetic test set generation -- where you can seed it with your data and control the difficulty, variety, and complexity.
If you want to get more involved with Ragas, check out our discord server. It's a fun community where we geek out about LLM, Retrieval, Production issues, and more.
+----------------------------------------------------------------------------+
| +----------------------------------------------------------------+ |
| | Developers: Those who built with `ragas`. | |
| | (You have `import ragas` somewhere in your project) | |
| | +----------------------------------------------------+ | |
| | | Contributors: Those who make `ragas` better. | | |
| | | (You make PR to this repo) | | |
| | +----------------------------------------------------+ | |
| +----------------------------------------------------------------+ |
+----------------------------------------------------------------------------+
We welcome contributions from the community! Whether it's bug fixes, feature additions, or documentation improvements, your input is valuable.
- Fork the repository
- Create your feature branch (git checkout -b feature/AmazingFeature)
- Commit your changes (git commit -m 'Add some AmazingFeature')
- Push to the branch (git push origin feature/AmazingFeature)
- Open a Pull Request
At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.
β No personal or company-identifying information
β Open-source data collection code
β Publicly available aggregated data
To opt-out, set the RAGAS_DO_NOT_TRACK
environment variable to true
.