RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems

2024/12/16 by Ioannis Papadimitriou, Papadimitriou, Ioannis, Ilias Gialampoukidis +5 · 2 citations
Computer Science · #AI-based Problem Solving and Planning #Computer science

paper · pdf · doi:10.48550/arxiv.2412.12322

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/12/16 · openalex created_date 2024/12/19 · openalex updated_date 2026/07/28

Abstract

We present RAG Playground, an open-source framework for systematic evaluation of Retrieval-Augmented Generation (RAG) systems. The framework implements and compares three retrieval approaches: naive vector search, reranking, and hybrid vector-keyword search, combined with ReAct agents using different prompting strategies. We introduce a comprehensive evaluation framework with novel metrics and provide empirical results comparing different language models (Llama 3.1 and Qwen 2.5) across various retrieval configurations. Our experiments demonstrate significant performance improvements through hybrid search methods and structured self-evaluation prompting, achieving up to 72.7% pass rate on our multi-metric evaluation framework. The results also highlight the importance of prompt engineering in RAG systems, with our custom-prompted agents showing consistent improvements in retrieval accuracy and response quality.

Cited by

• Human-Centered Evaluation of RAG outputs: a framework and questionnaire for human-AI collaboration
• Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

Related

• RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
• RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
• Design and Evaluation of a Retrieval-Augmented Generation Architecture for OWASP Security Data
• RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines
• RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation