THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

2025/05/16 by Rutu Mulkar, Patel, Udita, Jay Roberts +13
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.11626

openalex publication_date 2025/05/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications.

Citations

• On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
• Lost in the Middle: How Language Models Use Long Contexts
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• BERTScore: Evaluating Text Generation with BERT

Related

• SAGE: A Framework of Precise Retrieval for RAG
• Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
• AccurateRAG: A Framework for Building Accurate Retrieval-Augmented Question-Answering Applications
• Evaluation of RAG Metrics for Question Answering in the Telecom Domain
• MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems