2025/07/17 by Zhiwei Liu, Jielin Qiu, Liu, Zhiwei +21 · 15 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Fuzzy Logic and Control Systems #Time Series Analysis and Forecasting
paper · pdf · doi:10.48550/arxiv.2507.12806
openalex publication_date 2025/07/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28
The rapid rise of Large Language Models (LLMs)-based intelligent agents underscores the need for robust, scalable evaluation frameworks. Existing methods rely on static benchmarks and labor-intensive data collection, limiting practical assessment. We introduce MCPEval, an open-source Model Context Protocol (MCP)-based framework that automates end-to-end task generation and deep evaluation of LLM agents across diverse domains. MCPEval standardizes metrics, seamlessly integrates with native agent tools, and eliminates manual effort in building evaluation pipelines. Empirical results across five real-world domains show its effectiveness in revealing nuanced, domain-specific performance. We publicly release MCPEval https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and standardized LLM agent evaluation.