2025/11/18 by Sushant Mehta, Mehta, Sushant
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Software System Performance and Reliability
paper · pdf · doi:10.48550/arxiv.2511.14136
openalex publication_date 2025/11/18 · openalex created_date 2025/11/20 · openalex updated_date 2026/07/28
Current agentic AI benchmarks predominantly evaluate task completion accuracy, while overlooking critical enterprise requirements such as cost-efficiency, reliability, and operational stability. Through systematic analysis of 12 main benchmarks and empirical evaluation of state-of-the-art agents, we identify three fundamental limitations: (1) absence of cost-controlled evaluation leading to 50x cost variations for similar precision, (2) inadequate reliability assessment where agent performance drops from 60% (single run) to 25% (8-run consistency), and (3) missing multidimensional metrics for security, latency, and policy compliance. We propose CLEAR (Cost, Latency, Efficacy, Assurance, Reliability), a holistic evaluation framework specifically designed for enterprise deployment. Evaluation of six leading agents on 300 enterprise tasks demonstrates that optimizing for accuracy alone yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance. Expert evaluation (N=15) confirms that CLEAR better predicts production success (correlation ρ=0.83) compared to accuracy-only evaluation (ρ=0.41).