CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments

2025/10/30 by Lingyue Fu, Fu, Lingyue, Xin Ding +20
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.26852

openalex publication_date 2025/10/30 · openalex created_date 2025/11/05 · openalex updated_date 2026/07/30

Abstract

Current evaluation for Large Language Model (LLM) code agents predominantly focus on generating functional code in single-turn scenarios, which fails to evaluate the agent's capability for continuous code optimization and multi-turn iterative development. To bridge this gap, we introduce CATArena, a framework designed to evaluate the evolutionary capabilities of code agents via iterative tournaments. Agents engage in multi-turn tournaments and continuously refine their code through self-reflection and peer-learning based on comprehensive execution feedback. For evaluation, we propose a dual-metric system to decouple static generation proficiency from evolutionary potential. Extensive experiments reveal that an agent's evolutionary potential is not strictly correlated with its initial proficiency. Our analysis further reveals that current agents struggle to concurrently leverage both peer-learning and self-reflection for effective performance gains. Furthermore, the results validate CATArena's high extensibility and resistance to variance tasks, establishing it as a continuous and reliable standard for assessing the evolutionary capability of LLM code agents.

Citations

• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
• A Survey on Code Generation with LLM-based Agents
• LLM-based Multi-Agent System for Intelligent Refactoring of Haskell Code
• DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
• OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics
• Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
• Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation
• Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks
• Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codes
• TextArena
• SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
• RedCode: Risky Code Execution and Generation Benchmark for Code Agents
• Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
• DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
• Self-Refine: Iterative Refinement with Self-Feedback
• Language Models can Solve Computer Tasks
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence

Related

• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
• LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
• ART: Adaptive Response Tuning Framework -- A Multi-Agent Tournament-Based Approach to LLM Response Optimization
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
• DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments