CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

2025/04/29 by Yin, Wenjing, Sun, Tianze, Yu, Yijiong +19 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Software Engineering (cs.SE)

paper · doi:10.48550/arxiv.2504.20673

Abstract

Large language models (LLMs) play a crucial role in software engineering, excelling in tasks like code generation and maintenance. However, existing benchmarks are often narrow in scope, focusing on a specific task and lack a comprehensive evaluation framework that reflects real-world applications. To address these gaps, we introduce CoCo-Bench (Comprehensive Code Benchmark), designed to evaluate LLMs across four critical dimensions: code understanding, code generation, code modification, and code review. These dimensions capture essential developer needs, ensuring a more systematic and representative evaluation. CoCo-Bench includes multiple programming languages and varying task difficulties, with rigorous manual review to ensure data quality and accuracy. Empirical results show that CoCo-Bench aligns with existing benchmarks while uncovering significant variations in model performance, effectively highlighting strengths and weaknesses. By offering a holistic and objective evaluation, CoCo-Bench provides valuable insights to guide future research and technological advancements in code-oriented LLMs, establishing a reliable benchmark for the field.

Cited by

• TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework

Related

• A Survey on Evaluating Large Language Models in Code Generation Tasks
• Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks
• CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models
• Holistic Evaluation of State-of-the-Art LLMs for Code Generation
• Large Language Models for Code Generation: The Practitioners Perspective