SWE-Bench+: Enhanced Coding Benchmark for LLMs

2024/10/09 by Reem Aleithan, Huayuan Xue, Haoran Xue +10 · 19 voices · 25 citations
Computer Science · #Advanced Data Storage Technologies

paper · pdf · doi:10.48550/arxiv.2410.06992

Abstract

Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories. Several impressive LLM-based toolkits recently are developed and evaluated on this dataset. However, a systematic evaluation of the quality of SWE-bench remains missing. In this paper, we addressed this gap by presenting an empirical analysis of the SWE-bench dataset. We conducted a manual screening of instances where SWEAgent + GPT-4 successfully resolved issues by comparing the model-generated patches with the actual pull requests. SWE-Agent+GPT-4 was at the top of SWE-bench leaderboard during the time of our study. Our analysis reveals some critical issues with the SWE-bench dataset: 1) 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. We refer to as solution leakage problem. 2) 31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch. When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. We also observed that the same data quality issues also exist in the two variants of SWE-bench, i.e., SWE-bench Lite and SWE-Bench Verified. In addition, over 94% of the issues were created before LLM's knowledge cutoff dates, posing potential data leakage issues.

Cited by

• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes
• RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
• Agentic Software Issue Resolution with Large Language Models: A Survey
• SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories
• Does SWE-Bench-Verified Test Agent Ability or Model Memory?
• A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networks
• CodeR3: A GenAI-Powered Workflow Repair and Revival Ecosystem
• Gistify! Codebase-Level Understanding via Runtime Execution
• Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
• MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
• BloomAPR: A Bloom's Taxonomy-based Framework for Assessing the Capabilities of LLM-Powered APR Solutions
• PerfBench: Can Agents Resolve Real-World Performance Bugs?
• Estimating the Empowerment of Language Model Agents
• SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
• Large language model-empowered next-generation computer-aided engineering
• Agentic Software Engineering: Foundational Pillars and a Research Roadmap
• What You See Is What It Does: A Structural Pattern for Legible Software
• An Efficient and Adaptive Next Edit Suggestion Framework with Zero Human Instructions in IDEs
• Gotta catch 'em all! Towards File Localisation from Issues at Large
• Agentic Hardware Design as Repository-Level Code Evolution
• OmniCode: A Benchmark for Evaluating Software Engineering Agents
• SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

Discussions

• Some critical issues with the SWE-bench dataset [hn, 350 points, 116 comments]
• SWE-Bench+: Enhanced Coding Benchmark for LLMs [hn, 3 points, 0 comments]
• "Some critical issues with the SWE-bench dataset" The analysis of AI coding benchmarks shows big problems. Many reports from top firms are misleading, making the tools seem better than they are. Arti [bsky, 2 points, 0 comments]
• “32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments.” arxiv.org/abs/2410.06992 [bsky, 0 points, 0 comments]
• > SWE-Bench tainted by answer leakage; real pass rates significantly lower The grift continues. arxiv.org/abs/2410.06992 [bsky, 0 points, 0 comments]
• https://bsky.app/profile/hackernews.com.web.brid.gy/post/3liptpezeog52 [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower https://arxiv.org/abs/2410.06992 https://news.ycombinator.com/item?id=43130732 [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower https://arxiv.org/abs/2410.06992 (https://news.ycombinator.com/item?id=43130732) [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower https://arxiv.org/abs/2410.06992 [bsky, 0 points, 0 comments]
• Some critical issues with the SWE-bench dataset https://arxiv.org/abs/2410.06992 [comments] [300 points] [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower https://arxiv.org/abs/2410.06992 (http://news.ycombinator.com/item?id=43130732) [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower https://arxiv.org/abs/2410.06992 (http://news.ycombinator.com/item?id=43130732) [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower view on hacker news [bsky, 0 points, 0 comments]
• Some critical issues with the SWE-bench dataset https://arxiv.org/abs/2410.06992 arxiv.org [bsky, 0 points, 0 comments]
• Some critical issues with the SWE-bench dataset https://arxiv.org/abs/2410.06992 (https://news.ycombinator.com/item?id=43130732) [bsky, 0 points, 0 comments]
• SWE-Bench tainted by answer leakage; real pass rates significantly lower https://arxiv.org/abs/2410.06992 (https://news.ycombinator.com/item?id=43130732) [bsky, 0 points, 0 comments]

Related

• LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
• The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
• SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories