Long Code Arena: a Set of Benchmarks for Long-Context Code Models

2024/06/17 by Egor Bogomolov, Bogomolov, Egor, Aleksandra Eliseeva +18 · 15 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Model-Driven Software Engineering Techniques #Power Systems and Technologies #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2406.11612

openalex publication_date 2024/06/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes have increased by orders of magnitude over the last few years. However, there is a shortage of benchmarks for code processing that go beyond a single file of context, while the most popular ones are limited to a single method. With this work, we aim to close this gap by introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions based on popular LLMs to showcase the usage of the dataset and to simplify adoption by other researchers. We publish the benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace Hub for all the datasets, and link to the GitHub repository with baselines: https://huggingface.co/spaces/JetBrains-Research/long-code-arena.

Cited by

• Reformulate, Retrieve, Localize: Agents for Repository-Level Bug Localization
• Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgets
• LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding
• On Pretraining for Project-Level Code Completion
• Diff-XYZ: A Benchmark for Evaluating Diff Understanding
• How can we assess human-agent interactions? Case studies in software agent design
• Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches
• Challenge on Optimization of Context Collection for Code Completion
• LongCodeZip: Compress Long Context for Code Language Models
• Intuition to Evidence: Measuring AI's True Impact on Developer Productivity
• Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs
• A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
• Robot builds a robot's brain: AI generated drone command and control station hosted in the sky
• TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs
• Gotta catch 'em all! Towards File Localisation from Issues at Large

Related

• LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
• LongCodeZip: Compress Long Context for Code Language Models
• LongCoder: A Long-Range Pre-trained Language Model for Code Completion
• YABLoCo: Yet Another Benchmark for Long Context Code Generation
• LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering