CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance

2025/07/14 by Myeongsoo Kim, Kim, Myeongsoo, Shweta Garg +7 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Software Engineering (cs.SE) #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.10646

openalex publication_date 2025/07/14 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Programming assistants powered by large language models have improved dramatically, yet existing benchmarks still evaluate them in narrow code-generation settings. Recent efforts such as InfiBench and StackEval rely on Stack Overflow questions and remain limited to single-turn interactions, manually curated data, and isolated snippets rather than full project environments. We introduce CodeAssistBench (CAB), the first benchmark for evaluating multi-turn, project-grounded programming assistance at scale. CAB automatically constructs datasets from GitHub issues tagged as questions, using an LLM-driven pipeline that filters noise, extracts runnable contexts, builds executable containers, and verifies environment correctness. This enables continuous, automated expansion across diverse repositories without manual intervention. Using CAB, we create a testbed of 3,286 real-world issues across 214 repositories, spanning seven languages. Evaluating state-of-the-art models reveals a substantial gap: while models achieve 70-83% accuracy on Stack Overflow-style questions, they solve only 7.22-16.49% of CAB issues from post-training-cutoff repositories. These results highlight a fundamental challenge: current LLMs struggle to provide assistance in realistic, project-specific contexts despite strong performance on traditional Q&A benchmarks. CAB provides a scalable, reproducible framework for advancing research in multi-turn, codebase-grounded programming agents. The benchmark and pipeline are fully automated and publicly available at https://github.com/amazon-science/CodeAssistBench/.

Citations

• SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
• ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments
• StackEval: Benchmarking LLMs in Coding Assistance
• A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
• LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
• InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
• Leveraging Large Language Models to Improve REST API Testing
• BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• The Measurement of Observer Agreement for Categorical Data

Cited by

• CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMs
• CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents

Related

• CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments
• StackEval: Benchmarking LLMs in Coding Assistance
• PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Code
• InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
• CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs