CRQBench: A Benchmark of Code Reasoning Questions

2024/08/15 by Elizabeth Dinella, Satish Chandra, Dinella, Elizabeth +3 · 3 citations
Business, Management and Accounting · Computer Science · #Business Process Modeling and Analysis #FOS: Computer and information sciences #Software Engineering (cs.SE) #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2408.08453

openalex publication_date 2024/08/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models have demonstrated exceptional proficiency on coding tasks, but it is challenging to precisely evaluate their code reasoning ability. Existing benchmarks are insufficient as they are unrealistic and conflate semantic reasoning ability with performance on software engineering tasks. We introduce CRQBench, a benchmark of 100 C++ code reasoning questions and answers derived from contextualized code review comments. To curate CRQBench, we use an LLM assistant alongside human inspection, reducing manual effort. We conduct an evaluation of GPT-4 on CRQBench and find that it produces correct responses grounded in the given context for 65 of the 100 questions.

Cited by

• Towards Verified Code Reasoning by LLMs
• Evaluating LLM Agents on Automated Software Analysis Tasks
• CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks

Related

• OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
• CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation
• ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests
• ProBench: Benchmarking Large Language Models in Competitive Programming
• Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings