Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings

2025/12/16 by Liu, Changshu, Ghazanfari, Alireza, Chen, Yang +1
Computer Science · #Automated reasoning #Categorization #Code (set theory) #Construct (python library) #FOS: Computer and information sciences #Generalizability theory #Model-based reasoning #Natural Language Processing Techniques #Pipeline (software) #Python (programming language) #Reasoning system #Software Engineering (cs.SE) #Software Engineering Research #Topic Modeling

paper · doi:10.48550/arxiv.2512.14917

published in Open MIND

openalex publication_date 2025/12/16 · openalex created_date 2025/12/19 · openalex updated_date 2026/07/28

Abstract

Code reasoning tasks are becoming prevalent in large language model (LLM) assessments. Yet, there is a dearth of studies on the impact of real-world complexities on code reasoning, e.g., inter- or intra-procedural dependencies, API calls, deeply nested constructs, and non-primitive complex types. Evaluating LLMs under such a simplistic setting poses a significant threat to assumptions about their generalizability in practice. To enable a more realistic evaluation of code reasoning, we construct a dataset of 1200 reasoning problems from two sources: existing code reasoning benchmarks and popular GitHub Python repositories. Our pipeline leverages static and dynamic program analysis to automatically serialize/deserialize compound, complex, and custom types galore in real-world code, going far beyond only primitive types used in prior studies. A key feature of our dataset is categorizing each reasoning problem as Lower Complexity (LC) or Higher Complexity (HC) via a principled majority-vote mechanism over nine diverse and interpretable code-complexity metrics, yielding two well-separated, semantically meaningful categories of problem difficulty suitable for precise calibration of LLM reasoning ability. This categorization shows that the problems used in existing code-reasoning evaluation mostly belong to the LC category, failing to represent real-world complexity.

Citations

• Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity
• CWM: An Open-Weights LLM for Research on Code Generation with World Models
• Inverse Scaling in Test-Time Compute
• CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
• When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs
• CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
• Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
• L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution
• HoarePrompt: Structural Reasoning About Program Correctness in Natural Language
• RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code
• Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference
• S*: Test Time Scaling for Code Generation
• A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
• CoCoNUT: Structural Code Understanding does not fall out of a tree
• A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
• Top Pass: Improve Code Generation by Pass@k-Maximized Code Ranking
• Agentless: Demystifying LLM-based Software Engineering Agents
• Easy Problems That LLMs Get Wrong
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• Vulnerability Detection with Code Language Models: How Far Are We?
• CYCLE: Learning to Self-Refine the Code Generation
• Reasoning Runtime Behavior of a Program with LLM: How Far Are We?
• CodeMind: Evaluating Large Language Models for Code Reasoning
• Finetuning Large Language Models for Vulnerability Detection
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
• ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
• How is ChatGPT's behavior changing over time?
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Compositional Exemplars for In-context Learning
• A Survey on In-context Learning
• AVATAR: A Parallel Corpus for Java-Python Program Translation
• Evaluating Large Language Models Trained on Code
• Self-Supervised Bug Detection and Repair
• Language Models are Few-Shot Learners
• Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates

Cited by

• Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation

Related

• CodeMind: Evaluating Large Language Models for Code Reasoning
• ProBench: Benchmarking Large Language Models in Competitive Programming
• An Empirical Study of Reasoning Steps in Thinking Code LLMs
• OJBench: A Competition Level Code Benchmark For Large Language Models
• Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models