Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis

2025/12/10 by Jin, Naizhu, Li, Zhong, Yang, Guang +2
#FOS: Computer and information sciences #Software Engineering (cs.SE)

paper · doi:10.48550/arxiv.2512.09679

Abstract

Large language models (LLMs) achieve strong performance on code generation, but the mechanisms by which Chain-of-Thought (CoT) prompting helps remain unclear. We present a systematic empirical and information-theoretic study of CoT effectiveness in neural code generation, evaluating five paradigms (Zero-Shot, Zero-Shot CoT, Self-Planning, Structured CoT, Reasoning-CoT) across six Python benchmarks, a multilingual benchmark with 12 programming languages, and six models from 7B to 480B parameters, using conditional mutual information I(Y;C|X) as a conceptual lens. Our results show that externally guided CoT consistently outperforms direct generation, with structured methods improving Pass@1 by 5--12% on average while using substantially fewer tokens than reflective reasoning, and that CoT benefits depend on language type systems and model capacity. We further find that reasoning quality is critical: high-quality structured CoT from strong generators yields significantly higher accuracy than lightweight alternatives with the same template, whereas naive Zero-Shot CoT can even degrade performance. These findings provide practical guidance for choosing CoT strategies based on model capacity, language characteristics, and task complexity.

Citations

• Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
• MSCoT: Structured Chain-of-Thought Generation for Multiple Programming Languages
• Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
• A Survey on Large Language Models for Code Generation
• MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation
• Chain of Thoughtlessness? An Analysis of CoT in Planning
• StarCoder 2 and The Stack v2: The Next Generation
• HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
• Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models
• Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation
• Code Llama: Open Foundation Models for Code
• Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
• Structured Chain-of-Thought Prompting for Code Generation
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• How Important are Good Method Names in Neural Code Generation? A Model Robustness Perspective
• Large Language Models are Zero-Shot Reasoners
• InCoder: A Generative Model for Code Infilling and Synthesis
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Related

• Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models
• MSCoT: Structured Chain-of-Thought Generation for Multiple Programming Languages
• Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
• Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning