Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models

2023/12/09 by Guang Yang, Yang, Guang, Yu Zhou +9 · 14 citations
Computer Science · Materials Science · #FOS: Computer and information sciences #Machine Learning in Materials Science #Software Engineering (cs.SE) #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.05562

openalex publication_date 2023/12/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models (lLMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most lLMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated by COTTON boost various lLMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by gpt-3.5-turbo (175B). Our study also showcases the potential of lLMs in software engineering applications.

Cited by

• Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
• Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale
• Issue-Oriented Agent-Based Framework for Automated Review Comment Generation
• Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
• Large Language Models for Fault Localization: An Empirical Study
• Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models
• Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs
• The Cream Rises to the Top: Efficient Reranking Method for Verilog Code Generation
• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle
• NL in the Middle: Code Translation with LLMs and Intermediate Representations
• Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation
• Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
• MCTS-Refined CoT: High-Quality Fine-Tuning Data for LLM-Based Repository Issue Resolution

Related

• Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
• The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
• Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models
• Structured Chain-of-Thought Prompting for Code Generation
• Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation