Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models

2025/11/07 by Cong-Thanh Do, Do, Cong-Thanh, Rama Doddipatla +2
Computer Science · #Advanced Graph Neural Networks #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.05184

openalex publication_date 2025/11/07 · openalex created_date 2025/11/11 · openalex updated_date 2026/07/28

Abstract

Chain-of-Thought (CoT) prompting is a widely used method to improve the reasoning capability of Large Language Models (LLMs). More recently, CoT has been leveraged in Knowledge Distillation (KD) to transfer reasoning capability from a larger LLM to a smaller one. This paper examines the role of CoT in distilling the reasoning capability from larger LLMs to smaller LLMs using white-box KD, analysing its effectiveness in improving the performance of the distilled models for various natural language reasoning and understanding tasks. We conduct white-box KD experiments using LLMs from the Qwen and Llama2 families, employing CoT data from the CoT-Collection dataset. The distilled models are then evaluated on natural language reasoning and understanding tasks from the BIG-Bench-Hard (BBH) benchmark, which presents complex challenges for smaller LLMs. Experimental results demonstrate the role of CoT in improving white-box KD effectiveness, enabling the distilled models to achieve better average performance in natural language reasoning and understanding tasks from BBH.

Citations

• ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents
• Mentor-KD: Making Small Language Models Better Multi-step Reasoners
• Small Language Models: Survey, Measurements, and Insights
• LLM Pruning and Distillation in Practice: The Minitron Approach
• Keypoint-based Progressive Chain-of-Thought Distillation for LLMs
• TinyLlama: An Open-Source Small Language Model
• MCC-KD: Multi-CoT Consistent Knowledge Distillation
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Deductive Verification of Chain-of-Thought Reasoning
• The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• Specializing Smaller Language Models towards Multi-Step Reasoning
• Large Language Models Are Reasoning Teachers
• Teaching Small Language Models to Reason
• Distilling Reasoning Capabilities into Smaller Language Models
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Large Language Models are Zero-Shot Reasoners
• Logical Reasoning for Task Oriented Dialogue Systems
• Language Models are Few-shot Multilingual Learners
• Language Models are Few-Shot Learners
• Distilling the Knowledge in a Neural Network
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
• Distilling Reasoning Capabilities into Smaller Language Models
• Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
• Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
• Mixed Distillation Helps Smaller Language Model Better Reasoning