MCC-KD: Multi-CoT Consistent Knowledge Distillation

2023/10/23 by Hongzhan Chen, Siyue Wu, Chen, Hongzhan +9 · 5 citations
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.14747

openalex publication_date 2023/10/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have showcased remarkable capabilities in complex reasoning through chain of thought (CoT) prompting. Recently, there has been a growing interest in transferring these reasoning abilities from LLMs to smaller models. However, achieving both the diversity and consistency in rationales presents a challenge. In this paper, we focus on enhancing these two aspects and propose Multi-CoT Consistent Knowledge Distillation (MCC-KD) to efficiently distill the reasoning capabilities. In MCC-KD, we generate multiple rationales for each question and enforce consistency among the corresponding predictions by minimizing the bidirectional KL-divergence between the answer distributions. We investigate the effectiveness of MCC-KD with different model architectures (LLaMA/FlanT5) and various model scales (3B/7B/11B/13B) on both mathematical reasoning and commonsense reasoning benchmarks. The empirical results not only confirm MCC-KD's superior performance on in-distribution datasets but also highlight its robust generalization ability on out-of-distribution datasets.

Cited by

• Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
• Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales
• Dual-Head Reasoning Distillation: Improving Classifier Accuracy with Train-Time-Only Reasoning
• Learning from Diverse Reasoning Paths with Routing and Collaboration
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation

Related

• Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
• The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
• Distilling Reasoning Capabilities into Smaller Language Models
• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
• Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering