Specializing Smaller Language Models towards Multi-Step Reasoning

2023/01/30 by Yao Fu, Hao Peng, Fu, Yao +7 · 54 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2301.12726

openalex publication_date 2023/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The surprising ability of Large Language Models (LLMs) to perform well on complex reasoning with only few-shot chain-of-thought prompts is believed to emerge only in very large-scale models (100+ billion parameters). We show that such abilities can, in fact, be distilled down from GPT-3.5 (≥ 175B) to T5 variants (≤ 11B). We propose model specialization, to specialize the model's ability towards a target task. The hypothesis is that large models (commonly viewed as larger than 100B) have strong modeling power, but are spread on a large spectrum of tasks. Small models (commonly viewed as smaller than 10B) have limited model capacity, but if we concentrate their capacity on a specific target task, the model can achieve a decent improved performance. We use multi-step math reasoning as our testbed because it is a very typical emergent ability. We show two important aspects of model abilities: (1). there exists a very complex balance/ tradeoff between language models' multi-dimensional abilities; (2). by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability. We further give comprehensive discussions about important design choices for better generalization, including the tuning data format, the start model checkpoint, and a new model selection method. We hope our practice and discoveries can serve as an important attempt towards specialized smaller models in the new research paradigm set by LLMs.

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Understanding Generalization in Role-Playing Models via Information Theory
• Knowledge Distillation with Structured Chain-of-Thought for Text-to-SQL
• CDLM: Consistency Diffusion Language Models For Faster Sampling
• Fast Reasoning Segmentation for Images and Videos
• Knowledge-Augmented Long-CoT Generation for Complex Biomolecular Reasoning
• Self-Correction Distillation for Structured Data Question Answering
• Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
• A Controlled Candidate-Set Benchmark for Offline Satellite-Security Plan Decomposition
• Investigating the Impact of Rationales for LLMs on Natural Language Understanding
• Big Reasoning with Small Models: Instruction Retrieval at Inference Time
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• Self-Verifying Reflection Helps Transformers with CoT Reasoning
• Harnessing Consistency for Robust Test-Time LLM Ensemble
• ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
• Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales
• MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstraction
• Exploring LLM-based Frameworks for Fault Diagnosis
• It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL
• Transformer and statistical models for LCSH assignment: a comparative study in digital libraries
• MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning
• Probabilistic Token Alignment for Large Language Model Fusion
• DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation
• DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation
• You Don't Need Pre-built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures
• Modelling and Classifying the Components of a Literature Review
• Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)
• Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws
• AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
• LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
• EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation
• Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability
• Are Reasoning Models More Prone to Hallucination?
• Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection
• Does Rationale Quality Matter? Enhancing Mental Disorder Detection via Selective Reasoning Distillation
• The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
• Efficient Long CoT Reasoning in Small Language Models
• Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning
• Distilling LLM Agent into Small Models with Retrieval and Code Tools
• Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation
• LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead
• TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysis
• Learnware of Language Models: Specialized Small Language Models Can Do Big
• SmallPlan: Leverage Small Language Models for Sequential Path Planning with Simulation-Powered, LLM-Guided Distillation
• Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding
• Rethinking Reflection in Pre-Training
• Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability
• Synergistic Weak-Strong Collaboration by Aligning Preferences
• Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
• A Dual-Space Framework for General Knowledge Distillation of Large Language Models
• A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
• Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning

Related

• Towards Reasoning Ability of Small Language Models
• Stacking Small Language Models for Generalizability
• Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models
• Enhancing the Reasoning Capabilities of Small Language Models via\n Solution Guidance Fine-Tuning
• Reasoning Capabilities and Invariability of Large Language Models