SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding

2024/06/26 by Zhenglin Wang, Jialong Wu, Wang, Zhenglin +7 · 6 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Formal Methods in Verification #Logic, Reasoning, and Knowledge #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2406.18200

openalex publication_date 2024/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) demonstrate remarkable emergent abilities across various tasks, yet fall short of complex reasoning and planning tasks. The tree-search-based reasoning methods address this by surpassing the capabilities of chain-of-thought prompting, encouraging exploration of intermediate steps. However, such methods introduce significant inference latency due to the systematic exploration and evaluation of multiple thought paths. This paper introduces SeeD, a novel and efficient inference framework to optimize runtime speed and GPU memory management concurrently. By employing a scheduled speculative execution, SeeD efficiently handles multiple iterations for the thought generation and the state evaluation, leveraging a rounds-scheduled strategy to manage draft model dispatching. Extensive experimental evaluations on three reasoning datasets demonstrate superior speedup performance of SeeD, providing a viable path for batched inference in training-free speculative decoding.

Cited by

• ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations
• On the Role of Temperature Sampling in Test-Time Scaling
• Chain-in-Tree: Back to Sequential Reasoning in LLM Tree Search
• Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

Related

• Accelerating LLM Inference with Staged Speculative Decoding
• Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
• Accelerating Large Language Model Reasoning via Speculative Search
• ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations
• SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models