STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models

2025/07/21 by Cheng-Han Chiang, Xiaofei Wang, Chiang, Cheng-Han +15 · 10 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Intelligent Tutoring Systems and Adaptive Learning #Multimodal Machine Learning Applications #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.15375

openalex publication_date 2025/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses. However, current SLMs lack the ability to perform an internal, unspoken thinking process before responding. In contrast, humans typically engage in complex mental reasoning internally, enabling them to communicate ideas clearly and concisely. Thus, integrating an unspoken thought process into SLMs is highly desirable. While naively generating a complete chain-of-thought (CoT) reasoning before starting to talk can enable thinking for SLMs, this induces additional latency for the speech response, as the CoT reasoning can be arbitrarily long. To solve this issue, we propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks. Since the audio duration of a chunk of spoken response is much longer than the time to generate the tokens in a chunk of spoken response, we use the remaining free time to generate the unspoken reasoning tokens. When a chunk of audio is played to the user, the model continues to generate the next unspoken reasoning chunk, achieving simultaneous thinking and talking. Remarkably, Stitch matches the latency of baselines that cannot generate unspoken CoT by design while outperforming those baselines by 15% on math reasoning datasets; Stitch also performs equally well on non-reasoning datasets as those baseline models. Some animations and demonstrations are on the project page: https://d223302.github.io/STITCH.

Citations

• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model
• MathArena: Evaluating LLMs on Uncontaminated Math Competitions
• LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis
• Kimi-Audio Technical Report
• SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning
• On The Landscape of Spoken Language Models: A Comprehensive Survey
• Qwen2.5-Omni Technical Report
• Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering
• Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models
• Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction
• SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
• GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot
• Scaling Speech-Text Pre-training with Synthetic Interleaved Data
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Thinking LLMs: General Instruction Following with Thought Generation
• Moshi: a speech-text foundation model for real-time dialogue
• The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
• Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
• The Llama 3 Herd of Models
• Qwen2-Audio Technical Report
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Spirit LM: Interleaved Spoken and Written Language Model
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM
• Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
• Can Large Language Models Be an Alternative to Human Evaluations?
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Robust Speech Recognition via Large-Scale Weak Supervision
• Fast Inference from Transformers via Speculative Decoding
• High Fidelity Neural Audio Compression
• Large Language Models are Zero-Shot Reasoners
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• LoRA: Low-Rank Adaptation of Large Language Models
• Are NLP Models really able to Solve Simple Math Word Problems?
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Attention Is All You Need
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• Solving General Arithmetic Word Problems
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning

Cited by

• X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
• Towards Interactive Intelligence for Digital Humans
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• StreamingThinker: Large Language Models Can Think While Reading
• SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models
• Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations
• Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models
• Can Speech LLMs Think while Listening?
• SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
• Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap

Related

• SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
• Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models
• Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speech
• Can Speech LLMs Think while Listening?
• Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models