A Survey on Speech Large Language Models for Understanding

2024/10/24 by Jing Peng, Yucheng Wang, Peng, Jing +12 · 38 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2410.18908

openalex publication_date 2024/10/24 · openalex created_date 2024/11/13 · openalex updated_date 2026/08/03

Abstract

Speech understanding is essential for interpreting the diverse forms of information embedded in spoken language, including linguistic, paralinguistic, and non-linguistic cues that are vital for effective human-computer interaction. The rapid advancement of large language models (LLMs) has catalyzed the emergence of Speech Large Language Models (Speech LLMs), which marks a transformative shift toward general-purpose speech understanding systems. To further clarify and systematically delineate task objectives, in this paper, we formally define the concept of speech understanding and introduce a structured taxonomy encompassing its informational, functional, and format dimensions. Within this scope of definition, we present a comprehensive review of current Speech LLMs, analyzing their architectures through a three-stage abstraction: Modality Feature Extraction, Modality Information Fusion, and LLM Inference. In addition, we examine training strategies, discuss representative datasets, and review evaluation methodologies adopted in the field. Based on empirical analyses and experimental evidence, we identify two key challenges currently facing Speech LLMs: instruction sensitivity and degradation in semantic reasoning and propose concrete directions for addressing these issues. Through this systematic and detailed survey, we aim to offer a foundational reference for researchers and practitioners working toward more robust, generalizable, and human-aligned Speech LLMs.

Cited by

• POTSA: A Cross-Lingual Speech Alignment Framework for Low Resource Speech-to-Text Translation
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• Open Source State-Of-the-Art Solution for Romanian Speech Recognition
• TASU: Text-Only Alignment for Speech Understanding
• ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
• UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models
• Topic-aware Large Language Models for Summarizing the Lived Healthcare Experiences Described in Health Stories
• Can large audio language models understand child stuttering speech? speech summarization, and source separation
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
• Index-MSR: A high-efficiency multimodal fusion framework for speech recognition
• Acoustic-based Gender Differentiation in Speech-aware Language Models
• DRES: Benchmarking LLMs for Disfluency Removal
• Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs
• Interactive In-Meeting Speaker Correction with Human Feedback
• From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models
• SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings
• MOSA: Mixtures of Simple Adapters Outperform Monolithic Approaches in LLM-based Multilingual ASR
• LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
• Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speech
• LLMCARE: early detection of cognitive impairment via transformer models enhanced by LLM-generated synthetic data
• The Eloquence team submission for task 1 of MLC-SLM challenge
• Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model
• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
• Can Quantized Audio Language Models Perform Zero-Shot Spoofing Detection?
• OpusLM: A Family of Open Unified Speech Language Models
• Zero-Shot Cognitive Impairment Detection from Speech Using AudioLLM
• AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR
• Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
• Phi-Omni-ST: A multimodal language model for direct speech-to-speech translation
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model
• HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement
• Probing the Robustness Properties of Neural Speech Codecs
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data
• MFA-KWS: Effective Keyword Spotting with Multi-head Frame-asynchronous Decoding
• JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models
• A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
• On The Landscape of Spoken Language Models: A Comprehensive Survey

Related

• Roadmap towards Superhuman Speech Understanding using Large Language Models
• When Large Language Models Meet Speech: A Survey on Integration Approaches
• Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective
• Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond
• Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey