DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data

2024/09/30 by Ke-Han Lu, Lu, Ke-Han, Zhehuai Chen +13 · 29 citations
Computer Science · #Computer science #Language model #Linguistics #Natural language processing #Speech Recognition and Synthesis #Speech and dialogue systems #Speech recognition #Speech synthesis

paper · pdf · doi:10.48550/arxiv.2409.20007

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/09/30 · openalex created_date 2024/10/28 · openalex updated_date 2026/07/28

Abstract

Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) by incorporating pre-trained speech models. However, these SLMs often undergo extensive speech instruction-tuning to bridge the gap between speech and text modalities. This requires significant annotation efforts and risks catastrophic forgetting of the original language capabilities. In this work, we present a simple yet effective automatic process for creating speech-text pair data that carefully injects speech paralinguistic understanding abilities into SLMs while preserving the inherent language capabilities of the text-based LLM. Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data, achieving impressive performance on Dynamic-SUPERB and AIR-Bench-Chat benchmarks. Furthermore, our model exhibits the ability to follow complex instructions derived from LLMs, such as specific output formatting and chain-of-thought reasoning. Our approach not only enhances the versatility and effectiveness of SLMs but also reduces reliance on extensive annotated datasets, paving the way for more efficient and capable speech understanding systems.

Cited by

• Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs
• Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
• ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
• MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages
• Evaluating Emotion Recognition in Spoken Language Models on Emotionally Incongruent Speech
• SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models
• Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations
• AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning
• TAU: A Benchmark for Cultural Sound Understanding Beyond Semantics
• Dual Information Speech Language Models for Emotional Conversations
• Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models
• SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models
• MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
• DIFFA: Large Language Diffusion Models Can Listen and Understand
• GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awareness
• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
• Beyond Classification: Towards Speech Emotion Reasoning with Multitask AudioLLMs
• Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding
• AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Models
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data
• ALAS: An Automatic Latent Alignment Score for Audio Language Models
• Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models
• Towards Reliable Large Audio Language Model
• Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
• Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples
• Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
• SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information
• Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios
• SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases
• On The Landscape of Spoken Language Models: A Comprehensive Survey

Related

• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
• Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
• SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
• Speech Model Pre-training for End-to-End Spoken Language Understanding
• Instruction Data Generation and Unsupervised Adaptation for Speech Language Models