Towards Reliable Large Audio Language Model

2025/05/25 by Ziyang Ma, Xiquan Li, Ma, Ziyang +17 · 3 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Human-Computer Interaction (cs.HC) #Multimedia (cs.MM) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.19294

openalex publication_date 2025/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in large audio language models (LALMs) have demonstrated impressive results and promising prospects in universal understanding and reasoning across speech, music, and general sound. However, these models still lack the ability to recognize their knowledge boundaries and refuse to answer questions they don't know proactively. While there have been successful attempts to enhance the reliability of LLMs, reliable LALMs remain largely unexplored. In this paper, we systematically investigate various approaches towards reliable LALMs, including training-free methods such as multi-modal chain-of-thought (MCoT), and training-based methods such as supervised fine-tuning (SFT). Besides, we identify the limitations of previous evaluation metrics and propose a new metric, the Reliability Gain Index (RGI), to assess the effectiveness of different reliable methods. Our findings suggest that both training-free and training-based methods enhance the reliability of LALMs to different extents. Moreover, we find that awareness of reliability is a "meta ability", which can be transferred across different audio modalities, although significant structural and content differences exist among sound, music, and speech.

Citations

• State-Space Large Audio Language Models
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
• DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
• Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech
• DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
• Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
• Language Model Can Listen While Speaking
• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
• GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
• Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?
• Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets
• MuPT: A Generative Symbolic Music Pretrained Transformer
• Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
• ChatMusician: Understanding and Generating Music Intrinsically with LLM
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
• Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT
• Qwen Technical Report
• Connecting Speech Encoder and Large Language Model for ASR
• BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing
• Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
• On decoder-only architecture for speech-to-text and large language model integration
• Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• LoRA: Low-Rank Adaptation of Large Language Models
• R-Tuning: Instructing Large Language Models to Say `I Don't Know'

Cited by

• X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
• Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models
• A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

Related

• Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey
• Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models
• AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
• Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models
• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment