LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

2025/05/24 by Pooneh Mousavi, Shubham Gupta, Mousavi, Pooneh +5 · 2 citations
Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Diverse Musicological Studies #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.18517

openalex publication_date 2025/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Foundation models based on large language models (LLMs) have shown great success in handling various tasks and modalities. However, adapting these models for general-purpose audio-language tasks is challenging due to differences in acoustic environments and task variations. In this work, we introduce LiSTEN Learning Soft Token Embeddings for Neural Audio LLMs), a framework for adapting LLMs to speech and audio tasks. LiSTEN uses a dynamic prompt selection strategy with learnable key-value pairs, allowing the model to balance general and task-specific knowledge while avoiding overfitting in a multitask setting. Our approach reduces dependence on large-scale ASR or captioning datasets, achieves competitive performance with fewer trainable parameters, and simplifies training by using a single-stage process. Additionally, LiSTEN enhances interpretability by analyzing the diversity and overlap of selected prompts across different tasks.

Citations

• LAST SToP For Modeling Asynchronous Time Series
• Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
• Chain-of-Thought Prompting for Speech Translation
• Language Model Can Listen While Speaking
• Qwen2-Audio Technical Report
• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
• AudioBench: A Universal Benchmark for Audio Large Language Models
• WavLLM: Towards Robust and Adaptive Speech Large Language Model
• AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• Instruction Tuning for Large Language Models: A Survey
• AudioPaLM: A Large Language Model That Can Speak and Listen
• SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts
• QLoRA: Efficient Finetuning of Quantized LLMs
• Pengi: An Audio Language Model for Audio Tasks
• Listen, Think, and Understand
• Instruction Tuning with GPT-4
• SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks
• LLaMA: Open and Efficient Foundation Language Models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning
• SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks
• Learning to Prompt for Continual Learning
• SoundStream: An End-to-End Neural Audio Codec
• LoRA: Low-Rank Adaptation of Large Language Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Common Voice: A Massively-Multilingual Speech Corpus
• LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• Investigating Faithfulness in Large Audio Language Models
• ALAS: An Automatic Latent Alignment Score for Audio Language Models

Related

• PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs
• MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders
• Audio-Language Models for Audio-Centric Tasks: A Systematic Survey
• AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
• Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data