Domain Adaptation Method and Modality Gap Impact in Audio-Text Models for Prototypical Sound Classification

2025/06/04 by Acevedo, Emiliano, Rocamora, Martín, Fuentes, Magdalena
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2506.04376

Abstract

Audio-text models are widely used in zero-shot environmental sound classification as they alleviate the need for annotated data. However, we show that their performance severely drops in the presence of background sound sources. Our analysis reveals that this degradation is primarily driven by SNR levels of background soundscapes, and independent of background type. To address this, we propose a novel method that quantifies and integrates the contribution of background sources into the classification process, improving performance without requiring model retraining. Our domain adaptation technique enhances accuracy across various backgrounds and SNR conditions. Moreover, we analyze the modality gap between audio and text embeddings, showing that narrowing this gap improves classification performance. The method generalizes effectively across state-of-the-art prototypical approaches, showcasing its scalability and robustness for diverse environments.

Citations

• PALM: Few-Shot Prompt Learning for Audio Language Models
• Multi-label Zero-Shot Audio Classification with Temporal Attention
• It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
• Domain Adaptation for Contrastive Audio-Language Models
• Weakly-supervised Automated Audio Captioning via text only training
• A Multimodal Prototypical Approach for Unsupervised Sound Classification
• Audio-Text Models Do Not Yet Leverage Natural Language
• Robust Speech Recognition via Large-Scale Weak Supervision
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
• AudioCLIP: Extending CLIP to Image, Text and Audio
• A multi-device dataset for urban acoustic scene classification

Related

• A Multimodal Prototypical Approach for Unsupervised Sound Classification
• Synthetic training set generation using text-to-audio models for environmental sound classification
• Unsupervised Improvement of Audio-Text Cross-Modal Representations
• VOICe: A Sound Event Detection Dataset For Generalizable Domain Adaptation
• Trainingless Adaptation of Pretrained Models for Environmental Sound Classification