Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation

2023/10/18 by Yiyang Su, Su, Yiyang, Ali Vosoughi +7 · 1 citation
Computer Science · Engineering · Neuroscience · #Advanced Adaptive Filtering Techniques #Audio and Speech Processing (eess.AS) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Hearing Loss and Rehabilitation #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2310.11713

openalex publication_date 2023/10/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds lacking visible cues. This paper introduces a novel "Audio-Visual Scene-Aware Separation" (AVSA-Sep) framework. It includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation. AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness.

Cited by

• MARS-Sep: Multimodal-Aligned Reinforced Sound Separation

Related

• Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds
• Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics
• Improving On-Screen Sound Separation for Open-Domain Videos with\n Audio-Visual Self-Attention
• Learning to Separate Object Sounds by Watching Unlabeled Video
• Co-Separating Sounds of Visual Objects