AVFSNet: Audio-Visual Speech Separation for Flexible Number of Speakers with Multi-Scale and Multi-Task Learning

2025/07/17 by Daning Zhang, Ying Wei, Zhang, Daning +1
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.12972

openalex publication_date 2025/07/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Separating target speech from mixed signals containing flexible speaker quantities presents a challenging task. While existing methods demonstrate strong separation performance and noise robustness, they predominantly assume prior knowledge of speaker counts in mixtures. The limited research addressing unknown speaker quantity scenarios exhibits significantly constrained generalization capabilities in real acoustic environments. To overcome these challenges, this paper proposes AVFSNet -- an audio-visual speech separation model integrating multi-scale encoding and parallel architecture -- jointly optimized for speaker counting and multi-speaker separation tasks. The model independently separates each speaker in parallel while enhancing environmental noise adaptability through visual information integration. Comprehensive experimental evaluations demonstrate that AVFSNet achieves state-of-the-art results across multiple evaluation metrics and delivers outstanding performance on diverse datasets.

Citations

• Interspeech 2025 URGENT Speech Enhancement Challenge
• USED: Universal Speaker Extraction and Diarization
• IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation
• AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction
• Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model
• An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits
• Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding
• EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers
• FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement
• REAL-M: Towards Speech Separation on Real Mixtures
• USEV: Universal Speaker Extraction with Visual Cue
• Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual\n Speech Separation
• Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• Attention is All You Need in Speech Separation
• Muse: Multi-modal target speaker extraction with visual cues
• Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation
• Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals
• End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Voice Separation with an Unknown Number of Multiple Speakers
• Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
• Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation
• Time Domain Audio Visual Speech Separation
• Recursive speech separation for unknown number of speakers
• SDR - half-baked or well done?
• Supervised Speech Separation Based on Deep Learning: An Overview
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry\n and Semantics
• Deep clustering: Discriminative embeddings for segmentation and separation

Related

• AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling
• Improved Source Counting and Separation for Monaural Mixture
• From Coarse to Fine: Recursive Audio-Visual Semantic Enhancement for Speech Separation
• Voice Separation with an Unknown Number of Multiple Speakers
• Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model