Exploring Speaker Diarization with Mixture of Experts

2025/06/17 by Gaobin Yang, Maokui He, Yang, Gaobin +9
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Sound (cs.SD) #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2506.14750

openalex publication_date 2025/06/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a novel neural speaker diarization system using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S), which integrates a memory-aware multi-speaker embedding module with a sequence-to-sequence architecture. The system leverages a memory module to enhance speaker embeddings and employs a Seq2Seq framework to efficiently map acoustic features to speaker labels. Additionally, we explore the application of mixture of experts in speaker diarization, and introduce a Shared and Soft Mixture of Experts (SS-MoE) module to further mitigate model bias and enhance performance. Incorporating SS-MoE leads to the extended model NSD-MS2S-SSMoE. Experiments on multiple complex acoustic datasets, including CHiME-6, DiPCo, Mixer 6 and DIHARD-III evaluation sets, demonstrate meaningful improvements in robustness and generalization. The proposed methods achieve state-of-the-art results, showcasing their effectiveness in challenging real-world scenarios.

Citations

• NTT Multi-Speaker ASR System for the DASR Task of CHiME-8 Challenge
• Semi-supervised multi-channel speaker diarization with cross-channel attention
• The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios
• Target-Speaker Voice Activity Detection via Sequence-to-Sequence Prediction
• Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization
• DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
• A Review of Speaker Diarization: Recent Advances with Deep Learning
• Bayesian HMM clustering of x-vector sequences (VBx) in speaker\n diarization: theory, implementation and analysis on standard tasks
• The Third DIHARD Diarization Challenge
• Online End-to-End Neural Diarization with Speaker-Tracing Buffer
• End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors
• CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings
• Speaker detection in the wild: Lessons learned from JSALT 2019
• DiPCo -- Dinner Party Corpus
• End-to-End Neural Speaker Diarization with Self-attention
• End-to-End Neural Speaker Diarization with Permutation-Free Objectives
• Speaker Diarization with LSTM
• Generalized End-to-End Loss for Speaker Verification

Related

• Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecture
• Semi-supervised multi-channel speaker diarization with cross-channel attention
• End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification
• Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios
• End-to-End Neural Speaker Diarization with Permutation-Free Objectives