Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech

2021/05/19 by Keisuke Kinoshita, Marc Delcroix, Kinoshita, Keisuke +3 · 12 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2105.09040

5 pages, 1 figure, Interspeech2021. (Update to include a reference to the code)

openalex publication_date 2021/05/19 · openalex created_date 2021/05/24 · arxiv created 2021/08/31 · arxiv updated 2021/09/01 · openalex updated_date 2026/07/28

Abstract

Recently, we proposed a novel speaker diarization method called End-to-End-Neural-Diarization-vector clustering (EEND-vector clustering) that integrates clustering-based and end-to-end neural network-based diarization approaches into one framework. The proposed method combines advantages of both frameworks, i.e. high diarization performance and handling of overlapped speech based on EEND, and robust handling of long recordings with an arbitrary number of speakers based on clustering-based approaches. However, the method was only evaluated so far on simulated 2-speaker meeting-like data. This paper is to (1) report recent advances we made to this framework, including newly introduced robust constrained clustering algorithms, and (2) experimentally show that the method can now significantly outperform competitive diarization methods such as Encoder-Decoder Attractor (EDA)-EEND, on CALLHOME data which comprises real conversational speech data including overlapped speech and an arbitrary number of speakers. By further analyzing the experimental results, this paper also discusses pros and cons of the proposed method and reveals potential for further improvement. A set of the code to reproduce the results is available at https://github.com/nttcslab-sp/EEND-vector-clustering.

Citations

• A tutorial on spectral clustering
• Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds

Cited by

• O-EENC-SD: Efficient Online End-to-End Neural Clustering for Speaker Diarization
• On the Use of Self-Supervised Representation Learning for Speaker Diarization and Separation
• VBx for End-to-End Neural and Clustering-based Diarization
• Spatially Aware Self-Supervised Models for Multi-Channel Neural Speaker Diarization
• Benchmarking Diarization Models
• From Independence to Interaction: Speaker-Aware Simulation of Multi-Speaker Conversational Timing
• SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models
• Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges
• Can We Really Repurpose Multi-Speaker ASR Corpus for Speaker Diarization?
• Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
• Pretraining Multi-Speaker Identification for Neural Speaker Diarization
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Related

• Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds
• An Experimental Review of Speaker Diarization methods with application to Two-Speaker Conversational Telephone Speech recordings
• Robust End-to-end Speaker Diarization with Generic Neural Clustering
• Utterance-by-utterance overlap-aware neural diarization with Graph-PIT
• End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification