Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers

2020/06/19 by Kanda, Naoyuki, Gaur, Yashesh, Wang, Xiaofei +4 · 7 citations
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2006.10930

Abstract

We propose an end-to-end speaker-attributed automatic speech recognition model that unifies speaker counting, speech recognition, and speaker identification on monaural overlapped speech. Our model is built on serialized output training (SOT) with attention-based encoder-decoder, a recently proposed method for recognizing overlapped speech comprising an arbitrary number of speakers. We extend SOT by introducing a speaker inventory as an auxiliary input to produce speaker labels as well as multi-speaker transcriptions. All model parameters are optimized by speaker-attributed maximum mutual information criterion, which represents a joint probability for overlapped speech recognition and speaker identification. Experiments on LibriSpeech corpus show that our proposed method achieves significantly better speaker-attributed word error rate than the baseline that separately performs overlapped speech recognition and speaker identification.

Cited by

• Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio
• Spatially-Augmented Sequence-to-Sequence Neural Diarization for Meetings
• GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASR
• Data-independent Beamforming for End-to-end Multichannel Multi-speaker ASR
• The Impact of Automatic Speech Transcription on Speaker Attribution
• Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR
• Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios

Related

• Serialized Output Training for End-to-End Overlapped Speech Recognition
• Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordings
• SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition
• End-to-End Speaker-Attributed ASR with Transformer
• Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASR