Multi-Level Attention Aggregation for Language-Agnostic Speaker Replication

2024/03/06 by Yejin Jeon, Jeon, Yejin, Gary Geunbae Lee +1
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2403.04111

openalex publication_date 2024/03/06 · openalex created_date 2024/03/09 · openalex updated_date 2026/07/28

Abstract

This paper explores the task of language-agnostic speaker replication, a novel endeavor that seeks to replicate a speaker's voice irrespective of the language they are speaking. Towards this end, we introduce a multi-level attention aggregation approach that systematically probes and amplifies various speaker-specific attributes in a hierarchical manner. Through rigorous evaluations across a wide range of scenarios including seen and unseen speakers conversing in seen and unseen lingua, we establish that our proposed model is able to achieve substantial speaker similarity, and is able to generalize to out-of-domain (OOD) cases.

Related

• LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention
• SpeakerStew: Scaling to Many Languages with a Triaged Multilingual\n Text-Dependent and Text-Independent Speaker Verification System
• Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers
• Text-Independent Speaker Verification with Dual Attention Network
• Attention Mechanism in Speaker Recognition: What Does It Learn in Deep Speaker Embedding?