Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis

2025/07/02 by Carbonneau, Marc-André, van Niekerk, Benjamin, Seuté, Hugo +3
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2507.02176

Abstract

Modeling voice identity is challenging due to its multifaceted nature. In generative speech systems, identity is often assessed using automatic speaker verification (ASV) embeddings, designed for discrimination rather than characterizing identity. This paper investigates which aspects of a voice are captured in such representations. We find that widely used ASV embeddings focus mainly on static features like timbre and pitch range, while neglecting dynamic elements such as rhythm. We also identify confounding factors that compromise speaker similarity measurements and suggest mitigation strategies. To address these gaps, we propose U3D, a metric that evaluates speakers' dynamic rhythm patterns. This work contributes to the ongoing challenge of assessing speaker identity consistency in the context of ever-better voice cloning systems. We publicly release our code.

Citations

• Good practices for evaluation of synthesized speech
• Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
• Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement
• Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation
• Salmon: A Suite for Acoustic Language Model Evaluation
• VoxSim: A perceptual voice similarity dataset
• ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation
• Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model
• ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving
• CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models
• VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech Synthesis
• Rhythm Modeling for Voice Conversion
• The Singing Voice Conversion Challenge 2023
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Automatic Evaluation of Speaker Similarity
• Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• SpeechBrain: A General-Purpose Speech Toolkit
• The Multi-speaker Multi-style Voice Cloning Challenge 2021
• Investigating on Incorporating Pretrained and Learnable Speaker\n Representations for Multi-Speaker Multi-Style Text-to-Speech
• Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions
• Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings
• Generalized End-to-End Loss for Speaker Verification

Related

• Automatic Evaluation of Speaker Similarity
• From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint
• Evaluating Speaker Identity Coding in Self-supervised Models and Humans
• Explainable Attribute-Based Speaker Verification
• Estimating Uniqueness of I-Vector Representation of Human Voice