Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification

2021/10/12 by Zhengyang Chen, Chen, Zhengyang, Sanyuan Chen +13 · 23 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2110.05777

openalex publication_date 2021/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The speech representations learned from large-scale unlabeled data have shown better generalizability than those from supervised learning and thus attract a lot of interest to be applied for various downstream tasks. In this paper, we explore the limits of speech representations learned by different self-supervised objectives and datasets for automatic speaker verification (ASV), especially with a well-recognized SOTA ASV model, ECAPA-TDNN [1], as a downstream model. The representations from all hidden layers of the pre-trained model are firstly averaged with learnable weights and then fed into the ECAPA-TDNN as input features. The experimental results on Voxceleb dataset show that the weighted average representation is significantly superior to FBank, a conventional handcrafted feature for ASV. Our best single system achieves 0.537%, 0.569%, and 1.180% equal error rate (EER) on the three official trials of VoxCeleb1, separately. Accordingly, the ensemble system with three pre-trained models can further improve the EER to 0.479%, 0.536% and 1.023%. Among the three evaluation trials, our best system outperforms the winner system [2] of the VoxCeleb Speaker Recognition Challenge 2021 (VoxSRC2021) on the VoxCeleb1-E trial.

Cited by

• Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm
• Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification
• GLM-TTS Technical Report
• DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec
• SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity
• SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization
• Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning
• SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement
• Short-Segment Speaker Verification with Pre-trained Models and Multi-Resolution Encoder
• Group Relative Policy Optimization for Text-to-Speech with Large Language Models
• Nord-Parl-TTS: Finnish and Swedish TTS Dataset from Parliament Speech
• DAIEN-TTS: Disentangled Audio Infilling for Environment-Aware Text-to-Speech Synthesis
• SV-Mixer: Replacing the Transformer Encoder with Lightweight MLPs for Self-Supervised Model Compression in Speaker Verification
• MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement
• Emoanti: audio anti-deepfake with refined emotion-guided representations
• DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibration
• Accelerating Diffusion Transformer-Based Text-to-Speech with Transformer Layer Caching
• LibriQuote: A Speech Dataset of Fictional Character Utterances for Expressive Zero-Shot Speech Synthesis
• Hybrid Pruning: In-Situ Compression of Self-Supervised Speech Models for Speaker Verification and Anti-Spoofing
• MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech
• MDD: a Mask Diffusion Detector to Protect Speaker Verification Systems from Adversarial Perturbations
• Exploring Self-Supervised Audio Models for Generalized Anomalous Sound Detection
• Selection of Layers from Self-supervised Learning Models for Predicting Mean-Opinion-Score of Speech

Related

• Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models
• Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
• One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
• Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
• The Microsoft System for VoxCeleb Speaker Recognition Challenge 2022