LinearVC: Linear transformations of self-supervised features through the lens of voice conversion

2025/06/02 by Herman Kamper, Kamper, Herman, Benjamin van Niekerk +5
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.01510

openalex publication_date 2025/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce LinearVC, a simple voice conversion method that sheds light on the structure of self-supervised representations. First, we show that simple linear transformations of self-supervised features effectively convert voices. Next, we probe the geometry of the feature space by constraining the set of allowed transformations. We find that just rotating the features is sufficient for high-quality voice conversion. This suggests that content information is embedded in a low-dimensional subspace which can be linearly transformed to produce a target voice. To validate this hypothesis, we finally propose a method that explicitly factorizes content and speaker information using singular value decomposition; the resulting linear projection with a rank of just 100 gives competitive conversion results. Our work has implications for both practical voice conversion and a broader understanding of self-supervised speech representations. Samples and code: https://www.kamperh.com/linearvc/.

Citations

• Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equation
• Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR
• Optimal Transport Maps are Good Voice Converters
• A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings
• The VoicePrivacy 2022 Challenge: Progress and Perspectives in Voice Anonymisation
• DASB - Discrete Audio and Speech Benchmark
• How Should We Extract Discrete Audio Tokens from Self-Supervised Models?
• Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
• StreamVC: Real-Time Low-Latency Voice Conversion
• Self-Supervised Models of Speech Infer Universal Articulatory Kinematics
• Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and Textually Described Voices
• Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion
• Voice Conversion With Just Nearest Neighbors
• Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
• SoundStorm: Efficient Parallel Audio Generation
• QuickVC: Any-to-many Voice Conversion Using Inverse Short-time Fourier Transform for Faster Conversion
• Robust Speech Recognition via Large-Scale Weak Supervision
• Comparative layer-wise analysis of self-supervised speech models
• FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Layer-wise Analysis of a Self-supervised Speech Representation Model
• FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions
• Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• A Generalized Solution of the Orthogonal Procrustes Problem

Cited by

• Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

Related

• SelfVC: Voice Conversion With Iterative Refinement using Self Transformations
• DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning
• ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations
• Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features
• AdaptVC: High Quality Voice Conversion with Adaptive Learning