Controllable Singing Voice Synthesis using Phoneme-Level Energy Sequence

2025/09/08 by Ryu, Yerin, Inseop Shin, Chanwoo Kim +2
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.07038

openalex publication_date 2025/09/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Controllable Singing Voice Synthesis (SVS) aims to generate expressive singing voices reflecting user intent. While recent SVS systems achieve high audio quality, most rely on probabilistic modeling, limiting precise control over attributes such as dynamics. We address this by focusing on dynamic control--temporal loudness variation essential for musical expressiveness--and explicitly condition the SVS model on energy sequences extracted from ground-truth spectrograms, reducing annotation costs and improving controllability. We also propose a phoneme-level energy sequence for user-friendly control. To the best of our knowledge, this is the first attempt enabling user-driven dynamics control in SVS. Experiments show our method achieves over 50% reduction in mean absolute error of energy sequences for phoneme-level inputs compared to baseline and energy-predictor models, without compromising synthesis quality.

Citations

• TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching
• GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
• Period Singer: Integrating Periodic and Aperiodic Variational Autoencoders for Natural-Sounding End-to-End Singing Voice Synthesis
• MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance
• SiFiSinger: A High-Fidelity End-to-End Singing Voice Synthesizer based on Source-filter Model
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt
• Music ControlNet: Multiple Time-varying Controls for Music Generation
• RMSSinger: Realistic-Music-Score based Singing Voice Synthesis
• Singing Voice Synthesis with Vibrato Modeling and Latent Energy Representation
• Towards Improving the Expressiveness of Singing Voice Synthesis with BERT Derived Semantic Information
• MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling
• VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Denoising Diffusion Probabilistic Models
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Related

• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt
• Singing Voice Synthesis with Vibrato Modeling and Latent Energy Representation
• ConSinger: Efficient High-Fidelity Singing Voice Generation with Minimal Steps
• A Preliminary Investigation on Flexible Singing Voice Synthesis Through Decomposed Framework with Inferrable Features
• Singing Synthesis: with a little help from my attention