VS-Singer: Vision-Guided Stereo Singing Voice Synthesis with Consistency Schrödinger Bridge

2025/06/19 by Zhao, Zijing, Wang, Kai, Huang, Hao +3
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Face recognition and analysis #Music Technology and Sound Studies #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.16020

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To explore the potential advantages of utilizing spatial cues from images for generating stereo singing voices with room reverberation, we introduce VS-Singer, a vision-guided model designed to produce stereo singing voices with room reverberation from scene images. VS-Singer comprises three modules: firstly, a modal interaction network integrates spatial features into text encoding to create a linguistic representation enriched with spatial information. Secondly, the decoder employs a consistency Schrödinger bridge to facilitate one-step sample generation. Moreover, we utilize the SFE module to improve the consistency of audio-visual matching. To our knowledge, this study is the first to combine stereo singing voice synthesis with visual acoustic matching within a unified framework. Experimental results demonstrate that VS-Singer can effectively generate stereo singing voices that align with the scene perspective in a single step.

Citations

• Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech
• Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speech
• An End-to-End Approach for Chord-Conditioned Song Generation
• Self-Supervised Visual Acoustic Matching
• CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
• Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
• Consistency Models
• I²SB: Image-to-Image Schrödinger Bridge
• Novel-View Acoustic Synthesis
• Scalable Diffusion Models with Transformers
• UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis
• VISinger 2: High-Fidelity End-to-End Singing Voice Synthesis Enhanced by Digital Signal Processing Synthesizer
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Elucidating the Design Space of Diffusion-Based Generative Models
• Video Diffusion Models
• Learning the Beauty in Songs: Neural Singing Voice Beautifier
• Visual Acoustic Matching
• Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
• Score-Based Generative Modeling through Stochastic Differential Equations
• Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Deep Residual Learning for Image Recognition

Related

• SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
• WeSinger 2: Fully Parallel Singing Voice Synthesis via Multi-Singer Conditional Adversarial Training
• SUSing: SU-net for Singing Voice Synthesis
• Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer
• VISinger2+: End-to-End Singing Voice Synthesis Augmented by Self-Supervised Learning Representation