Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond

2025/05/07 by Jessie Richter-Powell, Antonio Torralba, Richter-Powell, Jessie +3 · 1 citation
Computer Science · #68T07 #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Generative Adversarial Networks and Image Synthesis #H.5.1 #H.5.5 #I.2.6 #Machine Learning (cs.LG) #Multimedia (cs.MM) #Music Technology and Sound Studies #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.04621

openalex publication_date 2025/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Audio-SDS, a generalization of Score Distillation Sampling (SDS) to text-conditioned audio diffusion models. While SDS was initially designed for text-to-3D generation using image diffusion, its core idea of distilling a powerful generative prior into a separate parametric representation extends to the audio domain. Leveraging a single pretrained model, Audio-SDS enables a broad range of tasks without requiring specialized datasets. In particular, we demonstrate how Audio-SDS can guide physically informed impact sound simulations, calibrate FM-synthesis parameters, and perform prompt-specified source separation. Our findings illustrate the versatility of distillation-based methods across modalities and establish a robust foundation for future work using generative priors in audio tasks.

Citations

• Multi-student Diffusion Distillation for Better One-step Generators
• JacNet: Learning Functions with Structured Jacobians
• The Llama 3 Herd of Models
• ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation
• Scalable Nested Optimization for Deep Learning
• Improving Hyperparameter Optimization with Checkpointed Model Weights
• Score Distillation via Reparametrized DDIM
• Training Data Attribution via Approximate Unrolled Differentiation
• Align Your Steps: Optimizing Sampling Schedules in Diffusion Models
• PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation
• LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis
• FlashTex: Fast Relightable Mesh Texturing with LightControlNet
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
• Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
• Graph Metanetworks for Processing Diverse Neural Architectures
• Using Large Language Models for Hyperparameter Optimization
• 4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling
• Text-to-3D using Gaussian Splatting
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
• Separate Anything You Describe
• DreamEditor: Text-Driven 3D Scene Editing with Neural Fields
• ATT3D: Amortized Text-to-3D Object Synthesis
• HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
• Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
• Zero-1-to-3: Zero-shot One Image to 3D Object
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• On Implicit Bias in Overparameterized Bilevel Optimization
• Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation
• SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction
• Magic3D: High-Resolution Text-to-3D Content Creation
• Diffusion-based Generative Speech Source Separation
• AudioGen: Textually Guided Audio Generation
• DreamFusion: Text-to-3D using 2D Diffusion
• Task Selection for AutoML System Evaluation
• Classifier-Free Diffusion Guidance
• Learning Neural Acoustic Fields
• Generative Modeling for Low Dimensional Speech Attributes with Neural Spline Flows
• Lyapunov Exponents for Diversity in Differentiable Games
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Meta-Learning to Improve Pre-Training
• SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
• Complex Momentum for Optimization in Games
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• WaveGrad: Estimating Gradients for Waveform Generation
• Denoising Diffusion Probabilistic Models
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• DDSP: Differentiable Digital Signal Processing
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• High Fidelity Speech Synthesis with Adversarial Networks
• Music Source Separation in the Waveform Domain
• Understanding Neural Architecture Search Techniques
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
• GANSynth: Adversarial Neural Audio Synthesis
• WaveGlow: A Flow-based Generative Network for Speech Synthesis
• Negative Momentum for Improved Game Dynamics
• Adversarial Audio Synthesis
• Population Based Training of Neural Networks
• WaveNet: A Generative Model for Raw Audio

Cited by

• PromptReverb: Multimodal Room Impulse Response Generation Through Latent Rectified Flow Matching

Related

• A Review on Score-based Generative Models for Audio Applications
• CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis
• SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
• EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis