DiffWave: A Versatile Diffusion Model for Audio Synthesis

2020/09/21 by Kong, Zhifeng, Ping, Wei, Huang, Jiaji +2 · 118 citations
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2009.09761

Abstract

In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

Cited by

• Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks
• Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation
• Reverse-Time Diffusion Processes for Discrete Time Linear and Nonlinear Systems with non-Gaussian Noise
• Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
• Advancing Marine Bioacoustics with Deep Generative Models: A Hybrid Augmentation Strategy for Southern Resident Killer Whale Detection
• Residual Prior Diffusion: A Probabilistic Framework Integrating Coarse Latent Priors with Diffusion Models
• Aliasing-Free Neural Audio Synthesis
• Preserving Spectral Structure and Statistics in Diffusion Models
• SFBD-OMNI: Bridge models for lossy measurement restoration with limited clean samples
• Pseudo-Cepstrum: Pitch Modification for Mel-Based Neural Vocoders
• CoPHo: Classifier-guided Conditional Topology Generation with Persistent Homology
• LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models
• Composite Classifier-Free Guidance for Multi-Modal Conditioning in Wind Dynamics Super-Resolution
• Unified Control for Inference-Time Guidance of Denoising Diffusion Models
• Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
• Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making
• Unsupervised Single-Channel Audio Separation with Diffusion Source Priors
• NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation
• ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models
• GLA-Grad++: An Improved Griffin-Lim Guided Diffusion Model for Speech Synthesis
• Advancing physiological time series reconstruction and imputation via mixture of receptive fields and experts fusion
• Generating Separated Singing Vocals Using a Diffusion Model Conditioned on Music Mixtures
• Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model
• Diffusion for Fusion: Designing Stellarators with Generative AI
• Temporal-Visual Semantic Alignment: A Unified Architecture for Transferring Spatial Priors from Vision Models to Zero-Shot Temporal Tasks
• Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
• Pathlet Variational Auto-Encoder for Robust Trajectory Generation
• Towards Stable and Structured Time Series Generation with Perturbation-Aware Flow Matching
• H-LDM: Hierarchical Latent Diffusion Models for Controllable and Interpretable PCG Synthesis from Clinical Metadata
• Multi-modal Deepfake Detection and Localization with FPN-Transformer
• TimeFlow: Towards Stochastic-Aware and Efficient Time Series Generation via Flow Matching Modeling
• Forecasting implied volatility surface with generative diffusion models
• BridgeVoC: Revitalizing Neural Vocoder from a Restoration Perspective
• Diffolio: A Diffusion Model for Multivariate Probabilistic Financial Time-Series Forecasting and Portfolio Construction
• Learning to Land Anywhere: Transferable Generative Models for Aircraft Trajectories
• HAGI++: Head-Assisted Gaze Imputation and Generation
• Dynamic Population Distribution Aware Human Trajectory Generation with Diffusion Model
• Effective Series Decomposition and Components Learning for Time Series Generation
• Diffusion Models at the Drug Discovery Frontier: A Review on Generating Small Molecules versus Therapeutic Peptides
• NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion
• Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals for Cross-Domain Audio Deepfake Detection
• Closing Gaps: An Imputation Analysis of ICU Vital Signs
• Information-Theoretic Discrete Diffusion
• Sequence Modeling with Spectral Mean Flows
• Simple Denoising Diffusion Language Models
• Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge
• BadGraph: A Backdoor Attack Against Latent Diffusion Model for Text-Guided Graph Generation
• AutoScape: Geometry-Consistent Long-Horizon Scene Generation
• Diffusion Buffer for Online Generative Speech Enhancement
• Gradient Variance Reveals Failure Modes in Flow-Based Generative Models
• Adaptive Discretization for Consistency Models
• Inference-Time Search using Side Information for Diffusion-based Image Reconstruction
• Counting Hallucinations in Diffusion Models
• Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
• Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
• Unlocking the Potential of Diffusion Language Models through Template Infilling
• DiffStyleTS: Diffusion Model for Style Transfer in Time Series
• WaveletDiff: Multilevel Wavelet Diffusion For Time Series Generation
• SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment
• O_O-VC: Synthetic Data-Driven One-to-One Alignment for Any-to-Any Voice Conversion
• Guitar Tone Morphing by Diffusion-based Model
• Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive
• Benchmarking Fake Voice Detection in the Fake Voice Generation Arms Race
• Traj-Transformer: Diffusion Models with Transformer for GPS Trajectory Generation
• Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
• Pitch-Conditioned Instrument Sound Synthesis From an Interactive Timbre Latent Space
• GDiffuSE: Diffusion-based speech enhancement with noise model guidance
• Optimal estimation of a factorizable density using diffusion models with ReLU neural networks
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
• Contrastive Diffusion Guidance for Spatial Inverse Problems
• Data-to-Energy Stochastic Dynamics
• MARS: Audio Generation via Multi-Channel Autoregression on Spectrograms
• Environment-Aware Satellite Image Generation with Diffusion Models
• VoiceBridge: Designing Latent Bridge Models for General Speech Restoration at Scale
• High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
• Shortcut Flow Matching for Speech Enhancement: Step-Invariant flows via single stage training
• Score-based Idempotent Distillation of Diffusion Models
• SPREAD: Sampling-based Pareto front Refinement via Efficient Adaptive Diffusion
• TF-Restormer: Complex Spectral Prediction for Speech Restoration
• Prompt-aware classifier free guidance for diffusion models
• T2I-Diff: fMRI Signal Generation via Time-Frequency Image Transform and Classifier-Free Denoising Diffusion Models
• SimDiff: Simulator-constrained Diffusion Model for Physically Plausible Motion Generation
• PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
• Learnable Sampler Distillation for Discrete Diffusion Models
• DynaFlow: Dynamics-embedded Flow Matching for Physically Consistent Motion Generation from State-only Demonstrations
• DiffQ: Unified Parameter Initialization for Variational Quantum Algorithms via Diffusion Models
• Audio Super-Resolution with Latent Bridge Models
• Discrete-Time Diffusion-Like Models for Speech Synthesis
• FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
• Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification
• Generative AI for Multimedia Communication: Recent Advances, An Information-Theoretic Framework, and Future Opportunities
• Scaling to Multimodal and Multichannel Heart Sound Classification with Synthetic and Augmented Biosignals
• AdaSTI: Conditional Diffusion Models with Adaptive Dependency Modeling for Spatio-Temporal Imputation
• Flow Straight and Fast in Hilbert Space: Functional Rectified Flow
• Automated Tuning for Diffusion Inverse Problem Solvers without Generative Prior Retraining
• MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection
• WildSpoof Challenge Evaluation Plan
• ArtifactGen: Benchmarking WGAN-GP vs Diffusion for Label-Aware EEG Artifact Synthesis
• DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
• Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Finance
• FlowECG: Using Flow Matching to Create a More Efficient ECG Signal Generator
• Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model
• Visually Grounded Narratives: Reducing Cognitive Burden in Researcher-Participant Interaction
• Towards High-Fidelity and Controllable Bioacoustic Generation via Enhanced Diffusion Learning
• Stage-Diff: Stage-wise Long-Term Time Series Generation Based on Diffusion Models
• WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration
• FoleySpace: Vision-Aligned Binaural Spatial Audio Generation
• Diffusion is a code repair operator and generator
• EEGDM: EEG Representation Learning via Generative Diffusion Model
• UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
• Enhancing Spectrogram Realism in Singing Voice Synthesis via Explicit Bandwidth Extension Prior to Vocoder
• Multi-Granularity Adaptive Time-Frequency Attention Framework for Audio Deepfake Detection under Real-World Communication Degradations
• Occlusion-robust Stylization for Drawing-based 3D Animation
• Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction
• Flow Matching Policy Gradients
• Learning Neural Vocoder from Range-Null Space Decomposition
• SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations
• SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusion

Related

• DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation
• WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching
• Progressive distillation diffusion for raw music generation
• EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis
• WaveGrad: Estimating Gradients for Waveform Generation