DiffWave: A Versatile Diffusion Model for Audio Synthesis
2020/09/21 by Zhifeng Kong, Wei Ping, Kong, Zhifeng +7 · 235 citations
Computer Science · Engineering · Mathematics · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.CL #cs.LG #cs.SD #eess.AS #electronic engineering #information engineering #stat.ML
paper · pdf · doi:10.48550/arxiv.2009.09761
ICLR 2021 (oral)
openalex publication_date 2020/09/21 · arxiv created 2021/03/30 · arxiv updated 2021/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.
Citations
Cited by
- Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks
- Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation
- Reverse-Time Diffusion Processes for Discrete Time Linear and Nonlinear Systems with non-Gaussian Noise
- Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
- Advancing Marine Bioacoustics with Deep Generative Models: A Hybrid Augmentation Strategy for Southern Resident Killer Whale Detection
- Residual Prior Diffusion: A Probabilistic Framework Integrating Coarse Latent Priors with Diffusion Models
- Aliasing-Free Neural Audio Synthesis
- Preserving Spectral Structure and Statistics in Diffusion Models
- SFBD-OMNI: Bridge models for lossy measurement restoration with limited clean samples
- Pseudo-Cepstrum: Pitch Modification for Mel-Based Neural Vocoders
- CoPHo: Classifier-guided Conditional Topology Generation with Persistent Homology
- LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models
- Composite Classifier-Free Guidance for Multi-Modal Conditioning in Wind Dynamics Super-Resolution
- Unified Control for Inference-Time Guidance of Denoising Diffusion Models
- Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
- Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making
- Unsupervised Single-Channel Audio Separation with Diffusion Source Priors
- NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation
- ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models
- GLA-Grad++: An Improved Griffin-Lim Guided Diffusion Model for Speech Synthesis
- Advancing time series completion via RFAMoE and MDFF
- Generating Separated Singing Vocals Using a Diffusion Model Conditioned on Music Mixtures
- Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model
- Diffusion for Fusion: Designing Stellarators with Generative AI
- Temporal-Visual Semantic Alignment: A Unified Architecture for Transferring Spatial Priors from Vision Models to Zero-Shot Temporal Tasks
- Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
- Pathlet Variational Auto-Encoder for Robust Trajectory Generation
- Towards Stable and Structured Time Series Generation with Perturbation-Aware Flow Matching
- H-LDM: Hierarchical Latent Diffusion Models for Controllable and Interpretable PCG Synthesis from Clinical Metadata
- Multi-modal Deepfake Detection and Localization with FPN-Transformer
- TimeFlow: Towards Stochastic-Aware and Efficient Time Series Generation via Flow Matching Modeling
- Forecasting implied volatility surface with generative diffusion models
- BridgeVoC: Revitalizing Neural Vocoder from a Restoration Perspective
- Diffolio: A Diffusion Model for Multivariate Probabilistic Financial Time-Series Forecasting and Portfolio Construction
- Learning to Land Anywhere: Transferable Generative Models for Aircraft Trajectories
- HAGI++: Head-Assisted Gaze Imputation and Generation
- Dynamic Population Distribution Aware Human Trajectory Generation with Diffusion Model
- Effective Series Decomposition and Components Learning for Time Series Generation
- Diffusion Models at the Drug Discovery Frontier: A Review on Generating Small Molecules versus Therapeutic Peptides
- NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion
- Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals for Cross-Domain Audio Deepfake Detection
- Closing Gaps: An Imputation Analysis of ICU Vital Signs
- Information-Theoretic Discrete Diffusion
- Sequence Modeling with Spectral Mean Flows
- Simple Denoising Diffusion Language Models
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge
- BadGraph: A Backdoor Attack Against Latent Diffusion Model for Text-Guided Graph Generation
- AutoScape: Geometry-Consistent Long-Horizon Scene Generation
- Diffusion Buffer for Online Generative Speech Enhancement
- Gradient Variance Reveals Failure Modes in Flow-Based Generative Models
- Adaptive Discretization for Consistency Models
- Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction
- Counting Hallucinations in Diffusion Models
- Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
- Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
- Unlocking the Potential of Diffusion Language Models through Template Infilling
- DiffStyleTS: Diffusion Model for Style Transfer in Time Series
- WaveletDiff: Multilevel Wavelet Diffusion For Time Series Generation
- SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment
- OO-VC: Synthetic Data-Driven One-to-One Alignment for Any-to-Any Voice Conversion
- Guitar Tone Morphing by Diffusion-based Model
- Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive
- Benchmarking Fake Voice Detection in the Fake Voice Generation Arms Race
- Traj-Transformer: Diffusion Models with Transformer for GPS Trajectory Generation
- Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
- Pitch-Conditioned Instrument Sound Synthesis From an Interactive Timbre Latent Space
- GDiffuSE: Diffusion-based speech enhancement with noise model guidance
- Optimal estimation of a factorizable density using diffusion models with ReLU neural networks
- Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
- Contrastive Diffusion Guidance for Spatial Inverse Problems
- Data-to-Energy Stochastic Dynamics
- MARS: Sound Generation via Multi-Channel Autoregression on Spectrograms
- Environment-Aware Satellite Image Generation with Diffusion Models
- VoiceBridge: Designing Latent Bridge Models for General Speech Restoration at Scale
- High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
- Shortcut Flow Matching for Speech Enhancement: Step-Invariant flows via single stage training
- Score-based Idempotent Distillation of Diffusion Models
- SPREAD: Sampling-based Pareto front Refinement via Efficient Adaptive Diffusion
- TF-Restormer: Complex Spectral Prediction for Speech Restoration
- Prompt-aware classifier free guidance for diffusion models
- T2I-Diff: fMRI Signal Generation via Time-Frequency Image Transform and Classifier-Free Denoising Diffusion Models
- SimDiff: Simulator-constrained Diffusion Model for Physically Plausible Motion Generation
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
- Learnable Sampler Distillation for Discrete Diffusion Models
- DynaFlow: Dynamics-embedded Flow Matching for Physically Consistent Motion Generation from State-only Demonstrations
- DiffQ: Unified Parameter Initialization for Variational Quantum Algorithms via Diffusion Models
- Audio Super-Resolution with Latent Bridge Models
- Discrete-Time Diffusion-Like Models for Speech Synthesis
- FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
- Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification
- BemaGANv2: Discriminator Combination Strategies for GAN-based Vocoders in Long-Term Audio Generation
- Generative AI for Multimedia Communication: Recent Advances, An Information-Theoretic Framework, and Future Opportunities
- Scaling to Multimodal and Multichannel Heart Sound Classification with Synthetic and Augmented Biosignals
- AdaSTI: Conditional Diffusion Models with Adaptive Dependency Modeling for Spatio-Temporal Imputation
- Flow Straight and Fast in Hilbert Space: Functional Rectified Flow
- Automated Tuning for Diffusion Inverse Problem Solvers without Generative Prior Retraining
- MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection
- WildSpoof Challenge Evaluation Plan
- ArtifactGen: Benchmarking WGAN-GP vs Diffusion for Label-Aware EEG Artifact Synthesis
- DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
- Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Finance
- FlowECG: Using Flow Matching to Create a More Efficient ECG Signal Generator
- Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model
- Visually Grounded Narratives: Reducing Cognitive Burden in Researcher-Participant Interaction
- Towards High-Fidelity and Controllable Bioacoustic Generation via Enhanced Diffusion Learning
- Stage-Diff: Stage-wise Long-Term Time Series Generation Based on Diffusion Models
- WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration
- FoleySpace: Vision-Aligned Binaural Spatial Audio Generation
- Diffusion is a code repair operator and generator
- EEGDM: EEG Representation Learning via Generative Diffusion Model
- UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
- Enhancing Spectrogram Realism in Singing Voice Synthesis via Explicit Bandwidth Extension Prior to Vocoder
- Multi-Granularity Adaptive Time-Frequency Attention Framework for Audio Deepfake Detection under Real-World Communication Degradations
- Occlusion-robust Stylization for Drawing-based 3D Animation
- Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction
- Flow Matching Policy Gradients
- Learning Neural Vocoder from Range-Null Space Decomposition
- CHORDS: Diffusion Sampling Accelerator with Multi-core Hierarchical ODE Solvers
- SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations
- SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusion
- A diffusion-based generative model for financial time series via geometric Brownian motion
- A Comprehensive Review of Diffusion Models in Smart Agriculture: Progress, Applications, and Challenges
- Distilling Parallel Gradients for Fast ODE Solvers of Diffusion Models
- Continual Speaker Identity Unlearning with Minimal Interference
- Diffusion Models for Solving Inverse Problems via Posterior Sampling with Piecewise Guidance
- Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
- Diffusion-based translation between unpaired spontaneous premature neonatal EEG and fetal MEG
- RODS: Robust Optimization Inspired Diffusion Sampling for Detecting and Reducing Hallucination in Generative Models
- A Simple Analysis of Discretization Error in Diffusion Models
- NeuTSFlow: Modeling Continuous Functions Behind Time Series Forecasting
- How Much To Guide: Revisiting Adaptive Guidance in Classifier-Free Guidance Text-to-Vision Diffusion Models
- Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance
- Knowing When to Quit: Probabilistic Early Exits for Speech Separation
- Physics vs Distributions: Pareto Optimal Flow Matching with Physics Constraints
- Warm Starts Accelerate Conditional Diffusion
- Beyond Scores: Proximal Diffusion Models
- Token-Based Audio Inpainting via Discrete Diffusion
- OnlineCache: Learning Dynamic Caching Policies with Error Correction for Efficient Diffusion Inference
- Leveraging the Spatial Hierarchy: Coarse-to-fine Trajectory Generation via Cascaded Hybrid Diffusion
- Generalization bounds for score-based generative models: a synthetic proof
- Exploring Efficient Waveform Diffusion Models for Foley Sound Generation
- ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization
- De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks
- A conversational gesture synthesis system based on emotions and semantics
- Flow-Modulated Scoring for Semantic-Aware Knowledge Graph Completion
- FunDiff: Diffusion Models over Function Spaces for Physics-Informed Generative Modeling
- SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture
- Stochastic and Non-local Closure Modeling for Nonlinear Dynamical Systems via Latent Score-based Generative Models
- HiWave: Training-Free High-Resolution Image Generation via Wavelet-Based Diffusion Sampling
- Ctrl-Z Sampling: Diffusion Sampling with Controlled Random Zigzag Explorations
- Regularity of the score function in generative models
- SceneCrafter: Controllable Multi-View Driving Scene Editing
- Guidance in the Frequency Domain Enables High-Fidelity Sampling at Low CFG Scales
- A Survey on World Models Grounded in Acoustic Physical Information
- Evolvable Conditional Diffusion
- Towards Physics-informed Diffusion for Anomaly Detection in Trajectories
- Harmonizing Geometry and Uncertainty: Diffusion with Hyperspheres
- What Exactly Does Guidance Do in Masked Discrete Diffusion Models
- The Diffusion Duality
- Geometric Regularity in Deterministic Sampling of Diffusion-based Generative Models
- Antithetic Noise in Diffusion Models
- Diffusion-Based Hierarchical Graph Neural Networks for Simulating Nonlinear Solid Mechanics
- How to Unlock Time Series Editing? Diffusion-Driven Approach with Multi-Grained Control
- Winner-takes-all for Multivariate Probabilistic Time Series Forecasting
- Survey on the Evaluation of Generative Models in Music
- Federated Learning Assisted Edge Caching Scheme Based on Lightweight Architecture DDPM
- Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond
- SFBD Flow: A Continuous-Optimization Framework for Training Diffusion Models with Noisy Samples
- Enabling Probabilistic Learning on Manifolds through Double Diffusion Maps
- Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradients
- Using Ensemble Diffusion to Estimate Uncertainty for End-to-End Autonomous Driving
- Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms
- ZeroSep: Separate Anything in Audio with Zero Training
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
- Score-based Generative Modeling for Conditional Independence Testing
- BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
- Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
- Physics-inspired Generative AI models via real hardware-based noisy quantum diffusion
- Almost Linear Convergence under Minimal Score Assumptions: Quantized Transition Diffusion
- Versatile Cardiovascular Signal Generation with a Unified Diffusion Transformer
- Is Noise Conditioning Necessary? A Unified Theory of Unconditional Graph Diffusion Models
- T2S: High-resolution Time Series Generation with Text-to-Series Diffusion Models
- Bilateral Denoising Diffusion Models
- Leveraging Diffusion Models for Parameterized Quantum Circuit Generation
- Conditional Diffusion Models with Classifier-Free Gibbs-like Guidance
- Learning Single Index Models with Diffusion Priors
- Time Series Generation Under Data Scarcity: A Unified Generative Modeling Approach
- Energy-based generator matching: A neural sampler for general state space
- DiffNMR: Advancing Inpainting of Randomly Sampled Nuclear Magnetic Resonance Signals
- Harnessing the Power of Training-Free Techniques in Text-to-2D Generation for Text-to-3D Generation via Score Distillation Sampling
- DiffusionRL: Efficient Training of Diffusion Policies for Robotic Grasping Using RL-Adapted Large-Scale Datasets
- Audio-to-Audio Emotion Conversion With Pitch And Duration Style Transfer
- Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling
- FlashBack: Consistency Model-Accelerated Shared Autonomy
- An Exploratory Study on Multi-modal Generative AI in AR Storytelling
- MSDformer: Multi-scale Discrete Transformer For Time Series Generation
- Challenges and Limitations of Generative AI in Synthesizing Wearable Sensor Data
- Learning to Integrate Diffusion ODEs by Averaging the Derivatives
- One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling
- Alternators With Noise Models
- VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloning
- Recent Advances in Diffusion Models for Hyperspectral Image Processing and Analysis: A Review
- DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion
- LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models
- SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset
- Distilling Drifting Transformers with Representation Autoencoders
- MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
- Local-Time Riemannian Score Matching on the Quantum Pure-State Manifold
- Research on Anomaly Detection Methods Based on Diffusion Models
- ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
- Diffusion Language Models: An Experimental Analysis
- E4GEN: Event-level Explainable Extreme-Enhanced Time-series Generation
- Q-Tag: watermarking quantum circuit generative models
- A Time-Series Data Augmentation Model through Diffusion and Transformer Integration
- Generalized Discrete Diffusion from Snapshots
- TriniMark: A Robust Generative Speech Watermarking Method for Trinity-Level Traceability
- Sparsely Supervised Diffusion
- Integration Flow Models
- Particle-Guided Diffusion Models for Partial Differential Equations
- BanglaRobustNet: A Hybrid Denoising-Attention Architecture for Robust Bangla Speech Recognition
- SynEnergy: Anomaly Semantic-Guided Diffusion for Synthetic Energy Data Generation
- OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films
- SOLIDO: A Robust Watermarking Method for Speech Synthesis via Low-Rank Adaptation
- Emergence and Evolution of Interpretable Concepts in Diffusion Models
- Diffusion-Driven Inertial Generated Data for Smartphone Location Classification
- Image Editing with Diffusion Models: A Survey
- Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning
- Generalized Visual Relation Detection with Diffusion Models
- Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech
- Scalable Motion In-betweening via Diffusion and Physics-Based Character Adaptation
- SD-ReID: View-aware Stable Diffusion for Aerial-Ground Person Re-Identification
- D2iT: Dynamic Diffusion Transformer for Accurate Image Generation
- On the Design of Diffusion-based Neural Speech Codecs
- SlimSpeech: Lightweight and Efficient Text-to-Speech with Slim Rectified Flow
- Probabilistic QoS Metric Forecasting in Delay-Tolerant Networks Using Conditional Diffusion Models on Latent Dynamics
Related