SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filtering

2025/09/23 by Jiarui Hai, Hai, Jiarui, Mounya Elhilali +1
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.18603

openalex publication_date 2025/09/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Data synthesis and augmentation are essential for Sound Event Detection (SED) due to the scarcity of temporally labeled data. While augmentation methods like SpecAugment and Mix-up can enhance model performance, they remain constrained by the diversity of existing samples. Recent generative models offer new opportunities, yet their direct application to SED is challenging due to the lack of precise temporal annotations and the risk of introducing noise through unreliable filtering. To address these challenges and enable generative-based augmentation for SED, we propose SynSonic, a data augmentation method tailored for this task. SynSonic leverages text-to-audio diffusion models guided by an energy-envelope ControlNet to generate temporally coherent sound events. A joint score filtering strategy with dual classifiers ensures sample quality, and we explore its practical integration into training pipelines. Experimental results show that SynSonic improves Polyphonic Sound Detection Scores (PSDS1 and PSDS2), enhancing both temporal localization and sound class discrimination.

Citations

• Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations
• Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data
• EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer
• SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer
• Improving Text-To-Audio Models with Synthetic Captions
• Scaling up masked audio encoder learning for general audio classification
• Music ControlNet: Multiple Time-varying Controls for Music Generation
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• Effective Data Augmentation With Diffusion Models
• Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Frequency Dynamic Convolution: Frequency-Adaptive Pattern Recognition for Sound Event Detection
• SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
• The Benefit Of Temporally-Strong Labels In Audio Event Classification
• A Framework for the Robust Evaluation of Sound Event Detection
• Unsupervised Data Augmentation for Consistency Training
• A multi-device dataset for urban acoustic scene classification
• mixup: Beyond Empirical Risk Minimization

Related

• DiffSED: Sound Event Detection with Denoising Diffusion
• Polyphonic Sound Event and Sound Activity Detection: A Multi-task approach
• A Multi-Task Learning Framework for Sound Event Detection using High-level Acoustic Characteristics of Sounds
• FMSG-JLESS Submission for DCASE 2024 Task4 on Sound Event Detection with Heterogeneous Training Dataset and Potentially Missing Labels
• A Robust framework for sound event localization and detection on real recordings