Investigating self-supervised learning for speech enhancement and separation

2022/03/15 by Shinji Watanabe, Huang, Zili, Shuwen Yang +6 · 3 citations
Computer Science · Engineering · #Advanced Adaptive Filtering Techniques #Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2203.07960

openalex publication_date 2022/03/15 · openalex created_date 2022/04/03 · openalex updated_date 2026/07/28

Abstract

Speech enhancement and separation are two fundamental tasks for robust speech processing. Speech enhancement suppresses background noise while speech separation extracts target speech from interfering speakers. Despite a great number of supervised learning-based enhancement and separation methods having been proposed and achieving good performance, studies on applying self-supervised learning (SSL) to enhancement and separation are limited. In this paper, we evaluate 13 SSL upstream methods on speech enhancement and separation downstream tasks. Our experimental results on Voicebank-DEMAND and Libri2Mix show that some SSL representations consistently outperform baseline features including the short-time Fourier transform (STFT) magnitude and log Mel filterbank (FBANK). Furthermore, we analyze the factors that make existing SSL frameworks difficult to apply to speech enhancement and separation and discuss the representation properties desired for both tasks. Our study is included as the official speech enhancement and separation downstreams for SUPERB.

Cited by

• PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement
• MT-HuBERT: Self-Supervised Mix-Training for Few-Shot Keyword Spotting in Mixed Speech
• Advances in Speech Separation: Techniques, Challenges, and Future Trends

Related

• Speech separation with large-scale self-supervised learning
• Downstream Task Agnostic Speech Enhancement with Self-Supervised Representation Loss
• Boosting Self-Supervised Embeddings for Speech Enhancement
• SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
• TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models