Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM

2025/05/29 by Zhiming Sun, Li Zhang, Sun, Zhaokai +7
Computer Science · Psychology · #Speech and Audio Processing #Speech Recognition and Synthesis #Emotion and Mood Recognition

paper · pdf · doi:10.48550/arxiv.2505.23207

Abstract

Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation, a critical challenge in multi-party speech processing. This work proposes a speaker-aware progressive OSD model that leverages a progressive training strategy to enhance the correlation between subtasks such as voice activity detection (VAD) and overlap detection. To improve acoustic representation, we explore the effectiveness of state-of-the-art self-supervised learning (SSL) models, including WavLM and wav2vec 2.0, while incorporating a speaker attention module to enrich features with frame-level speaker information. Experimental results show that the proposed method achieves state-of-the-art performance, with an F1 score of 82.76% on the AMI test set, demonstrating its robustness and effectiveness in OSD.

Related

• Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains
• Channel-Combination Algorithms for Robust Distant Voice Activity and Overlapped Speech Detection
• Large-Scale Learning on Overlapped Speech Detection: New Benchmark and New General System
• Joint Speech Activity and Overlap Detection with Multi-Exit Architecture
• Three-class Overlapped Speech Detection using a Convolutional Recurrent\n Neural Network