2021/08/05 by Yuanhang Zhang, Susan Liang, Shuang Yang +4
Computer Science · Engineering · #Aggregate (composite) #Context (archaeology) #Key (lock) #Margin (machine learning) #Music and Audio Processing #Pattern recognition (psychology) #Process (computing) #Robustness (evolution) #Spatial contextual awareness #Speech Recognition and Synthesis #Speech and Audio Processing #cs.CV #cs.MM #cs.SD #eess.AS #eess.IV
paper · pdf · doi:10.1145/3474085.3475275
10 pages, 6 figures; to appear at ACM Multimedia 2021
arxiv created 2021/08/05 · arxiv updated 2021/08/06 · openalex created_date 2021/08/16 · openalex publication_date 2021/10/17 · openalex updated_date 2026/08/05
We propose a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD). Traditional methods for ASD usually operate on each candidate's pre-cropped face track separately and do not sufficiently consider the relationships among the candidates. This potentially limits performance, especially in challenging scenarios with low-resolution faces, multiple candidates, etc. Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information: spatial context to indicate the position and scale of each candidate's face, relational context to capture the visual relationships among the candidates and contrast audio-visual affinities with each other, and temporal context to aggregate long-term information and smooth out local uncertainties. Based on such information, our model optimizes all candidates in a unified process for robust and reliable ASD. A thorough ablation study is performed on several challenging ASD benchmarks under different settings. In particular, our method outperforms the state-of-the-art by a large margin of about 15% mean Average Precision (mAP) absolute on two challenging subsets: one with three candidate speakers, and the other with faces smaller than 64 pixels. Together, our UniCon achieves 92.0% mAP on the AVA-ActiveSpeaker validation set, surpassing 90% for the first time on this challenging dataset at the time of submission. Project website: https://unicon-asd.github.io/.