2022/04/26 by Hanyu Xuan, Xuan, Hanyu, Yihong Xu +11
Computer Science · #FOS: Computer and information sciences #Multimedia (cs.MM) #Music and Audio Processing #Speech and Audio Processing #Video Analysis and Summarization #cs.MM
paper · pdf · doi:10.48550/arxiv.2204.12366
7 pages, 4 figures, accepted at IJCAI 2022
arxiv created 2022/04/26 · openalex publication_date 2022/04/26 · arxiv updated 2022/04/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The recent success of audio-visual representation learning can be largely attributed to their pervasive property of audio-visual synchronization, which can be used as self-annotated supervision. As a state-of-the-art solution, Audio-Visual Instance Discrimination (AVID) extends instance discrimination to the audio-visual realm. Existing AVID methods construct the contrastive set by random sampling based on the assumption that the audio and visual clips from all other videos are not semantically related. We argue that this assumption is rough, since the resulting contrastive sets have a large number of faulty negatives. In this paper, we overcome this limitation by proposing a novel Active Contrastive Set Mining (ACSM) that aims to mine the contrastive sets with informative and diverse negatives for robust AVID. Moreover, we also integrate a semantically-aware hard-sample mining strategy into our ACSM. The proposed ACSM is implemented into two most recent state-of-the-art AVID methods and significantly improves their performance. Extensive experiments conducted on both action and sound recognition on multiple datasets show the remarkably improved performance of our method.