Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing

2025/09/17 by Chen, Yaru, Guo, Ruohao, Gao, Liting +4
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimedia (cs.MM)

paper · doi:10.48550/arxiv.2509.14097

Abstract

Weakly-supervised audio-visual video parsing (AVVP) seeks to detect audible, visible, and audio-visual events without temporal annotations. Previous work has emphasized refining global predictions through contrastive or collaborative learning, but neglected stable segment-level supervision and class-aware cross-modal alignment. To address this, we propose two strategies: (1) an exponential moving average (EMA)-guided pseudo supervision framework that generates reliable segment-level masks via adaptive thresholds or top-k selection, offering stable temporal guidance beyond video-level labels; and (2) a class-aware cross-modal agreement (CMA) loss that aligns audio and visual embeddings at reliable segment-class pairs, ensuring consistency across modalities while preserving temporal structure. Evaluations on LLP and UnAV-100 datasets shows that our method achieves state-of-the-art (SOTA) performance across multiple metrics.

Citations

• Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
• Boosting Audio Visual Question Answering via Key Semantic-Aware Cues
• CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing
• Audio-Visual Instance Segmentation
• CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing
• Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser
• Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
• Learning Transferable Visual Models From Natural Language Supervision
• Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
• CNN Architectures for Large-Scale Audio Classification

Related

• CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing
• Cross-Modal learning for Audio-Visual Video Parsing
• UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing
• Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective
• LINK: Adaptive Modality Interaction for Audio-Visual Video Parsing