Audiovisual SlowFast Networks for Video Recognition

2020/01/23 by Fanyi Xiao, Yong Jae Lee, Xiao, Fanyi +7 · 18 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Music and Audio Processing #Speech and Audio Processing #cs.CV

paper · pdf · doi:10.48550/arxiv.2001.08740

Technical report

openalex publication_date 2020/01/23 · arxiv created 2020/03/09 · arxiv updated 2020/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, we perform hierarchical audiovisual synchronization to learn joint audiovisual features. We report state-of-the-art results on six video action classification and detection datasets, perform detailed ablation studies, and show the generalization of AVSlowFast to learn self-supervised audiovisual features. Code will be made available at: https://github.com/facebookresearch/SlowFast.

Citations

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improving neural networks by preventing co-adaptation of feature detectors
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Spatiotemporal Residual Networks for Video Action Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Compressed Video Action Recognition
• Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videos
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Revisiting Self-Supervised Visual Representation Learning
• FBK-HUPBA Submission to the EPIC-Kitchens 2019 Action Recognition Challenge
• Learning Video Representations using Contrastive Bidirectional Transformer
• Momentum Contrast for Unsupervised Visual Representation Learning
• What Makes Training Multi-Modal Classification Networks Hard?

Cited by

• TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
• EgoEMS: A High-Fidelity Multimodal Egocentric Dataset for Cognitive Assistance in Emergency Medical Services
• D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition
• MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization
• VGGSounder: Audio-Visual Evaluations for Foundation Models
• Attention-Driven Multimodal Alignment for Long-term Action Quality Assessment
• Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos
• OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning
• DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding
• Domain Generalization using Action Sequences for Egocentric Action Recognition
• DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation
• Where and When: Space-Time Attention for Audio-Visual Explanations
• Can audio-visual integration strengthen robustness under multimodal attacks?
• LSTC: Boosting Atomic Action Detection with Long-Short-Term Context
• Attention Bottlenecks for Multimodal Fusion
• Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning

Related

• SlowFast Networks for Video Recognition
• Slow-Fast Auditory Streams For Audio Recognition
• V-SlowFast Network for Efficient Visual Sound Separation
• AENet: Learning Deep Audio Features for Video Analysis
• Multi-level Attention Fusion Network for Audio-visual Event Recognition