ConvNet Architecture Search for Spatiotemporal Feature Learning

2017/08/16 by Du Tran, Tran, Du, Jamie Ray +8 · 9 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.1708.05038

arxiv created 2017/08/16 · openalex publication_date 2017/08/16 · arxiv updated 2017/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.

Cited by

• DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
• Automated identification of neural cells in the multi-photon images using deep-neural networks
• AVD: Adversarial Video Distillation
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Spatio-Temporal Channel Correlation Networks for Action Classification
• Flow-Distilled IP Two-Stream Networks for Compressed Video Action Recognition
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance
• Fish feeding intensity quantification using machine vision and a lightweight 3D ResNet-GloRe network

Related

• Learning Spatiotemporal Features with 3D Convolutional Networks
• Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• Initialization Strategies of Spatio-Temporal Convolutional Neural Networks
• Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification