2017/11/16 by Chih‐Yao Ma, Chih-Yao Ma, Ma, Chih-Yao +10 · 5 citations
Computer Science · #Artificial intelligence #Closed captioning #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Human–computer interaction #Image (mathematics) #Multimedia #Multimodal Machine Learning Applications #Object (grammar) #Programming language #State (computer science) #cs.CV
paper · pdf · doi:10.48550/arxiv.1711.06354
published in arXiv (Cornell University) (Cornell University) · arXiv admin note: substantial text overlap with arXiv:1711.06330
arxiv created 2017/11/16 · openalex publication_date 2017/11/16 · arxiv updated 2017/11/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
We address the problem of video captioning by grounding language generation on object interactions in the video. Existing work mostly focuses on overall scene understanding with often limited or no emphasis on object interactions to address the problem of video understanding. In this paper, we propose SINet-Caption that learns to generate captions grounded over higher-order interactions between arbitrary groups of objects for fine-grained video understanding. We discuss the challenges and benefits of such an approach. We further demonstrate state-of-the-art results on the ActivityNet Captions dataset using our model, SINet-Caption based on this approach.