2020/07/28 by Shaoxiang Chen, Chen, Shaoxiang, Wenhao Jiang +6 · 11 citations
Computer Science · #Artificial intelligence #Benchmark (surveying) #Closed captioning #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Event (particle physics) #Exploit #FOS: Computer and information sciences #Human Pose and Action Recognition #Image (mathematics) #Leverage (statistics) #Machine learning #Modalities #Modality (human–computer interaction) #Multimodal Machine Learning Applications #Natural language processing #Pairwise comparison #Sentence #Speech recognition #Task (project management) #Video Analysis and Summarization #cs.CV
paper · pdf · doi:10.48550/arxiv.2007.14164
published in arXiv (Cornell University) (Cornell University)
arxiv created 2020/07/28 · openalex publication_date 2020/07/28 · arxiv updated 2020/07/29 · openalex created_date 2020/08/03 · openalex updated_date 2026/08/05
Automatically generating sentences to describe events and temporally localizing sentences in a video are two important tasks that bridge language and videos. Recent techniques leverage the multimodal nature of videos by using off-the-shelf features to represent videos, but interactions between modalities are rarely explored. Inspired by the fact that there exist cross-modal interactions in the human brain, we propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos and thus improve performances on both tasks. We model modality interaction in both the sequence and channel levels in a pairwise fashion, and the pairwise interaction also provides some explainability for the predictions of target tasks. We demonstrate the effectiveness of our method and validate specific design choices through extensive ablation studies. Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets: MSVD and MSR-VTT (event captioning task), and Charades-STA and ActivityNet Captions (temporal sentence localization task).