2019/10/15 by Alejandro Cartas, Jordi Luque, Cartas, Alejandro +7
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #cs.CV #cs.LG #eess.AS #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1910.06693
Accepted for the Fifth International Workshop on Egocentric Perception, Interaction and Computing (EPIC) at the International Conference on Computer Vision (ICCV) 2019
arxiv created 2019/10/15 · arxiv updated 2019/10/16
Our interaction with the world is an inherently multimodal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial, and temporal streams. Experimental results on the EPIC-Kitchens dataset show that multimodal integration leads to better performance than unimodal approaches. In particular, we achieved a 5.18% improvement over the state of the art on verb classification.