Efficient Egocentric Action Recognition with Multimodal Data

2025/06/02 by Calzavara, Marco, Kastrati, Ard, Macchini, Matteo +2
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.01757

Abstract

The increasing availability of wearable XR devices opens new perspectives for Egocentric Action Recognition (EAR) systems, which can provide deeper human understanding and situation awareness. However, deploying real-time algorithms on these devices can be challenging due to the inherent trade-offs between portability, battery life, and computational resources. In this work, we systematically analyze the impact of sampling frequency across different input modalities - RGB video and 3D hand pose - on egocentric action recognition performance and CPU usage. By exploring a range of configurations, we provide a comprehensive characterization of the trade-offs between accuracy and computational efficiency. Our findings reveal that reducing the sampling rate of RGB frames, when complemented with higher-frequency 3D hand pose input, can preserve high accuracy while significantly lowering CPU demands. Notably, we observe up to a 3x reduction in CPU usage with minimal to no loss in recognition performance. This highlights the potential of multimodal input strategies as a viable approach to achieving efficient, real-time EAR on XR devices.

Citations

• On the Utility of 3D Hand Poses for Action Recognition
• HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
• Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model
• H2O: Two Hands Manipulating Objects for First Person Interaction Recognition
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Designing Network Design Spaces
• What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention
• Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Distilling the Knowledge in a Neural Network
• Two-Stream Convolutional Networks for Action Recognition in Videos

Related

• Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input
• Egocentric RGB+Depth Action Recognition in Industry-Like Settings
• x-RAGE: eXtended Reality -- Action & Gesture Events Dataset
• Improving Keystep Recognition in Ego-Video via Dexterous Focus
• EgoCHARM: Resource-Efficient Hierarchical Activity Recognition using an Egocentric IMU Sensor