2020/06/11 by Osama Mazhar, Mazhar, Osama, Sofiane Ramdani +3
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Human Pose and Action Recognition #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Robot Manipulation and Learning #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2006.06321
openalex publication_date 2020/06/11 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
Intuitive user interfaces are indispensable to interact with the human\ncentric smart environments. In this paper, we propose a unified framework that\nrecognizes both static and dynamic gestures, using simple RGB vision (without\ndepth sensing). This feature makes it suitable for inexpensive human-robot\ninteraction in social or industrial settings. We employ a pose-driven spatial\nattention strategy, which guides our proposed Static and Dynamic gestures\nNetwork - StaDNet. From the image of the human upper body, we estimate his/her\ndepth, along with the region-of-interest around his/her hands. The\nConvolutional Neural Network in StaDNet is fine-tuned on a\nbackground-substituted hand gestures dataset. It is utilized to detect 10\nstatic gestures for each hand as well as to obtain the hand image-embeddings.\nThese are subsequently fused with the augmented pose vector and then passed to\nthe stacked Long Short-Term Memory blocks. Thus, human-centred frame-wise\ninformation from the augmented pose vector and from the left/right hands\nimage-embeddings are aggregated in time to predict the dynamic gestures of the\nperforming person. In a number of experiments, we show that the proposed\napproach surpasses the state-of-the-art results on the large-scale Chalearn\n2016 dataset. Moreover, we transfer the knowledge learned through the proposed\nmethodology to the Praxis gestures dataset, and the obtained results also\noutscore the state-of-the-art on this dataset.\n