Hand Pose Estimation through Semi-Supervised and Weakly-Supervised\n Learning

2015/11/20 by Natalia Neverova, Christian Wolf, Neverova, Natalia +5 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Human Pose and Action Recognition #Machine Learning (cs.LG) #Robot Manipulation and Learning

paper · pdf · doi:10.48550/arxiv.1511.06728

openalex publication_date 2015/11/20 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

We propose a method for hand pose estimation based on a deep regressor\ntrained on two different kinds of input. Raw depth data is fused with an\nintermediate representation in the form of a segmentation of the hand into\nparts. This intermediate representation contains important topological\ninformation and provides useful cues for reasoning about joint locations. The\nmapping from raw depth to segmentation maps is learned in a\nsemi/weakly-supervised way from two different datasets: (i) a synthetic dataset\ncreated through a rendering pipeline including densely labeled ground truth\n(pixelwise segmentations); and (ii) a dataset with real images for which ground\ntruth joint positions are available, but not dense segmentations. Loss for\ntraining on real images is generated from a patch-wise restoration process,\nwhich aligns tentative segmentation maps with a large dictionary of synthetic\nposes. The underlying premise is that the domain shift between synthetic and\nreal data is smaller in the intermediate representation, where labels carry\ngeometric and topological meaning, than in the raw input domain. Experiments on\nthe NYU dataset show that the proposed training method decreases error on\njoints over direct regression of joints from depth data by 15.7%.\n

Citations

• OpenGM: A C++ Library for Discrete Graphical Models
• Model-based Deep Hand Pose Estimation
• Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs
• Stacked Hourglass Networks for Human Pose Estimation
• Depth-based hand pose estimation: methods, data, and challenges
• Learning Deconvolution Network for Semantic Segmentation

Cited by

• AugmentGest: Can Random Data Cropping Augmentation Boost Gesture Recognition Performance?

Related

• Pushing the Envelope for Depth-Based Semi-Supervised 3D Hand Pose Estimation with Consistency Training
• End-to-end Global to Local CNN Learning for Hand Pose Recovery in Depth Data
• DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth
• Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation
• 3D Hand Pose Estimation using Simulation and Partial-Supervision with a Shared Latent Space