MediaPipe Hands: On-device Real-time Hand Tracking

2020/06/18 by Zhang, Fan, Bazarevsky, Valentin, Vakunov, Andrey +4 · 22 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2006.10214

Abstract

We present a real-time on-device hand tracking pipeline that predicts hand skeleton from single RGB camera for AR/VR applications. The pipeline consists of two models: 1) a palm detector, 2) a hand landmark model. It's implemented via MediaPipe, a framework for building cross-platform ML solutions. The proposed model and pipeline architecture demonstrates real-time inference speed on mobile GPUs and high prediction quality. MediaPipe Hands is open sourced at https://mediapipe.dev.

Cited by

• SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching
• Towards Senior-Robot Interaction: Reactive Robot Dog Gestures
• BLINDSPOT: Enabling Bystander-Controlled Privacy Signaling for Camera-Enabled Devices
• ACE-F: A Cross Embodiment Foldable System with Force Feedback for Dexterous Teleoperation
• TrackStudio: An Integrated Toolkit for Markerless Tracking
• Contextual Role Modulates Object Representational Geometry in the Human Brain
• How Pragmatics Shape Articulation: A Computational Case Study in STEM ASL Discourse
• Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
• Traj2Action: A Co-Denoising Framework for Trajectory-Guided Human-to-Robot Skill Transfer
• XR Blocks: Accelerating Human-centered AI + XR Innovation
• DexDirect: Direct Kinesthetic Arm Guidance for Efficient Dexterous Demonstration Collection
• Puzzle: Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors
• Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation
• EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View
• PianoVAM: A Multimodal Piano Performance Dataset
• EHWGesture -- A dataset for multimodal understanding of clinical gestures
• Pointing-Guided Target Estimation via Transformer-Based Attention
• EclipseTouch: Touch Segmentation on Ad Hoc Surfaces using Worn Infrared Shadow Casting
• EgoTouch: On-Body Touch Input Using AR/VR Headset Cameras
• Towards Spatially Consistent Image Generation: On Incorporating Intrinsic Scene Properties into Diffusion Models
• A Multimodal Framework for Understanding Collaborative Design Processes
• Human-Exoskeleton Kinematic Calibration to Improve Hand Tracking for Dexterous Teleoperation

Related

• On-device Real-time Hand Gesture Recognition
• Efficient Hand Articulations Tracking using Adaptive Hand Model and Depth map
• Tracking Fast by Learning Slow: An Event-based Speed Adaptive Hand Tracker Leveraging Knowledge in RGB Domain
• Hand tracking for clinical applications: validation of the Google MediaPipe Hand (GMH) and the depth-enhanced GMH-D frameworks
• EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View