Biomechanically consistent real-time action recognition for human-robot interaction

2025/10/21 by Wanchen Li, Kahina Chalabi, Li, Wanchen +13
Computer Science · Engineering · #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Human Pose and Action Recognition #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2510.18373

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

This paper presents a novel framework for real-time human action recognition in industrial contexts, using standard 2D cameras. We introduce a complete pipeline for robust and real-time estimation of human joint kinematics, input to a temporally smoothed Transformer-based network, for action recognition. We rely on a new dataset including 11 subjects performing various actions, to evaluate our approach. Unlike most of the literature that relies on joint center positions (JCP) and is offline, ours uses biomechanical prior, eg. joint angles, for fast and robust real-time recognition. Besides, joint angles make the proposed method agnostic to sensor and subject poses as well as to anthropometric differences, and ensure robustness across environments and subjects. Our proposed learning model outperforms the best baseline model, running also in real-time, along various metrics. It achieves 88% accuracy and shows great generalization ability, for subjects not facing the cameras. Finally, we demonstrate the robustness and usefulness of our technique, through an online interaction experiment, with a simulated robot controlled in real-time via the recognized actions.

Citations

• OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics
• RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
• YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
• YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
• Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition
• Long Short-Term Transformer for Online Action Detection
• Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition
• HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation
• 3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information
• You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization
• MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
• View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition
• Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
• Attention Is All You Need
• Online Action Detection
• Long-term Temporal Convolutions for Action Recognition
• ISB recommendation on definitions of joint coordinate systems of various joints for the reporting of human joint motion—Part II: shoulder, elbow, wrist and hand
• ISB recommendation on definitions of joint coordinate system of various joints for the reporting of human joint motion—part I: ankle, hip, and spine
• A flexible new technique for camera calibration

Related

• Simultaneous Learning from Human Pose and Object Cues for Real-Time Activity Recognition
• ActionXPose: A Novel 2D Multi-view Pose-based Algorithm for Real-time Human Action Recognition
• Using joint angles based on the international biomechanical standards for human action recognition and related tasks
• Simple yet efficient real-time pose-based action recognition
• Attention-Oriented Action Recognition for Real-Time Human-Robot Interaction