Learning Contact Dynamics through Touching: Action-conditional Graph Neural Networks for Robotic Peg Insertion

2025/09/15 by Zongyao Yi, Yi, Zongyao, Joachim Hertzberg +3
Computer Science · Engineering · Medicine · #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Robot Manipulation and Learning #Robotics (cs.RO) #Stroke Rehabilitation and Recovery

paper · pdf · doi:10.48550/arxiv.2509.12151

openalex publication_date 2025/09/15 · openalex created_date 2025/10/12 · openalex updated_date 2026/07/30

Abstract

We present a learnable physics-based predictive model that provides accurate motion and force-torque prediction of the robot end effector in contact-rich manipulation. The proposed model extends the state-of-the-art GNN-based simulator (FIGNet) with novel node and edge types, enabling action-conditional predictions for control and state estimation in the context of robotic peg insertion. Our model learns in a self-supervised manner, using only joint encoder and force-torque data while the robot is touching the environment. In simulation, the MPC agent using our model matches the performance of the same controller with the ground truth dynamics model in a challenging peg-in-hole task, while in the real-world experiment, our model achieves a 50% improvement in motion prediction accuracy and 3× increase in force-torque prediction precision over the baseline physics simulator. Finally, we apply the model to track the robot end effector with a particle filter during real-world peg insertion, demonstrating a practical application of its predictive accuracy.

Citations

• PyG 2.0: Scalable Learning on Real World Graphs
• Learning the Contact Manifold for Accurate Pose Estimation During Peg-in-Hole Insertion of Complex Geometries
• Integrating Physics and Topology in Neural Networks for Learning Rigid Body Dynamics
• Tactile Probabilistic Contact Dynamics Estimation of Unknown Objects
• Efficient Online Learning of Contact Force Models for Connector Insertion
• Learning rigid dynamics with face interaction graph networks
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Multi-Stage Learning for Grasp-Constrained Object Manipulation with a Simulated Panda Robot
• Sample-efficient Cross-Entropy Method for Real-time Planning
• Accelerating 3D Deep Learning with PyTorch3D
• Propagation Networks for Model-Based Control Under Partial Observation
• Graph networks as learnable physics engines for inference and control

Related

• Simultaneous Learning of Contact and Continuous Dynamics
• Augmenting Neural Networks-Based Model Approximators in Robotic Force-Tracking Tasks
• Dynamic Inference on Graphs using Structured Transition Models
• Physics-Encoded Graph Neural Networks for Deformation Prediction under Contact
• Combining Physical Simulators and Object-Based Networks for Control