AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation

2024/07/10 by Kaifeng Zhang, Baoyu Li, Zhang, Kaifeng +5 · 18 citations
Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Modular Robots and Swarm Intelligence #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2407.07889

openalex publication_date 2024/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Predictive models are a crucial component of many robotic systems. Yet, constructing accurate predictive models for a variety of deformable objects, especially those with unknown physical properties, remains a significant challenge. This paper introduces AdaptiGraph, a learning-based dynamics modeling approach that enables robots to predict, adapt to, and control a wide array of challenging deformable materials with unknown physical properties. AdaptiGraph leverages the highly flexible graph-based neural dynamics (GBND) framework, which represents material bits as particles and employs a graph neural network (GNN) to predict particle motion. Its key innovation is a unified physical property-conditioned GBND model capable of predicting the motions of diverse materials with varying physical properties without retraining. Upon encountering new materials during online deployment, AdaptiGraph utilizes a physical property optimization process for a few-shot adaptation of the model, enhancing its fit to the observed interaction data. The adapted models can precisely simulate the dynamics and predict the motion of various deformable materials, such as ropes, granular media, rigid boxes, and cloth, while adapting to different physical properties, including stiffness, granular size, and center of pressure. On prediction and manipulation tasks involving a diverse set of real-world deformable objects, our method exhibits superior prediction accuracy and task proficiency over non-material-conditioned and non-adaptive models. The project page is available at https://robopil.github.io/adaptigraph/ .

Cited by

• Closing the Train-Test Gap in World Models for Gradient-Based Planning
• SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models
• Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions
• Heuristic Adaptation of Potentially Misspecified Domain Support for Likelihood-Free Inference in Stochastic Dynamical Systems
• PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis
• Hierarchical DLO Routing with Reinforcement Learning and In-Context Vision-language Models
• Ego-Vision World Model for Humanoid Contact Planning
• NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
• DYMO-Hair: Generalizable Volumetric Dynamics Modeling for Robot Hair Manipulation
• PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
• Learning Geometry-Aware Nonprehensile Pushing and Pulling with Dexterous Hands
• Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation
• CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints
• ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation
• Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos
• Sparse Imagination for Efficient Visual World Model Planning
• RLVR-World: Training World Models with Reinforcement Learning
• Causal World Modeling for Robot Control

Related

• RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks
• Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects
• PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics
• Physics-Encoded Graph Neural Networks for Deformation Prediction under Contact
• Bimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model