Jacobian Exploratory Dual-Phase Reinforcement Learning for Dynamic Endoluminal Navigation of Deformable Continuum Robots

2025/08/30 by Yu Tian, Tian, Yu, Chi Kit Ng +3
Engineering · #Soft Robotics and Applications #Prosthetics and Rehabilitation Robotics #Robotic Mechanisms and Dynamics

paper · pdf · doi:10.48550/arxiv.2509.00329

Abstract

Deformable continuum robots (DCRs) present unique planning challenges due to nonlinear deformation mechanics and partial state observability, violating the Markov assumptions of conventional reinforcement learning (RL) methods. While Jacobian-based approaches offer theoretical foundations for rigid manipulators, their direct application to DCRs remains limited by time-varying kinematics and underactuated deformation dynamics. This paper proposes Jacobian Exploratory Dual-Phase RL (JEDP-RL), a framework that decomposes planning into phased Jacobian estimation and policy execution. During each training step, we first perform small-scale local exploratory actions to estimate the deformation Jacobian matrix, then augment the state representation with Jacobian features to restore approximate Markovianity. Extensive SOFA surgical dynamic simulations demonstrate JEDP-RL's three key advantages over proximal policy optimization (PPO) baselines: 1) Convergence speed: 3.2x faster policy convergence, 2) Navigation efficiency: requires 25% fewer steps to reach the target, and 3) Generalization ability: achieve 92% success rate under material property variations and achieve 83% (33% higher than PPO) success rate in the unseen tissue environment.

Citations

• Generalizable whole-body global manipulation of deformable linear objects by dual-arm robot in 3-D constrained environments
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Neural Network Dynamics for Model-Based Deep Reinforcement Learning with\n Model-Free Fine-Tuning
• Proximal Policy Optimization Algorithms
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
• Continuous control with deep reinforcement learning
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Design and Kinematic Modeling of Constant Curvature Continuum Robots: A Review

Related

• Efficient reinforcement learning control for continuum robots based on Inexplicit Prior Knowledge
• Deep Reinforcement Learning-Based Motion Planning and PDE Control for Flexible Manipulators
• Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects
• Real-Robot Deep Reinforcement Learning: Improving Trajectory Tracking of Flexible-Joint Manipulator with Reference Correction
• Learning-based Control for Tendon-Driven Continuum Robotic Arms