2020/09/24 by Yidan Qin, Qin, Yidan, Seyedshams Feyzabadi +7 · 1 citation
Computer Science · Engineering · Medicine · #Anatomy and Medical Technology #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO) #Soft Robotics and Applications #Surgical Simulation and Training #cs.CV #cs.LG #cs.RO
paper · pdf · doi:10.48550/arxiv.2009.11937
Accepted to IROS 2020
arxiv created 2020/09/24 · openalex publication_date 2020/09/24 · arxiv updated 2020/09/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.