2024/10/02 by Ahmed Tawfik Aboukhadra, Nadia Robertini, Aboukhadra, Ahmed Tawfik +9 · 1 citation
Engineering · Medicine · #Anatomy and Medical Technology #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Medical Imaging and Analysis #Surgical Simulation and Training
paper · pdf · doi:10.48550/arxiv.2410.01293
openalex publication_date 2024/10/02 · openalex created_date 2024/10/30 · openalex updated_date 2026/07/28
Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the lack of annotated datasets and the complexity of the task, only a few works have addressed this problem. In this work, we present SurgeoNet, a real-time neural network pipeline to accurately detect and track surgical instruments from a stereo VR view. Our multi-stage approach is inspired by state-of-the-art neural-network architectural design, like YOLO and Transformers. We demonstrate the generalization capabilities of SurgeoNet in challenging real-world scenarios, achieved solely through training on synthetic data. The approach can be easily extended to any new set of articulated surgical instruments. SurgeoNet's code and data are publicly available.