2022/03/09 by Abdishakour Awale, Awale, Abdishakour, Duygu Sarıkaya +1 · 1 citation
Computer Science · Medicine · Social Sciences · #Augmented Reality Applications #Computer Vision and Pattern Recognition (cs.CV) #Delphi Technique in Research #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Surgical Simulation and Training #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2203.04752
openalex publication_date 2022/03/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attention mechanism for activity recognition in surgical videos. Our model consists of an I3D-based architecture, learns spatio-temporal features using 3D convolutions, as well as learns an attention map using human gaze as supervision. We evaluate our model on the Suturing task of JIGSAWS which is a publicly available surgical video understanding dataset. To our knowledge, we are the first to use human gaze for surgical activity recognition. Our results and ablation studies support the contribution of using human gaze to guide attention by outperforming state-of-the art models with an accuracy of 85.4%.