2021/11/30 by Aman Bhargava, Bhargava, Aman, Alice X. Zhou +5
Computer Science · Neuroscience · #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Human-Computer Interaction (cs.HC) #Interactive and Immersive Displays #Tactile and Sensory Interactions #cs.HC
paper · pdf · doi:10.48550/arxiv.2111.15053
15 pages, 11 figures
arxiv created 2021/11/30 · openalex publication_date 2021/11/30 · arxiv updated 2021/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The vibrations generated from scratching and tapping on surfaces can be highly expressive and recognizable, and have therefore been proposed as a method of natural user interface (NUI). Previous systems require custom sensor hardware such as contact microphones and have struggled with gesture classification accuracy. We propose a deep learning approach to scratch input. Using smartphones and tablets laid on tabletops or other similar surfaces, our system achieved a gesture classification accuracy of 95.8%, substantially reducing gesture misclassification from previous works. Further, our system achieved this performance when tested on a wide variety of surfaces, mobile devices, and in high noise environments. The results indicate high potential for the application of deep learning techniques to natural user interface (NUI) systems that can readily convert large unpowered surfaces into a user interface using just a smartphone with no special-purpose sensors or hardware.