2014/03/19 by Qinggang Yue, Yue, Qinggang, Zhen Ling +7
Computer Science · #Advanced Malware Detection Techniques #Biometric Identification and Security #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #User Authentication and Security Systems #cs.CR
paper · pdf · doi:10.48550/arxiv.1403.4829
arxiv created 2014/03/19 · openalex publication_date 2014/03/19 · arxiv updated 2014/03/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this paper, we introduce a novel computer vision based attack that discloses inputs on a touch enabled device, while the attacker cannot see any text or popups from a video of the victim tapping on the touch screen. In the attack, we use the optical flow algorithm to identify touching frames where the finger touches the screen surface. We innovatively use intersections of detected edges of the touch screen to derive the homography matrix mapping the touch screen surface in video frames to a reference image of the virtual keyboard. We analyze the shadow formation around the fingertip and use the k-means clustering algorithm to identify touched points. Homography can then map these touched points to keys of the virtual keyboard. Our work is substantially different from existing work. We target password input and are able to achieve a high success rate. We target scenarios like classrooms, conferences and similar gathering places and use a webcam or smartphone camera. In these scenes, single-lens reflex (SLR) cameras and high-end camcorders used in related work will appear suspicious. To defeat such computer vision based attacks, we design, implement and evaluate the Privacy Enhancing Keyboard (PEK) where a randomized virtual keyboard is used to input sensitive information.