Real-Time Sign Language Detection using Human Pose Estimation

2020/08/11 by Moryossef, Amit, Tsochantaridis, Ioannis, Aharoni, Roee +2 · 2 citations
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2008.04637

Abstract

We propose a lightweight real-time sign language detection model, as we identify the need for such a case in videoconferencing. We extract optical flow features based on human pose estimation and, using a linear classifier, show these features are meaningful with an accuracy of 80%, evaluated on the DGS Corpus. Using a recurrent model directly on the input, we see improvements of up to 91% accuracy, while still working under 4ms. We describe a demo application to sign language detection in the browser in order to demonstrate its usage possibility in videoconferencing applications.

Cited by

• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment
• POSESTITCH-SLT: Linguistically Inspired Pose-Stitching for End-to-End Sign Language Translation

Related

• Combining Efficient and Precise Sign Language Recognition: Good pose estimation library is all you need
• SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space
• Pose-Based Sign Language Spotting via an End-to-End Encoder Architecture
• Isolated Sign Language Recognition with Segmentation and Pose Estimation
• Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison