2024/06/11 by Shester Gueuwou, Gueuwou, Shester, Xiaodan Du +5 · 8 citations
Arts and Humanities · Chemistry · Computer Science · Mathematics · Psychology · #Cartography #Chemistry #Computer science #Geography #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Mathematics #Natural language processing #Scale (ratio) #Sign (mathematics) #Subtitles and Audiovisual Media #Translation (biology)
paper · pdf · doi:10.48550/arxiv.2406.06907
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
A persistent challenge in sign language video processing, including the task of sign to written language translation, is how we learn representations of sign language in an effective and efficient way that preserves the important attributes of these languages, while remaining invariant to irrelevant visual differences. Informed by the nature and linguistics of signed languages, our proposed method focuses on just the most relevant parts in a signing video: the face, hands and body pose of the signer. However, instead of fully relying on pose estimation from off-the-shelf pose tracking models, which have inconsistent performance for hands and faces, we propose to learn a representation of the complex handshapes and facial expressions of sign languages in a self-supervised fashion. Our approach is based on learning from individual frames (rather than video sequences) and is therefore much more efficient than prior work on sign language pre-training. Compared to a recent model that established a new state of the art in sign language translation on the How2Sign dataset, our approach yields similar translation performance, using less than 3% of the compute.