2020/03/30 by Necati Cihan Camgöz, Oscar Koller, Camgoz, Necati Cihan +5 · 22 citations
Computer Science · Engineering · Psychology · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Gait Recognition and Analysis #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2003.13830
openalex publication_date 2020/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Prior work on Sign Language Translation has shown that having a mid-level\nsign gloss representation (effectively recognizing the individual signs)\nimproves the translation performance drastically. In fact, the current\nstate-of-the-art in translation requires gloss level tokenization in order to\nwork. We introduce a novel transformer based architecture that jointly learns\nContinuous Sign Language Recognition and Translation while being trainable in\nan end-to-end manner. This is achieved by using a Connectionist Temporal\nClassification (CTC) loss to bind the recognition and translation problems into\na single unified architecture. This joint approach does not require any\nground-truth timing information, simultaneously solving two co-dependant\nsequence-to-sequence learning problems and leads to significant performance\ngains.\n We evaluate the recognition and translation performances of our approaches on\nthe challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset. We report\nstate-of-the-art sign language recognition and translation results achieved by\nour Sign Language Transformers. Our translation networks outperform both sign\nvideo to spoken language and gloss to spoken language translation models, in\nsome cases more than doubling the performance (9.58 vs. 21.80 BLEU-4 Score). We\nalso share new baseline translation results using transformer networks for\nseveral other text-to-text sign language translation tasks.\n