2022/10/11 by Zifan Jiang, Jiang, Zifan, Amit Moryossef +5 · 2 citations
Computer Science · Psychology · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Natural Language Processing Techniques
paper · doi:10.48550/arxiv.2210.05404
openalex publication_date 2022/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup--translating from American Sign Language to (American) English--our method achieves over 30 BLEU, while in two multilingual setups--translating in both directions between spoken languages and signed languages--we achieve over 20 BLEU. We find that common MT techniques used to improve spoken language translation similarly affect the performance of sign language translation. These findings validate our use of an intermediate text representation for signed languages to include them in natural language processing research.