An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation

2023/05/28 by Amit Moryossef, Moryossef, Amit, Mathias Müller +9 · 2 citations
Computer Science · Psychology · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2305.17714

openalex publication_date 2023/05/28 · openalex created_date 2023/05/31 · openalex updated_date 2026/07/28

Abstract

Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.

Cited by

• Introducing A Bangla Sentence - Gloss Pair Dataset for Bangla Sign Language Translation and Research
• Meaningful Pose-Based Sign Language Evaluation

Related

• Data Augmentation for Sign Language Gloss Translation
• Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
• Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation
• Hierarchical Feature Alignment for Gloss-Free Sign Language Translation
• Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing